ada tools update
This commit is contained in:
Josh at WLTechBlog
375
test/README.md
Normal file
375
test/README.md
Normal file
@@ -0,0 +1,375 @@
|
||||
# Cremote Accessibility Integration Tests
|
||||
|
||||
## Overview
|
||||
|
||||
This directory contains comprehensive integration tests for Cremote's accessibility testing tools. The tests validate all ADA/WCAG compliance features against known accessible and inaccessible test pages.
|
||||
|
||||
## Test Coverage
|
||||
|
||||
### Tools Tested
|
||||
|
||||
1. **Axe-Core Integration** (`web_inject_axe_cremotemcp`, `web_run_axe_cremotemcp`)
|
||||
- Library injection
|
||||
- WCAG 2.1 AA/AAA automated testing
|
||||
- Violation detection and reporting
|
||||
|
||||
2. **Contrast Checking** (`web_contrast_check_cremotemcp`)
|
||||
- WCAG contrast ratio calculations
|
||||
- AA/AAA compliance validation
|
||||
- Violation reporting with element details
|
||||
|
||||
3. **Keyboard Navigation** (`web_keyboard_test_cremotemcp`)
|
||||
- Focusability testing
|
||||
- Focus indicator validation
|
||||
- Keyboard trap detection
|
||||
- Tab order verification
|
||||
|
||||
4. **Zoom Testing** (`web_zoom_test_cremotemcp`)
|
||||
- Multi-level zoom testing (100%, 200%, 400%)
|
||||
- Horizontal scroll detection
|
||||
- Text readability validation
|
||||
- Overflow detection
|
||||
|
||||
5. **Reflow Testing** (`web_reflow_test_cremotemcp`)
|
||||
- Responsive breakpoint testing (320px, 1280px)
|
||||
- Horizontal scroll detection
|
||||
- Responsive layout validation
|
||||
- Overflow detection
|
||||
|
||||
6. **Screenshot Enhancements** (`web_screenshot_cremotemcp`)
|
||||
- Zoom level screenshots
|
||||
- Viewport size screenshots
|
||||
- Full page screenshots
|
||||
|
||||
7. **Library Injection** (`console_command_cremotemcp`)
|
||||
- External library injection (jQuery, Lodash, etc.)
|
||||
- Custom URL injection
|
||||
- Command execution with injected libraries
|
||||
|
||||
8. **Accessibility Tree** (`get_accessibility_tree_cremotemcp`)
|
||||
- Full tree retrieval
|
||||
- Depth limiting
|
||||
- Contrast annotation support
|
||||
|
||||
## Test Files
|
||||
|
||||
### Test Pages
|
||||
|
||||
- **`testdata/test-accessible.html`** - WCAG 2.1 AA compliant page
|
||||
- High contrast colors (4.5:1+ for normal text)
|
||||
- Visible focus indicators
|
||||
- Semantic HTML
|
||||
- Proper ARIA labels
|
||||
- Responsive design (no horizontal scroll at 320px)
|
||||
- Keyboard accessible controls
|
||||
- Proper heading hierarchy
|
||||
- Form labels and associations
|
||||
|
||||
- **`testdata/test-inaccessible.html`** - Page with known violations
|
||||
- Low contrast text (3.2:1, 1.5:1)
|
||||
- Missing focus indicators
|
||||
- Non-keyboard accessible elements
|
||||
- Missing alt text
|
||||
- Fixed width causing horizontal scroll
|
||||
- Missing form labels
|
||||
- Heading hierarchy violations
|
||||
- Duplicate IDs
|
||||
- Empty links and buttons
|
||||
- Incorrect ARIA usage
|
||||
|
||||
### Test Suite
|
||||
|
||||
- **`accessibility_integration_test.go`** - Main test suite
|
||||
- `TestAccessibilityToolsIntegration` - Tests all tools on accessible page
|
||||
- `TestInaccessiblePage` - Tests violation detection on inaccessible page
|
||||
- `BenchmarkAxeCore` - Performance benchmarks
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### 1. Start Cremote Daemon
|
||||
|
||||
```bash
|
||||
# Start the daemon on default port 9223
|
||||
cremote-daemon
|
||||
```
|
||||
|
||||
### 2. Start Test HTTP Server
|
||||
|
||||
You need a simple HTTP server to serve the test HTML files:
|
||||
|
||||
```bash
|
||||
# Option 1: Python 3
|
||||
cd test/testdata
|
||||
python3 -m http.server 8080
|
||||
|
||||
# Option 2: Python 2
|
||||
cd test/testdata
|
||||
python -m SimpleHTTPServer 8080
|
||||
|
||||
# Option 3: Node.js (http-server)
|
||||
cd test/testdata
|
||||
npx http-server -p 8080
|
||||
|
||||
# Option 4: Go
|
||||
cd test/testdata
|
||||
go run -m http.server 8080
|
||||
```
|
||||
|
||||
The test server should serve files at `http://localhost:8080/`.
|
||||
|
||||
### 3. Set Environment Variable
|
||||
|
||||
```bash
|
||||
export INTEGRATION_TESTS=true
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Run All Integration Tests
|
||||
|
||||
```bash
|
||||
cd test
|
||||
INTEGRATION_TESTS=true go test -v
|
||||
```
|
||||
|
||||
### Run Specific Test
|
||||
|
||||
```bash
|
||||
cd test
|
||||
INTEGRATION_TESTS=true go test -v -run TestAccessibilityToolsIntegration
|
||||
```
|
||||
|
||||
### Run Specific Sub-Test
|
||||
|
||||
```bash
|
||||
cd test
|
||||
INTEGRATION_TESTS=true go test -v -run TestAccessibilityToolsIntegration/AxeCoreIntegration
|
||||
```
|
||||
|
||||
### Run Benchmarks
|
||||
|
||||
```bash
|
||||
cd test
|
||||
INTEGRATION_TESTS=true go test -v -bench=. -benchmem
|
||||
```
|
||||
|
||||
### Run with Coverage
|
||||
|
||||
```bash
|
||||
cd test
|
||||
INTEGRATION_TESTS=true go test -v -cover -coverprofile=coverage.out
|
||||
go tool cover -html=coverage.out
|
||||
```
|
||||
|
||||
## Expected Results
|
||||
|
||||
### Accessible Page Tests
|
||||
|
||||
When testing `test-accessible.html`, expect:
|
||||
|
||||
- **Axe-Core**: Few or no violations (< 5)
|
||||
- **Contrast**: High AA pass rate (> 80%)
|
||||
- **Keyboard**: High focusability rate (> 80%)
|
||||
- **Zoom**: No horizontal scroll at 200%
|
||||
- **Reflow**: No horizontal scroll at 320px
|
||||
|
||||
### Inaccessible Page Tests
|
||||
|
||||
When testing `test-inaccessible.html`, expect:
|
||||
|
||||
- **Axe-Core**: Multiple violations detected
|
||||
- **Contrast**: Multiple AA failures
|
||||
- **Keyboard**: Multiple focusability issues
|
||||
- **Zoom**: Potential horizontal scroll issues
|
||||
- **Reflow**: Horizontal scroll at 320px
|
||||
|
||||
## Test Output Example
|
||||
|
||||
```
|
||||
=== RUN TestAccessibilityToolsIntegration
|
||||
=== RUN TestAccessibilityToolsIntegration/AxeCoreIntegration
|
||||
accessibility_integration_test.go:72: Axe-core results: 2 violations, 45 passes, 3 incomplete, 12 inapplicable
|
||||
=== RUN TestAccessibilityToolsIntegration/ContrastChecking
|
||||
accessibility_integration_test.go:98: Contrast check: 38 elements, 36 AA pass, 2 AA fail, 28 AAA pass, 10 AAA fail
|
||||
=== RUN TestAccessibilityToolsIntegration/KeyboardNavigation
|
||||
accessibility_integration_test.go:124: Keyboard test: 25 elements, 23 focusable, 2 not focusable, 1 missing focus indicator, 0 keyboard traps
|
||||
=== RUN TestAccessibilityToolsIntegration/ZoomTesting
|
||||
accessibility_integration_test.go:152: Zoom test: tested 3 levels, found 0 issues
|
||||
=== RUN TestAccessibilityToolsIntegration/ReflowTesting
|
||||
accessibility_integration_test.go:180: Reflow test: tested 2 breakpoints, found 0 issues
|
||||
=== RUN TestAccessibilityToolsIntegration/ScreenshotEnhancements
|
||||
accessibility_integration_test.go:208: Screenshot functionality verified
|
||||
=== RUN TestAccessibilityToolsIntegration/LibraryInjection
|
||||
accessibility_integration_test.go:230: Library injection functionality verified
|
||||
=== RUN TestAccessibilityToolsIntegration/AccessibilityTree
|
||||
accessibility_integration_test.go:248: Accessibility tree: 156 nodes
|
||||
--- PASS: TestAccessibilityToolsIntegration (15.23s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/AxeCoreIntegration (3.45s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/ContrastChecking (2.12s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/KeyboardNavigation (1.89s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/ZoomTesting (2.34s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/ReflowTesting (1.98s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/ScreenshotEnhancements (0.45s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/LibraryInjection (1.23s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/AccessibilityTree (1.77s)
|
||||
PASS
|
||||
ok git.teamworkapps.com/shortcut/cremote/test 15.234s
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Tests Skip with "Skipping integration tests"
|
||||
|
||||
**Problem:** Tests are being skipped.
|
||||
|
||||
**Solution:** Set the environment variable:
|
||||
```bash
|
||||
export INTEGRATION_TESTS=true
|
||||
```
|
||||
|
||||
### "Failed to navigate to test page"
|
||||
|
||||
**Problem:** Test HTTP server is not running or not accessible.
|
||||
|
||||
**Solution:**
|
||||
1. Start HTTP server on port 8080
|
||||
2. Verify server is accessible: `curl http://localhost:8080/test-accessible.html`
|
||||
3. Check firewall settings
|
||||
|
||||
### "Failed to connect to daemon"
|
||||
|
||||
**Problem:** Cremote daemon is not running.
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Start daemon
|
||||
cremote-daemon
|
||||
|
||||
# Verify it's running
|
||||
ps aux | grep cremote-daemon
|
||||
```
|
||||
|
||||
### "Failed to inject axe-core"
|
||||
|
||||
**Problem:** Network issues or CDN unavailable.
|
||||
|
||||
**Solution:**
|
||||
1. Check internet connectivity
|
||||
2. Try alternative axe-core version
|
||||
3. Check if CDN is accessible: `curl https://cdn.jsdelivr.net/npm/axe-core@4.8.0/axe.min.js`
|
||||
|
||||
### Tests Timeout
|
||||
|
||||
**Problem:** Tests are timing out.
|
||||
|
||||
**Solution:**
|
||||
1. Increase timeout values in test code
|
||||
2. Check system resources (CPU, memory)
|
||||
3. Verify Chromium is not hanging
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
name: Accessibility Integration Tests
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
||||
- name: Set up Go
|
||||
uses: actions/setup-go@v2
|
||||
with:
|
||||
go-version: 1.21
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y chromium-browser
|
||||
|
||||
- name: Start cremote daemon
|
||||
run: |
|
||||
./cremote-daemon &
|
||||
sleep 5
|
||||
|
||||
- name: Start test server
|
||||
run: |
|
||||
cd test/testdata
|
||||
python3 -m http.server 8080 &
|
||||
sleep 2
|
||||
|
||||
- name: Run integration tests
|
||||
run: |
|
||||
cd test
|
||||
INTEGRATION_TESTS=true go test -v -cover
|
||||
|
||||
- name: Upload coverage
|
||||
uses: codecov/codecov-action@v2
|
||||
with:
|
||||
files: ./test/coverage.out
|
||||
```
|
||||
|
||||
## Adding New Tests
|
||||
|
||||
### 1. Create Test Function
|
||||
|
||||
```go
|
||||
func testNewFeature(c *client.Client) func(*testing.T) {
|
||||
return func(t *testing.T) {
|
||||
// Test implementation
|
||||
result, err := c.NewFeature("", params, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed: %v", err)
|
||||
}
|
||||
|
||||
// Assertions
|
||||
if result == nil {
|
||||
t.Error("Result is nil")
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Add to Test Suite
|
||||
|
||||
```go
|
||||
func TestAccessibilityToolsIntegration(t *testing.T) {
|
||||
// ... existing setup ...
|
||||
|
||||
t.Run("NewFeature", testNewFeature(c))
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Update Documentation
|
||||
|
||||
Update this README with:
|
||||
- Tool description
|
||||
- Expected results
|
||||
- Common issues
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
Expected performance on modern hardware:
|
||||
|
||||
- **Axe-Core**: ~2-4 seconds per run
|
||||
- **Contrast Check**: ~1-2 seconds for 50 elements
|
||||
- **Keyboard Test**: ~1-2 seconds for 30 elements
|
||||
- **Zoom Test**: ~2-3 seconds for 3 zoom levels
|
||||
- **Reflow Test**: ~1-2 seconds for 2 breakpoints
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
- Check the main documentation: `docs/ADA_TESTING_GUIDE.md`
|
||||
- Review LLM guide: `docs/llm_ada_testing.md`
|
||||
- File an issue: git.teamworkapps.com/shortcut/cremote/issues
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2025-10-02
|
||||
**Version:** 1.0.0
|
||||
|
||||
Reference in New Issue
Block a user