ada tools update
This commit is contained in:
375
test/README.md
Normal file
375
test/README.md
Normal file
@@ -0,0 +1,375 @@
|
||||
# Cremote Accessibility Integration Tests
|
||||
|
||||
## Overview
|
||||
|
||||
This directory contains comprehensive integration tests for Cremote's accessibility testing tools. The tests validate all ADA/WCAG compliance features against known accessible and inaccessible test pages.
|
||||
|
||||
## Test Coverage
|
||||
|
||||
### Tools Tested
|
||||
|
||||
1. **Axe-Core Integration** (`web_inject_axe_cremotemcp`, `web_run_axe_cremotemcp`)
|
||||
- Library injection
|
||||
- WCAG 2.1 AA/AAA automated testing
|
||||
- Violation detection and reporting
|
||||
|
||||
2. **Contrast Checking** (`web_contrast_check_cremotemcp`)
|
||||
- WCAG contrast ratio calculations
|
||||
- AA/AAA compliance validation
|
||||
- Violation reporting with element details
|
||||
|
||||
3. **Keyboard Navigation** (`web_keyboard_test_cremotemcp`)
|
||||
- Focusability testing
|
||||
- Focus indicator validation
|
||||
- Keyboard trap detection
|
||||
- Tab order verification
|
||||
|
||||
4. **Zoom Testing** (`web_zoom_test_cremotemcp`)
|
||||
- Multi-level zoom testing (100%, 200%, 400%)
|
||||
- Horizontal scroll detection
|
||||
- Text readability validation
|
||||
- Overflow detection
|
||||
|
||||
5. **Reflow Testing** (`web_reflow_test_cremotemcp`)
|
||||
- Responsive breakpoint testing (320px, 1280px)
|
||||
- Horizontal scroll detection
|
||||
- Responsive layout validation
|
||||
- Overflow detection
|
||||
|
||||
6. **Screenshot Enhancements** (`web_screenshot_cremotemcp`)
|
||||
- Zoom level screenshots
|
||||
- Viewport size screenshots
|
||||
- Full page screenshots
|
||||
|
||||
7. **Library Injection** (`console_command_cremotemcp`)
|
||||
- External library injection (jQuery, Lodash, etc.)
|
||||
- Custom URL injection
|
||||
- Command execution with injected libraries
|
||||
|
||||
8. **Accessibility Tree** (`get_accessibility_tree_cremotemcp`)
|
||||
- Full tree retrieval
|
||||
- Depth limiting
|
||||
- Contrast annotation support
|
||||
|
||||
## Test Files
|
||||
|
||||
### Test Pages
|
||||
|
||||
- **`testdata/test-accessible.html`** - WCAG 2.1 AA compliant page
|
||||
- High contrast colors (4.5:1+ for normal text)
|
||||
- Visible focus indicators
|
||||
- Semantic HTML
|
||||
- Proper ARIA labels
|
||||
- Responsive design (no horizontal scroll at 320px)
|
||||
- Keyboard accessible controls
|
||||
- Proper heading hierarchy
|
||||
- Form labels and associations
|
||||
|
||||
- **`testdata/test-inaccessible.html`** - Page with known violations
|
||||
- Low contrast text (3.2:1, 1.5:1)
|
||||
- Missing focus indicators
|
||||
- Non-keyboard accessible elements
|
||||
- Missing alt text
|
||||
- Fixed width causing horizontal scroll
|
||||
- Missing form labels
|
||||
- Heading hierarchy violations
|
||||
- Duplicate IDs
|
||||
- Empty links and buttons
|
||||
- Incorrect ARIA usage
|
||||
|
||||
### Test Suite
|
||||
|
||||
- **`accessibility_integration_test.go`** - Main test suite
|
||||
- `TestAccessibilityToolsIntegration` - Tests all tools on accessible page
|
||||
- `TestInaccessiblePage` - Tests violation detection on inaccessible page
|
||||
- `BenchmarkAxeCore` - Performance benchmarks
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### 1. Start Cremote Daemon
|
||||
|
||||
```bash
|
||||
# Start the daemon on default port 9223
|
||||
cremote-daemon
|
||||
```
|
||||
|
||||
### 2. Start Test HTTP Server
|
||||
|
||||
You need a simple HTTP server to serve the test HTML files:
|
||||
|
||||
```bash
|
||||
# Option 1: Python 3
|
||||
cd test/testdata
|
||||
python3 -m http.server 8080
|
||||
|
||||
# Option 2: Python 2
|
||||
cd test/testdata
|
||||
python -m SimpleHTTPServer 8080
|
||||
|
||||
# Option 3: Node.js (http-server)
|
||||
cd test/testdata
|
||||
npx http-server -p 8080
|
||||
|
||||
# Option 4: Go
|
||||
cd test/testdata
|
||||
go run -m http.server 8080
|
||||
```
|
||||
|
||||
The test server should serve files at `http://localhost:8080/`.
|
||||
|
||||
### 3. Set Environment Variable
|
||||
|
||||
```bash
|
||||
export INTEGRATION_TESTS=true
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Run All Integration Tests
|
||||
|
||||
```bash
|
||||
cd test
|
||||
INTEGRATION_TESTS=true go test -v
|
||||
```
|
||||
|
||||
### Run Specific Test
|
||||
|
||||
```bash
|
||||
cd test
|
||||
INTEGRATION_TESTS=true go test -v -run TestAccessibilityToolsIntegration
|
||||
```
|
||||
|
||||
### Run Specific Sub-Test
|
||||
|
||||
```bash
|
||||
cd test
|
||||
INTEGRATION_TESTS=true go test -v -run TestAccessibilityToolsIntegration/AxeCoreIntegration
|
||||
```
|
||||
|
||||
### Run Benchmarks
|
||||
|
||||
```bash
|
||||
cd test
|
||||
INTEGRATION_TESTS=true go test -v -bench=. -benchmem
|
||||
```
|
||||
|
||||
### Run with Coverage
|
||||
|
||||
```bash
|
||||
cd test
|
||||
INTEGRATION_TESTS=true go test -v -cover -coverprofile=coverage.out
|
||||
go tool cover -html=coverage.out
|
||||
```
|
||||
|
||||
## Expected Results
|
||||
|
||||
### Accessible Page Tests
|
||||
|
||||
When testing `test-accessible.html`, expect:
|
||||
|
||||
- **Axe-Core**: Few or no violations (< 5)
|
||||
- **Contrast**: High AA pass rate (> 80%)
|
||||
- **Keyboard**: High focusability rate (> 80%)
|
||||
- **Zoom**: No horizontal scroll at 200%
|
||||
- **Reflow**: No horizontal scroll at 320px
|
||||
|
||||
### Inaccessible Page Tests
|
||||
|
||||
When testing `test-inaccessible.html`, expect:
|
||||
|
||||
- **Axe-Core**: Multiple violations detected
|
||||
- **Contrast**: Multiple AA failures
|
||||
- **Keyboard**: Multiple focusability issues
|
||||
- **Zoom**: Potential horizontal scroll issues
|
||||
- **Reflow**: Horizontal scroll at 320px
|
||||
|
||||
## Test Output Example
|
||||
|
||||
```
|
||||
=== RUN TestAccessibilityToolsIntegration
|
||||
=== RUN TestAccessibilityToolsIntegration/AxeCoreIntegration
|
||||
accessibility_integration_test.go:72: Axe-core results: 2 violations, 45 passes, 3 incomplete, 12 inapplicable
|
||||
=== RUN TestAccessibilityToolsIntegration/ContrastChecking
|
||||
accessibility_integration_test.go:98: Contrast check: 38 elements, 36 AA pass, 2 AA fail, 28 AAA pass, 10 AAA fail
|
||||
=== RUN TestAccessibilityToolsIntegration/KeyboardNavigation
|
||||
accessibility_integration_test.go:124: Keyboard test: 25 elements, 23 focusable, 2 not focusable, 1 missing focus indicator, 0 keyboard traps
|
||||
=== RUN TestAccessibilityToolsIntegration/ZoomTesting
|
||||
accessibility_integration_test.go:152: Zoom test: tested 3 levels, found 0 issues
|
||||
=== RUN TestAccessibilityToolsIntegration/ReflowTesting
|
||||
accessibility_integration_test.go:180: Reflow test: tested 2 breakpoints, found 0 issues
|
||||
=== RUN TestAccessibilityToolsIntegration/ScreenshotEnhancements
|
||||
accessibility_integration_test.go:208: Screenshot functionality verified
|
||||
=== RUN TestAccessibilityToolsIntegration/LibraryInjection
|
||||
accessibility_integration_test.go:230: Library injection functionality verified
|
||||
=== RUN TestAccessibilityToolsIntegration/AccessibilityTree
|
||||
accessibility_integration_test.go:248: Accessibility tree: 156 nodes
|
||||
--- PASS: TestAccessibilityToolsIntegration (15.23s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/AxeCoreIntegration (3.45s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/ContrastChecking (2.12s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/KeyboardNavigation (1.89s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/ZoomTesting (2.34s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/ReflowTesting (1.98s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/ScreenshotEnhancements (0.45s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/LibraryInjection (1.23s)
|
||||
--- PASS: TestAccessibilityToolsIntegration/AccessibilityTree (1.77s)
|
||||
PASS
|
||||
ok git.teamworkapps.com/shortcut/cremote/test 15.234s
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Tests Skip with "Skipping integration tests"
|
||||
|
||||
**Problem:** Tests are being skipped.
|
||||
|
||||
**Solution:** Set the environment variable:
|
||||
```bash
|
||||
export INTEGRATION_TESTS=true
|
||||
```
|
||||
|
||||
### "Failed to navigate to test page"
|
||||
|
||||
**Problem:** Test HTTP server is not running or not accessible.
|
||||
|
||||
**Solution:**
|
||||
1. Start HTTP server on port 8080
|
||||
2. Verify server is accessible: `curl http://localhost:8080/test-accessible.html`
|
||||
3. Check firewall settings
|
||||
|
||||
### "Failed to connect to daemon"
|
||||
|
||||
**Problem:** Cremote daemon is not running.
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Start daemon
|
||||
cremote-daemon
|
||||
|
||||
# Verify it's running
|
||||
ps aux | grep cremote-daemon
|
||||
```
|
||||
|
||||
### "Failed to inject axe-core"
|
||||
|
||||
**Problem:** Network issues or CDN unavailable.
|
||||
|
||||
**Solution:**
|
||||
1. Check internet connectivity
|
||||
2. Try alternative axe-core version
|
||||
3. Check if CDN is accessible: `curl https://cdn.jsdelivr.net/npm/axe-core@4.8.0/axe.min.js`
|
||||
|
||||
### Tests Timeout
|
||||
|
||||
**Problem:** Tests are timing out.
|
||||
|
||||
**Solution:**
|
||||
1. Increase timeout values in test code
|
||||
2. Check system resources (CPU, memory)
|
||||
3. Verify Chromium is not hanging
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
name: Accessibility Integration Tests
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
||||
- name: Set up Go
|
||||
uses: actions/setup-go@v2
|
||||
with:
|
||||
go-version: 1.21
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y chromium-browser
|
||||
|
||||
- name: Start cremote daemon
|
||||
run: |
|
||||
./cremote-daemon &
|
||||
sleep 5
|
||||
|
||||
- name: Start test server
|
||||
run: |
|
||||
cd test/testdata
|
||||
python3 -m http.server 8080 &
|
||||
sleep 2
|
||||
|
||||
- name: Run integration tests
|
||||
run: |
|
||||
cd test
|
||||
INTEGRATION_TESTS=true go test -v -cover
|
||||
|
||||
- name: Upload coverage
|
||||
uses: codecov/codecov-action@v2
|
||||
with:
|
||||
files: ./test/coverage.out
|
||||
```
|
||||
|
||||
## Adding New Tests
|
||||
|
||||
### 1. Create Test Function
|
||||
|
||||
```go
|
||||
func testNewFeature(c *client.Client) func(*testing.T) {
|
||||
return func(t *testing.T) {
|
||||
// Test implementation
|
||||
result, err := c.NewFeature("", params, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed: %v", err)
|
||||
}
|
||||
|
||||
// Assertions
|
||||
if result == nil {
|
||||
t.Error("Result is nil")
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Add to Test Suite
|
||||
|
||||
```go
|
||||
func TestAccessibilityToolsIntegration(t *testing.T) {
|
||||
// ... existing setup ...
|
||||
|
||||
t.Run("NewFeature", testNewFeature(c))
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Update Documentation
|
||||
|
||||
Update this README with:
|
||||
- Tool description
|
||||
- Expected results
|
||||
- Common issues
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
Expected performance on modern hardware:
|
||||
|
||||
- **Axe-Core**: ~2-4 seconds per run
|
||||
- **Contrast Check**: ~1-2 seconds for 50 elements
|
||||
- **Keyboard Test**: ~1-2 seconds for 30 elements
|
||||
- **Zoom Test**: ~2-3 seconds for 3 zoom levels
|
||||
- **Reflow Test**: ~1-2 seconds for 2 breakpoints
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
- Check the main documentation: `docs/ADA_TESTING_GUIDE.md`
|
||||
- Review LLM guide: `docs/llm_ada_testing.md`
|
||||
- File an issue: git.teamworkapps.com/shortcut/cremote/issues
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2025-10-02
|
||||
**Version:** 1.0.0
|
||||
|
||||
422
test/accessibility_integration_test.go
Normal file
422
test/accessibility_integration_test.go
Normal file
@@ -0,0 +1,422 @@
|
||||
package test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.teamworkapps.com/shortcut/cremote/client"
|
||||
)
|
||||
|
||||
const (
|
||||
testServerURL = "http://localhost:8080"
|
||||
daemonAddr = "localhost:9223"
|
||||
)
|
||||
|
||||
// TestAccessibilityToolsIntegration runs comprehensive integration tests for all accessibility tools
|
||||
func TestAccessibilityToolsIntegration(t *testing.T) {
|
||||
// Skip if not in integration test mode
|
||||
if os.Getenv("INTEGRATION_TESTS") != "true" {
|
||||
t.Skip("Skipping integration tests. Set INTEGRATION_TESTS=true to run.")
|
||||
}
|
||||
|
||||
// Create client
|
||||
c := client.NewClient(daemonAddr)
|
||||
|
||||
// Wait for daemon to be ready
|
||||
time.Sleep(2 * time.Second)
|
||||
|
||||
// Navigate to test page
|
||||
err := c.Navigate("", testServerURL+"/test-accessible.html", false, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to navigate to test page: %v", err)
|
||||
}
|
||||
|
||||
// Wait for page load
|
||||
time.Sleep(2 * time.Second)
|
||||
|
||||
// Run all accessibility tests
|
||||
t.Run("AxeCoreIntegration", testAxeCoreIntegration(c))
|
||||
t.Run("ContrastChecking", testContrastChecking(c))
|
||||
t.Run("KeyboardNavigation", testKeyboardNavigation(c))
|
||||
t.Run("ZoomTesting", testZoomTesting(c))
|
||||
t.Run("ReflowTesting", testReflowTesting(c))
|
||||
t.Run("ScreenshotEnhancements", testScreenshotEnhancements(c))
|
||||
t.Run("LibraryInjection", testLibraryInjection(c))
|
||||
t.Run("AccessibilityTree", testAccessibilityTree(c))
|
||||
}
|
||||
|
||||
// testAxeCoreIntegration tests axe-core injection and execution
|
||||
func testAxeCoreIntegration(c *client.Client) func(*testing.T) {
|
||||
return func(t *testing.T) {
|
||||
// Inject axe-core
|
||||
err := c.InjectAxeCore("", "4.8.0", 30)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to inject axe-core: %v", err)
|
||||
}
|
||||
|
||||
// Run axe tests
|
||||
runOnly := []string{"wcag2a", "wcag2aa"}
|
||||
result, err := c.RunAxeCore("", runOnly, nil, 30)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to run axe-core: %v", err)
|
||||
}
|
||||
|
||||
// Verify result structure
|
||||
if result == nil {
|
||||
t.Fatal("Axe result is nil")
|
||||
}
|
||||
|
||||
// Check that we got some results
|
||||
totalChecks := len(result.Violations) + len(result.Passes) + len(result.Incomplete) + len(result.Inapplicable)
|
||||
if totalChecks == 0 {
|
||||
t.Error("Axe-core returned no results")
|
||||
}
|
||||
|
||||
t.Logf("Axe-core results: %d violations, %d passes, %d incomplete, %d inapplicable",
|
||||
len(result.Violations), len(result.Passes), len(result.Incomplete), len(result.Inapplicable))
|
||||
|
||||
// For accessible page, we expect few or no violations
|
||||
if len(result.Violations) > 5 {
|
||||
t.Errorf("Expected few violations on accessible page, got %d", len(result.Violations))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// testContrastChecking tests color contrast checking
|
||||
func testContrastChecking(c *client.Client) func(*testing.T) {
|
||||
return func(t *testing.T) {
|
||||
result, err := c.CheckContrast("", "body", 10)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to check contrast: %v", err)
|
||||
}
|
||||
|
||||
if result == nil {
|
||||
t.Fatal("Contrast result is nil")
|
||||
}
|
||||
|
||||
// Verify result structure
|
||||
if result.TotalElements == 0 {
|
||||
t.Error("No elements checked for contrast")
|
||||
}
|
||||
|
||||
t.Logf("Contrast check: %d elements, %d AA pass, %d AA fail, %d AAA pass, %d AAA fail",
|
||||
result.TotalElements, result.WCAGAAPass, result.WCAGAAFail, result.WCAGAAAPass, result.WCAGAAAFail)
|
||||
|
||||
// For accessible page, we expect high AA pass rate
|
||||
if result.TotalElements > 0 {
|
||||
passRate := float64(result.WCAGAAPass) / float64(result.TotalElements)
|
||||
if passRate < 0.8 {
|
||||
t.Errorf("Expected high AA pass rate on accessible page, got %.2f%%", passRate*100)
|
||||
}
|
||||
}
|
||||
|
||||
// Verify violation details
|
||||
for _, violation := range result.Violations {
|
||||
if violation.Ratio == 0 {
|
||||
t.Error("Violation has zero contrast ratio")
|
||||
}
|
||||
if violation.Element == "" {
|
||||
t.Error("Violation missing element selector")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// testKeyboardNavigation tests keyboard navigation testing
|
||||
func testKeyboardNavigation(c *client.Client) func(*testing.T) {
|
||||
return func(t *testing.T) {
|
||||
result, err := c.TestKeyboard("", 10)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to test keyboard navigation: %v", err)
|
||||
}
|
||||
|
||||
if result == nil {
|
||||
t.Fatal("Keyboard test result is nil")
|
||||
}
|
||||
|
||||
// Verify result structure
|
||||
if result.TotalElements == 0 {
|
||||
t.Error("No interactive elements found")
|
||||
}
|
||||
|
||||
t.Logf("Keyboard test: %d elements, %d focusable, %d not focusable, %d missing focus indicator, %d keyboard traps",
|
||||
result.TotalElements, result.Focusable, result.NotFocusable, result.MissingFocusIndicator, result.KeyboardTraps)
|
||||
|
||||
// For accessible page, we expect high focusability rate
|
||||
if result.TotalElements > 0 {
|
||||
focusableRate := float64(result.Focusable) / float64(result.TotalElements)
|
||||
if focusableRate < 0.8 {
|
||||
t.Errorf("Expected high focusability rate on accessible page, got %.2f%%", focusableRate*100)
|
||||
}
|
||||
}
|
||||
|
||||
// Verify tab order
|
||||
if len(result.TabOrder) == 0 {
|
||||
t.Error("Tab order is empty")
|
||||
}
|
||||
|
||||
// Verify issue details
|
||||
for _, issue := range result.Issues {
|
||||
if issue.Type == "" {
|
||||
t.Error("Issue missing type")
|
||||
}
|
||||
if issue.Element == "" {
|
||||
t.Error("Issue missing element selector")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// testZoomTesting tests zoom functionality testing
|
||||
func testZoomTesting(c *client.Client) func(*testing.T) {
|
||||
return func(t *testing.T) {
|
||||
zoomLevels := []float64{1.0, 2.0, 4.0}
|
||||
result, err := c.TestZoom("", zoomLevels, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to test zoom: %v", err)
|
||||
}
|
||||
|
||||
if result == nil {
|
||||
t.Fatal("Zoom test result is nil")
|
||||
}
|
||||
|
||||
// Verify we tested all zoom levels
|
||||
if len(result.ZoomLevels) != len(zoomLevels) {
|
||||
t.Errorf("Expected %d zoom levels, got %d", len(zoomLevels), len(result.ZoomLevels))
|
||||
}
|
||||
|
||||
t.Logf("Zoom test: tested %d levels, found %d issues", len(result.ZoomLevels), len(result.Issues))
|
||||
|
||||
// Verify zoom level details
|
||||
for _, zoomTest := range result.ZoomLevels {
|
||||
if zoomTest.ZoomLevel == 0 {
|
||||
t.Error("Zoom level is zero")
|
||||
}
|
||||
if zoomTest.ViewportWidth == 0 || zoomTest.ViewportHeight == 0 {
|
||||
t.Error("Viewport dimensions are zero")
|
||||
}
|
||||
|
||||
// For accessible page at 200%, we expect no horizontal scroll
|
||||
if zoomTest.ZoomLevel == 2.0 && zoomTest.HasHorizontalScroll {
|
||||
t.Error("Expected no horizontal scroll at 200% zoom on accessible page")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// testReflowTesting tests responsive reflow testing
|
||||
func testReflowTesting(c *client.Client) func(*testing.T) {
|
||||
return func(t *testing.T) {
|
||||
widths := []int{320, 1280}
|
||||
result, err := c.TestReflow("", widths, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to test reflow: %v", err)
|
||||
}
|
||||
|
||||
if result == nil {
|
||||
t.Fatal("Reflow test result is nil")
|
||||
}
|
||||
|
||||
// Verify we tested all widths
|
||||
if len(result.Breakpoints) != len(widths) {
|
||||
t.Errorf("Expected %d breakpoints, got %d", len(widths), len(result.Breakpoints))
|
||||
}
|
||||
|
||||
t.Logf("Reflow test: tested %d breakpoints, found %d issues", len(result.Breakpoints), len(result.Issues))
|
||||
|
||||
// Verify breakpoint details
|
||||
for _, breakpoint := range result.Breakpoints {
|
||||
if breakpoint.Width == 0 {
|
||||
t.Error("Breakpoint width is zero")
|
||||
}
|
||||
if breakpoint.Height == 0 {
|
||||
t.Error("Breakpoint height is zero")
|
||||
}
|
||||
|
||||
// For accessible page at 320px, we expect no horizontal scroll
|
||||
if breakpoint.Width == 320 && breakpoint.HasHorizontalScroll {
|
||||
t.Error("Expected no horizontal scroll at 320px on accessible page")
|
||||
}
|
||||
|
||||
// Verify responsive layout
|
||||
if !breakpoint.ResponsiveLayout {
|
||||
t.Errorf("Expected responsive layout at %dpx", breakpoint.Width)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// testScreenshotEnhancements tests enhanced screenshot functionality
|
||||
func testScreenshotEnhancements(c *client.Client) func(*testing.T) {
|
||||
return func(t *testing.T) {
|
||||
// Test basic screenshot
|
||||
err := c.TakeScreenshot("", "/tmp/test-screenshot.png", false, 5)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to take screenshot: %v", err)
|
||||
}
|
||||
|
||||
// Verify file exists
|
||||
if _, err := os.Stat("/tmp/test-screenshot.png"); os.IsNotExist(err) {
|
||||
t.Error("Screenshot file was not created")
|
||||
}
|
||||
|
||||
// Clean up
|
||||
os.Remove("/tmp/test-screenshot.png")
|
||||
|
||||
t.Log("Screenshot functionality verified")
|
||||
}
|
||||
}
|
||||
|
||||
// testLibraryInjection tests library injection via console_command
|
||||
func testLibraryInjection(c *client.Client) func(*testing.T) {
|
||||
return func(t *testing.T) {
|
||||
// Test injecting jQuery
|
||||
params := map[string]string{
|
||||
"command": "typeof jQuery",
|
||||
"inject_library": "jquery",
|
||||
"timeout": "10",
|
||||
}
|
||||
|
||||
resp, err := c.SendCommand("console-command", params)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to execute console command with library injection: %v", err)
|
||||
}
|
||||
|
||||
if !resp.Success {
|
||||
t.Fatalf("Console command failed: %s", resp.Error)
|
||||
}
|
||||
|
||||
// Verify jQuery was injected
|
||||
if resp.Data == nil {
|
||||
t.Error("Console command returned no data")
|
||||
}
|
||||
|
||||
t.Log("Library injection functionality verified")
|
||||
}
|
||||
}
|
||||
|
||||
// testAccessibilityTree tests accessibility tree retrieval
|
||||
func testAccessibilityTree(c *client.Client) func(*testing.T) {
|
||||
return func(t *testing.T) {
|
||||
depth := 3
|
||||
result, err := c.GetAccessibilityTree("", &depth, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to get accessibility tree: %v", err)
|
||||
}
|
||||
|
||||
if result == nil {
|
||||
t.Fatal("Accessibility tree result is nil")
|
||||
}
|
||||
|
||||
// Verify we got nodes
|
||||
if len(result.Nodes) == 0 {
|
||||
t.Error("Accessibility tree has no nodes")
|
||||
}
|
||||
|
||||
t.Logf("Accessibility tree: %d nodes", len(result.Nodes))
|
||||
|
||||
// Verify node structure
|
||||
foundRootNode := false
|
||||
for _, node := range result.Nodes {
|
||||
if node.NodeID == "" {
|
||||
t.Error("Node missing NodeID")
|
||||
}
|
||||
|
||||
// Check for root node
|
||||
if node.Role != nil && node.Role.Value == "WebArea" {
|
||||
foundRootNode = true
|
||||
}
|
||||
}
|
||||
|
||||
if !foundRootNode {
|
||||
t.Error("No root node found in accessibility tree")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestInaccessiblePage tests tools against a page with known accessibility issues
|
||||
func TestInaccessiblePage(t *testing.T) {
|
||||
// Skip if not in integration test mode
|
||||
if os.Getenv("INTEGRATION_TESTS") != "true" {
|
||||
t.Skip("Skipping integration tests. Set INTEGRATION_TESTS=true to run.")
|
||||
}
|
||||
|
||||
c := client.NewClient(daemonAddr)
|
||||
|
||||
// Navigate to inaccessible test page
|
||||
err := c.Navigate("", testServerURL+"/test-inaccessible.html", false, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to navigate to test page: %v", err)
|
||||
}
|
||||
|
||||
time.Sleep(2 * time.Second)
|
||||
|
||||
// Inject and run axe-core
|
||||
err = c.InjectAxeCore("", "4.8.0", 30)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to inject axe-core: %v", err)
|
||||
}
|
||||
|
||||
runOnly := []string{"wcag2a", "wcag2aa"}
|
||||
result, err := c.RunAxeCore("", runOnly, nil, 30)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to run axe-core: %v", err)
|
||||
}
|
||||
|
||||
// For inaccessible page, we expect violations
|
||||
if len(result.Violations) == 0 {
|
||||
t.Error("Expected violations on inaccessible page, got none")
|
||||
}
|
||||
|
||||
t.Logf("Found %d violations on inaccessible page (expected)", len(result.Violations))
|
||||
|
||||
// Test contrast checking - expect failures
|
||||
contrastResult, err := c.CheckContrast("", "body", 10)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to check contrast: %v", err)
|
||||
}
|
||||
|
||||
if contrastResult.WCAGAAFail == 0 {
|
||||
t.Error("Expected contrast failures on inaccessible page, got none")
|
||||
}
|
||||
|
||||
t.Logf("Found %d contrast failures on inaccessible page (expected)", contrastResult.WCAGAAFail)
|
||||
}
|
||||
|
||||
// BenchmarkAxeCore benchmarks axe-core execution
|
||||
func BenchmarkAxeCore(b *testing.B) {
|
||||
if os.Getenv("INTEGRATION_TESTS") != "true" {
|
||||
b.Skip("Skipping integration tests. Set INTEGRATION_TESTS=true to run.")
|
||||
}
|
||||
|
||||
c := client.NewClient(daemonAddr)
|
||||
|
||||
// Navigate once
|
||||
c.Navigate("", testServerURL+"/test-accessible.html", false, 10)
|
||||
time.Sleep(2 * time.Second)
|
||||
|
||||
// Inject once
|
||||
c.InjectAxeCore("", "4.8.0", 30)
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
runOnly := []string{"wcag2aa"}
|
||||
_, err := c.RunAxeCore("", runOnly, nil, 30)
|
||||
if err != nil {
|
||||
b.Fatalf("Failed to run axe-core: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Helper function to pretty print JSON
|
||||
func prettyPrint(v interface{}) string {
|
||||
b, err := json.MarshalIndent(v, "", " ")
|
||||
if err != nil {
|
||||
return fmt.Sprintf("%+v", v)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
282
test/testdata/test-accessible.html
vendored
Normal file
282
test/testdata/test-accessible.html
vendored
Normal file
@@ -0,0 +1,282 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Accessible Test Page</title>
|
||||
<style>
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: Arial, sans-serif;
|
||||
line-height: 1.6;
|
||||
max-width: 1200px;
|
||||
margin: 0 auto;
|
||||
padding: 20px;
|
||||
background-color: #ffffff;
|
||||
color: #000000;
|
||||
}
|
||||
|
||||
h1, h2, h3 {
|
||||
color: #000000;
|
||||
}
|
||||
|
||||
/* High contrast text */
|
||||
.high-contrast {
|
||||
color: #000000;
|
||||
background-color: #ffffff;
|
||||
}
|
||||
|
||||
/* Visible focus indicators */
|
||||
a:focus, button:focus, input:focus, select:focus, textarea:focus {
|
||||
outline: 3px solid #0066cc;
|
||||
outline-offset: 2px;
|
||||
}
|
||||
|
||||
/* Responsive layout */
|
||||
.container {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 20px;
|
||||
}
|
||||
|
||||
.card {
|
||||
flex: 1 1 300px;
|
||||
padding: 20px;
|
||||
border: 1px solid #000000;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
/* Form styles */
|
||||
label {
|
||||
display: block;
|
||||
margin-bottom: 5px;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
input, select, textarea {
|
||||
width: 100%;
|
||||
padding: 8px;
|
||||
margin-bottom: 15px;
|
||||
border: 1px solid #000000;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
button {
|
||||
padding: 10px 20px;
|
||||
background-color: #0066cc;
|
||||
color: #ffffff;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
cursor: pointer;
|
||||
font-size: 16px;
|
||||
}
|
||||
|
||||
button:hover {
|
||||
background-color: #0052a3;
|
||||
}
|
||||
|
||||
/* Navigation */
|
||||
nav {
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
nav ul {
|
||||
list-style: none;
|
||||
padding: 0;
|
||||
display: flex;
|
||||
gap: 20px;
|
||||
}
|
||||
|
||||
nav a {
|
||||
color: #0066cc;
|
||||
text-decoration: none;
|
||||
padding: 5px 10px;
|
||||
}
|
||||
|
||||
nav a:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
/* Skip link */
|
||||
.skip-link {
|
||||
position: absolute;
|
||||
top: -40px;
|
||||
left: 0;
|
||||
background: #000000;
|
||||
color: #ffffff;
|
||||
padding: 8px;
|
||||
text-decoration: none;
|
||||
z-index: 100;
|
||||
}
|
||||
|
||||
.skip-link:focus {
|
||||
top: 0;
|
||||
}
|
||||
|
||||
/* Responsive images */
|
||||
img {
|
||||
max-width: 100%;
|
||||
height: auto;
|
||||
}
|
||||
|
||||
/* Media queries for responsive design */
|
||||
@media (max-width: 768px) {
|
||||
.container {
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
nav ul {
|
||||
flex-direction: column;
|
||||
gap: 10px;
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 320px) {
|
||||
body {
|
||||
padding: 10px;
|
||||
}
|
||||
|
||||
.card {
|
||||
padding: 10px;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<a href="#main-content" class="skip-link">Skip to main content</a>
|
||||
|
||||
<header>
|
||||
<h1>Accessible Test Page</h1>
|
||||
<nav aria-label="Main navigation">
|
||||
<ul>
|
||||
<li><a href="#home">Home</a></li>
|
||||
<li><a href="#about">About</a></li>
|
||||
<li><a href="#services">Services</a></li>
|
||||
<li><a href="#contact">Contact</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<main id="main-content">
|
||||
<section>
|
||||
<h2>Welcome</h2>
|
||||
<p class="high-contrast">
|
||||
This is an accessible test page designed to pass WCAG 2.1 Level AA criteria.
|
||||
It includes proper semantic HTML, high contrast colors, keyboard navigation,
|
||||
and responsive design.
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Features</h2>
|
||||
<div class="container">
|
||||
<div class="card">
|
||||
<h3>High Contrast</h3>
|
||||
<p>All text meets WCAG AA contrast requirements (4.5:1 for normal text, 3:1 for large text).</p>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Keyboard Navigation</h3>
|
||||
<p>All interactive elements are keyboard accessible with visible focus indicators.</p>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Responsive Design</h3>
|
||||
<p>Content reflows properly at 320px width without horizontal scrolling.</p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Interactive Elements</h2>
|
||||
<button type="button">Click Me</button>
|
||||
<a href="#example">Example Link</a>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Contact Form</h2>
|
||||
<form>
|
||||
<div>
|
||||
<label for="name">Name:</label>
|
||||
<input type="text" id="name" name="name" required aria-required="true">
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<label for="email">Email:</label>
|
||||
<input type="email" id="email" name="email" required aria-required="true">
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<label for="subject">Subject:</label>
|
||||
<select id="subject" name="subject">
|
||||
<option value="">Select a subject</option>
|
||||
<option value="general">General Inquiry</option>
|
||||
<option value="support">Support</option>
|
||||
<option value="feedback">Feedback</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<label for="message">Message:</label>
|
||||
<textarea id="message" name="message" rows="5" required aria-required="true"></textarea>
|
||||
</div>
|
||||
|
||||
<button type="submit">Submit</button>
|
||||
</form>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Images</h2>
|
||||
<img src="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='400' height='300'%3E%3Crect fill='%230066cc' width='400' height='300'/%3E%3Ctext x='50%25' y='50%25' dominant-baseline='middle' text-anchor='middle' fill='white' font-size='24'%3EAccessible Image%3C/text%3E%3C/svg%3E"
|
||||
alt="Accessible image with descriptive alt text">
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Lists</h2>
|
||||
<ul>
|
||||
<li>Semantic HTML elements</li>
|
||||
<li>Proper heading hierarchy</li>
|
||||
<li>ARIA labels where appropriate</li>
|
||||
<li>Keyboard accessible controls</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2>Table</h2>
|
||||
<table>
|
||||
<caption>WCAG 2.1 Compliance Summary</caption>
|
||||
<thead>
|
||||
<tr>
|
||||
<th scope="col">Criterion</th>
|
||||
<th scope="col">Level</th>
|
||||
<th scope="col">Status</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>1.4.3 Contrast (Minimum)</td>
|
||||
<td>AA</td>
|
||||
<td>Pass</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>2.1.1 Keyboard</td>
|
||||
<td>A</td>
|
||||
<td>Pass</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>1.4.10 Reflow</td>
|
||||
<td>AA</td>
|
||||
<td>Pass</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
<footer>
|
||||
<p>© 2025 Accessible Test Page. All rights reserved.</p>
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
178
test/testdata/test-inaccessible.html
vendored
Normal file
178
test/testdata/test-inaccessible.html
vendored
Normal file
@@ -0,0 +1,178 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<title>Inaccessible Test Page</title>
|
||||
<style>
|
||||
body {
|
||||
font-family: Arial, sans-serif;
|
||||
padding: 20px;
|
||||
background-color: #ffffff;
|
||||
width: 1400px; /* Fixed width causes horizontal scroll at small sizes */
|
||||
}
|
||||
|
||||
/* Low contrast text - WCAG violation */
|
||||
.low-contrast {
|
||||
color: #999999;
|
||||
background-color: #ffffff;
|
||||
}
|
||||
|
||||
/* Very low contrast - WCAG violation */
|
||||
.very-low-contrast {
|
||||
color: #cccccc;
|
||||
background-color: #ffffff;
|
||||
}
|
||||
|
||||
/* No focus indicators - WCAG violation */
|
||||
a:focus, button:focus, input:focus {
|
||||
outline: none;
|
||||
}
|
||||
|
||||
/* Clickable div without keyboard access - WCAG violation */
|
||||
.fake-button {
|
||||
padding: 10px 20px;
|
||||
background-color: #0066cc;
|
||||
color: #ffffff;
|
||||
cursor: pointer;
|
||||
display: inline-block;
|
||||
}
|
||||
|
||||
/* Fixed font sizes prevent zoom - WCAG violation */
|
||||
.fixed-size {
|
||||
font-size: 12px !important;
|
||||
}
|
||||
|
||||
/* Form without labels - WCAG violation */
|
||||
input {
|
||||
margin: 10px 0;
|
||||
padding: 5px;
|
||||
}
|
||||
|
||||
/* Image without alt text will be added in HTML */
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<!-- Missing lang attribute on html element - WCAG violation -->
|
||||
<!-- Missing skip link - WCAG violation -->
|
||||
|
||||
<h1>Inaccessible Test Page</h1>
|
||||
|
||||
<!-- Low contrast text - WCAG 1.4.3 violation -->
|
||||
<p class="low-contrast">
|
||||
This text has insufficient contrast ratio (3.2:1) and fails WCAG AA requirements.
|
||||
</p>
|
||||
|
||||
<p class="very-low-contrast">
|
||||
This text has very low contrast (1.5:1) and is barely readable.
|
||||
</p>
|
||||
|
||||
<!-- Clickable div without keyboard access - WCAG 2.1.1 violation -->
|
||||
<div class="fake-button" onclick="alert('Clicked')">
|
||||
Click Me (Not Keyboard Accessible)
|
||||
</div>
|
||||
|
||||
<!-- Link without focus indicator - WCAG 2.4.7 violation -->
|
||||
<a href="#nowhere">Link Without Focus Indicator</a>
|
||||
|
||||
<!-- Form without labels - WCAG 1.3.1, 3.3.2 violations -->
|
||||
<form>
|
||||
<input type="text" placeholder="Name">
|
||||
<input type="email" placeholder="Email">
|
||||
<input type="submit" value="Submit">
|
||||
</form>
|
||||
|
||||
<!-- Image without alt text - WCAG 1.1.1 violation -->
|
||||
<img src="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='200' height='150'%3E%3Crect fill='%23ff0000' width='200' height='150'/%3E%3C/svg%3E">
|
||||
|
||||
<!-- Empty link - WCAG 2.4.4 violation -->
|
||||
<a href="#"></a>
|
||||
|
||||
<!-- Button with no accessible name - WCAG 4.1.2 violation -->
|
||||
<button></button>
|
||||
|
||||
<!-- Heading hierarchy skip - WCAG 1.3.1 violation -->
|
||||
<h1>Heading 1</h1>
|
||||
<h3>Heading 3 (skipped h2)</h3>
|
||||
|
||||
<!-- Table without headers - WCAG 1.3.1 violation -->
|
||||
<table>
|
||||
<tr>
|
||||
<td>Name</td>
|
||||
<td>Email</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>John Doe</td>
|
||||
<td>john@example.com</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<!-- Fixed width content causes horizontal scroll at 320px - WCAG 1.4.10 violation -->
|
||||
<div style="width: 1200px; background-color: #f0f0f0; padding: 20px;">
|
||||
This content has a fixed width and will cause horizontal scrolling on small screens.
|
||||
</div>
|
||||
|
||||
<!-- Text with fixed pixel size prevents zoom - WCAG 1.4.4 violation -->
|
||||
<p class="fixed-size">
|
||||
This text has a fixed pixel size and may not resize properly when zoomed.
|
||||
</p>
|
||||
|
||||
<!-- Keyboard trap (simulated with tabindex) -->
|
||||
<div tabindex="0" onkeydown="event.preventDefault()">
|
||||
This element traps keyboard focus
|
||||
</div>
|
||||
|
||||
<!-- Color used as only visual means - WCAG 1.4.1 violation -->
|
||||
<p>
|
||||
<span style="color: red;">Required fields</span> must be filled out.
|
||||
</p>
|
||||
|
||||
<!-- Insufficient color contrast on interactive element -->
|
||||
<button style="background-color: #ffff00; color: #ffffff; border: none; padding: 10px;">
|
||||
Low Contrast Button
|
||||
</button>
|
||||
|
||||
<!-- Missing form field association -->
|
||||
<div>
|
||||
<span>Username</span>
|
||||
<input type="text">
|
||||
</div>
|
||||
|
||||
<!-- Redundant link text - WCAG 2.4.4 violation -->
|
||||
<a href="#link1">Click here</a>
|
||||
<a href="#link2">Click here</a>
|
||||
<a href="#link3">Click here</a>
|
||||
|
||||
<!-- Empty heading - WCAG 2.4.6 violation -->
|
||||
<h2></h2>
|
||||
|
||||
<!-- Iframe without title - WCAG 4.1.2 violation -->
|
||||
<iframe src="about:blank"></iframe>
|
||||
|
||||
<!-- Duplicate ID - WCAG 4.1.1 violation -->
|
||||
<div id="duplicate">First</div>
|
||||
<div id="duplicate">Second</div>
|
||||
|
||||
<!-- Missing required ARIA attributes -->
|
||||
<div role="button">Button Without Tabindex</div>
|
||||
|
||||
<!-- Incorrect ARIA usage -->
|
||||
<div role="heading">Not Actually a Heading</div>
|
||||
|
||||
<!-- Text that's too small -->
|
||||
<p style="font-size: 8px;">This text is too small to read comfortably.</p>
|
||||
|
||||
<!-- Link that opens in new window without warning - WCAG 3.2.2 violation -->
|
||||
<a href="https://example.com" target="_blank">Opens in New Window</a>
|
||||
|
||||
<!-- Form with no submit button -->
|
||||
<form>
|
||||
<input type="text" placeholder="Search">
|
||||
</form>
|
||||
|
||||
<!-- Content that requires horizontal scrolling -->
|
||||
<div style="overflow-x: scroll; white-space: nowrap;">
|
||||
<p>This content requires horizontal scrolling which violates WCAG 1.4.10 at narrow viewports.</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
Reference in New Issue
Block a user