cremote/test/README.md

Cremote Accessibility Integration Tests

Overview

This directory contains comprehensive integration tests for Cremote's accessibility testing tools. The tests validate all ADA/WCAG compliance features against known accessible and inaccessible test pages.

Test Coverage

Tools Tested

1. Axe-Core Integration (web_inject_axe_cremotemcp, web_run_axe_cremotemcp)

   • Library injection
   • WCAG 2.1 AA/AAA automated testing
   • Violation detection and reporting

2. Contrast Checking (web_contrast_check_cremotemcp)

   • WCAG contrast ratio calculations
   • AA/AAA compliance validation
   • Violation reporting with element details

3. Keyboard Navigation (web_keyboard_test_cremotemcp)

   • Focusability testing
   • Focus indicator validation
   • Keyboard trap detection
   • Tab order verification

4. Zoom Testing (web_zoom_test_cremotemcp)

   • Multi-level zoom testing (100%, 200%, 400%)
   • Horizontal scroll detection
   • Text readability validation
   • Overflow detection

5. Reflow Testing (web_reflow_test_cremotemcp)

   • Responsive breakpoint testing (320px, 1280px)
   • Horizontal scroll detection
   • Responsive layout validation
   • Overflow detection

6. Screenshot Enhancements (web_screenshot_cremotemcp)

   • Zoom level screenshots
   • Viewport size screenshots
   • Full page screenshots

7. Library Injection (console_command_cremotemcp)

   • External library injection (jQuery, Lodash, etc.)
   • Custom URL injection
   • Command execution with injected libraries

8. Accessibility Tree (get_accessibility_tree_cremotemcp)

   • Full tree retrieval
   • Depth limiting
   • Contrast annotation support

Test Files

Test Pages

• testdata/test-accessible.html - WCAG 2.1 AA compliant page

  • High contrast colors (4.5:1+ for normal text)
  • Visible focus indicators
  • Semantic HTML
  • Proper ARIA labels
  • Responsive design (no horizontal scroll at 320px)
  • Keyboard accessible controls
  • Proper heading hierarchy
  • Form labels and associations

• testdata/test-inaccessible.html - Page with known violations

  • Low contrast text (3.2:1, 1.5:1)
  • Missing focus indicators
  • Non-keyboard accessible elements
  • Missing alt text
  • Fixed width causing horizontal scroll
  • Missing form labels
  • Heading hierarchy violations
  • Duplicate IDs
  • Empty links and buttons
  • Incorrect ARIA usage

Test Suite

• accessibility_integration_test.go - Main test suite
  • TestAccessibilityToolsIntegration - Tests all tools on accessible page
  • TestInaccessiblePage - Tests violation detection on inaccessible page
  • BenchmarkAxeCore - Performance benchmarks

Prerequisites

1. Start Cremote Daemon

# Start the daemon on default port 9223
cremote-daemon

2. Start Test HTTP Server

You need a simple HTTP server to serve the test HTML files:

# Option 1: Python 3
cd test/testdata
python3 -m http.server 8080

# Option 2: Python 2
cd test/testdata
python -m SimpleHTTPServer 8080

# Option 3: Node.js (http-server)
cd test/testdata
npx http-server -p 8080

# Option 4: Go
cd test/testdata
go run -m http.server 8080

The test server should serve files at http://localhost:8080/.

3. Set Environment Variable

export INTEGRATION_TESTS=true

Running Tests

Run All Integration Tests

cd test
INTEGRATION_TESTS=true go test -v

Run Specific Test

cd test
INTEGRATION_TESTS=true go test -v -run TestAccessibilityToolsIntegration

Run Specific Sub-Test

cd test
INTEGRATION_TESTS=true go test -v -run TestAccessibilityToolsIntegration/AxeCoreIntegration

Run Benchmarks

cd test
INTEGRATION_TESTS=true go test -v -bench=. -benchmem

Run with Coverage

cd test
INTEGRATION_TESTS=true go test -v -cover -coverprofile=coverage.out
go tool cover -html=coverage.out

Expected Results

Accessible Page Tests

When testing test-accessible.html, expect:

• Axe-Core: Few or no violations (< 5)
• Contrast: High AA pass rate (> 80%)
• Keyboard: High focusability rate (> 80%)
• Zoom: No horizontal scroll at 200%
• Reflow: No horizontal scroll at 320px

Inaccessible Page Tests

When testing test-inaccessible.html, expect:

• Axe-Core: Multiple violations detected
• Contrast: Multiple AA failures
• Keyboard: Multiple focusability issues
• Zoom: Potential horizontal scroll issues
• Reflow: Horizontal scroll at 320px

Test Output Example

=== RUN   TestAccessibilityToolsIntegration
=== RUN   TestAccessibilityToolsIntegration/AxeCoreIntegration
    accessibility_integration_test.go:72: Axe-core results: 2 violations, 45 passes, 3 incomplete, 12 inapplicable
=== RUN   TestAccessibilityToolsIntegration/ContrastChecking
    accessibility_integration_test.go:98: Contrast check: 38 elements, 36 AA pass, 2 AA fail, 28 AAA pass, 10 AAA fail
=== RUN   TestAccessibilityToolsIntegration/KeyboardNavigation
    accessibility_integration_test.go:124: Keyboard test: 25 elements, 23 focusable, 2 not focusable, 1 missing focus indicator, 0 keyboard traps
=== RUN   TestAccessibilityToolsIntegration/ZoomTesting
    accessibility_integration_test.go:152: Zoom test: tested 3 levels, found 0 issues
=== RUN   TestAccessibilityToolsIntegration/ReflowTesting
    accessibility_integration_test.go:180: Reflow test: tested 2 breakpoints, found 0 issues
=== RUN   TestAccessibilityToolsIntegration/ScreenshotEnhancements
    accessibility_integration_test.go:208: Screenshot functionality verified
=== RUN   TestAccessibilityToolsIntegration/LibraryInjection
    accessibility_integration_test.go:230: Library injection functionality verified
=== RUN   TestAccessibilityToolsIntegration/AccessibilityTree
    accessibility_integration_test.go:248: Accessibility tree: 156 nodes
--- PASS: TestAccessibilityToolsIntegration (15.23s)
    --- PASS: TestAccessibilityToolsIntegration/AxeCoreIntegration (3.45s)
    --- PASS: TestAccessibilityToolsIntegration/ContrastChecking (2.12s)
    --- PASS: TestAccessibilityToolsIntegration/KeyboardNavigation (1.89s)
    --- PASS: TestAccessibilityToolsIntegration/ZoomTesting (2.34s)
    --- PASS: TestAccessibilityToolsIntegration/ReflowTesting (1.98s)
    --- PASS: TestAccessibilityToolsIntegration/ScreenshotEnhancements (0.45s)
    --- PASS: TestAccessibilityToolsIntegration/LibraryInjection (1.23s)
    --- PASS: TestAccessibilityToolsIntegration/AccessibilityTree (1.77s)
PASS
ok      git.teamworkapps.com/shortcut/cremote/test      15.234s

Troubleshooting

Tests Skip with "Skipping integration tests"

Problem: Tests are being skipped.

Solution: Set the environment variable:

export INTEGRATION_TESTS=true

"Failed to navigate to test page"

Problem: Test HTTP server is not running or not accessible.

Solution:

1. Start HTTP server on port 8080
2. Verify server is accessible: curl http://localhost:8080/test-accessible.html
3. Check firewall settings

"Failed to connect to daemon"

Problem: Cremote daemon is not running.

Solution:

# Start daemon
cremote-daemon

# Verify it's running
ps aux | grep cremote-daemon

"Failed to inject axe-core"

Problem: Network issues or CDN unavailable.

Solution:

1. Check internet connectivity
2. Try alternative axe-core version
3. Check if CDN is accessible: curl https://cdn.jsdelivr.net/npm/axe-core@4.8.0/axe.min.js

Tests Timeout

Problem: Tests are timing out.

Solution:

1. Increase timeout values in test code
2. Check system resources (CPU, memory)
3. Verify Chromium is not hanging

CI/CD Integration

GitHub Actions Example

name: Accessibility Integration Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      
      - name: Set up Go
        uses: actions/setup-go@v2
        with:
          go-version: 1.21
      
      - name: Install dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y chromium-browser
      
      - name: Start cremote daemon
        run: |
          ./cremote-daemon &
          sleep 5
      
      - name: Start test server
        run: |
          cd test/testdata
          python3 -m http.server 8080 &
          sleep 2
      
      - name: Run integration tests
        run: |
          cd test
          INTEGRATION_TESTS=true go test -v -cover
      
      - name: Upload coverage
        uses: codecov/codecov-action@v2
        with:
          files: ./test/coverage.out

Adding New Tests

1. Create Test Function

func testNewFeature(c *client.Client) func(*testing.T) {
    return func(t *testing.T) {
        // Test implementation
        result, err := c.NewFeature("", params, 10)
        if err != nil {
            t.Fatalf("Failed: %v", err)
        }
        
        // Assertions
        if result == nil {
            t.Error("Result is nil")
        }
    }
}

2. Add to Test Suite

func TestAccessibilityToolsIntegration(t *testing.T) {
    // ... existing setup ...
    
    t.Run("NewFeature", testNewFeature(c))
}

3. Update Documentation

Update this README with:

• Tool description
• Expected results
• Common issues

Performance Benchmarks

Expected performance on modern hardware:

• Axe-Core: ~2-4 seconds per run
• Contrast Check: ~1-2 seconds for 50 elements
• Keyboard Test: ~1-2 seconds for 30 elements
• Zoom Test: ~2-3 seconds for 3 zoom levels
• Reflow Test: ~1-2 seconds for 2 breakpoints

Support

For issues or questions:

• Check the main documentation: docs/ADA_TESTING_GUIDE.md
• Review LLM guide: docs/llm_ada_testing.md
• File an issue: git.teamworkapps.com/shortcut/cremote/issues

---

Last Updated: 2025-10-02 Version: 1.0.0