ada tools update
This commit is contained in:
Josh at WLTechBlog
442
docs/llm_ada_testing.md
Normal file
442
docs/llm_ada_testing.md
Normal file
@@ -0,0 +1,442 @@
|
||||
# LLM Agent Guide: ADA/WCAG Accessibility Testing with Cremote
|
||||
|
||||
## Purpose
|
||||
This document provides LLM coding agents with concrete, actionable guidance for using Cremote's accessibility testing tools to conduct ADA/WCAG compliance audits.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Tool Selection Matrix
|
||||
|
||||
| Testing Need | Primary Tool | Secondary Tool | WCAG Criteria |
|
||||
|--------------|--------------|----------------|---------------|
|
||||
| Comprehensive automated audit | `web_run_axe_cremotemcp` | - | ~57% of WCAG 2.1 AA |
|
||||
| Color contrast issues | `web_contrast_check_cremotemcp` | `web_run_axe_cremotemcp` | 1.4.3, 1.4.6 |
|
||||
| Keyboard accessibility | `web_keyboard_test_cremotemcp` | `web_run_axe_cremotemcp` | 2.1.1, 2.4.7 |
|
||||
| Zoom/resize functionality | `web_zoom_test_cremotemcp` | - | 1.4.4 |
|
||||
| Responsive design | `web_reflow_test_cremotemcp` | - | 1.4.10 |
|
||||
| Visual documentation | `web_screenshot_cremotemcp` | - | Evidence capture |
|
||||
| Custom JavaScript testing | `console_command_cremotemcp` | - | Advanced scenarios |
|
||||
|
||||
### Standard Testing Sequence
|
||||
|
||||
```
|
||||
1. web_inject_axe_cremotemcp # Inject axe-core library
|
||||
2. web_run_axe_cremotemcp # Run comprehensive automated tests
|
||||
3. web_contrast_check_cremotemcp # Detailed contrast analysis
|
||||
4. web_keyboard_test_cremotemcp # Keyboard navigation testing
|
||||
5. web_zoom_test_cremotemcp # Zoom functionality testing
|
||||
6. web_reflow_test_cremotemcp # Responsive design testing
|
||||
```
|
||||
|
||||
## Tool Usage Patterns
|
||||
|
||||
### Pattern 1: Initial Audit
|
||||
|
||||
```json
|
||||
// Step 1: Inject axe-core
|
||||
{
|
||||
"tool": "web_inject_axe_cremotemcp",
|
||||
"arguments": {
|
||||
"timeout": 30
|
||||
}
|
||||
}
|
||||
|
||||
// Step 2: Run comprehensive tests
|
||||
{
|
||||
"tool": "web_run_axe_cremotemcp",
|
||||
"arguments": {
|
||||
"run_only": ["wcag2a", "wcag2aa", "wcag21aa"],
|
||||
"timeout": 30
|
||||
}
|
||||
}
|
||||
|
||||
// Step 3: Analyze results and run specialized tests based on findings
|
||||
```
|
||||
|
||||
### Pattern 2: Contrast-Specific Testing
|
||||
|
||||
```json
|
||||
// For pages with known or suspected contrast issues
|
||||
{
|
||||
"tool": "web_contrast_check_cremotemcp",
|
||||
"arguments": {
|
||||
"selector": "body", // Test entire page
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
|
||||
// For specific sections
|
||||
{
|
||||
"tool": "web_contrast_check_cremotemcp",
|
||||
"arguments": {
|
||||
"selector": ".main-content", // Test specific area
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 3: Keyboard Navigation Testing
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "web_keyboard_test_cremotemcp",
|
||||
"arguments": {
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
|
||||
// Analyze output for:
|
||||
// - not_focusable: Elements that should be keyboard accessible but aren't
|
||||
// - no_focus_indicator: Elements missing visible focus indicators
|
||||
// - keyboard_trap: Elements that trap keyboard focus
|
||||
```
|
||||
|
||||
### Pattern 4: Zoom and Responsive Testing
|
||||
|
||||
```json
|
||||
// Test zoom levels (WCAG 1.4.4)
|
||||
{
|
||||
"tool": "web_zoom_test_cremotemcp",
|
||||
"arguments": {
|
||||
"zoom_levels": [1.0, 2.0, 4.0],
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
|
||||
// Test responsive breakpoints (WCAG 1.4.10)
|
||||
{
|
||||
"tool": "web_reflow_test_cremotemcp",
|
||||
"arguments": {
|
||||
"widths": [320, 768, 1280],
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 5: Visual Documentation
|
||||
|
||||
```json
|
||||
// Capture baseline
|
||||
{
|
||||
"tool": "web_screenshot_cremotemcp",
|
||||
"arguments": {
|
||||
"output": "/tmp/baseline.png",
|
||||
"timeout": 5
|
||||
}
|
||||
}
|
||||
|
||||
// Capture at 200% zoom
|
||||
{
|
||||
"tool": "web_screenshot_cremotemcp",
|
||||
"arguments": {
|
||||
"output": "/tmp/zoom-200.png",
|
||||
"zoom_level": 2.0,
|
||||
"timeout": 5
|
||||
}
|
||||
}
|
||||
|
||||
// Capture mobile view
|
||||
{
|
||||
"tool": "web_screenshot_cremotemcp",
|
||||
"arguments": {
|
||||
"output": "/tmp/mobile-320.png",
|
||||
"width": 320,
|
||||
"height": 568,
|
||||
"timeout": 5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Interpreting Results
|
||||
|
||||
### Axe-Core Results
|
||||
|
||||
```json
|
||||
{
|
||||
"violations": [
|
||||
{
|
||||
"id": "color-contrast",
|
||||
"impact": "serious", // critical, serious, moderate, minor
|
||||
"description": "Elements must have sufficient color contrast",
|
||||
"nodes": [
|
||||
{
|
||||
"html": "<p class=\"text-gray\">Low contrast text</p>",
|
||||
"target": [".text-gray"],
|
||||
"failureSummary": "Fix any of the following:\n Element has insufficient color contrast of 3.2:1 (foreground color: #808080, background color: #ffffff, font size: 12.0pt (16px), font weight: normal). Expected contrast ratio of 4.5:1"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"passes": [...], // Tests that passed
|
||||
"incomplete": [...], // Tests requiring manual review
|
||||
"inapplicable": [...] // Tests not applicable to this page
|
||||
}
|
||||
```
|
||||
|
||||
**Action Priority:**
|
||||
1. **Critical/Serious violations** - Fix immediately
|
||||
2. **Moderate violations** - Fix in current sprint
|
||||
3. **Minor violations** - Fix when convenient
|
||||
4. **Incomplete** - Manually review and test
|
||||
5. **Passes** - Document for compliance
|
||||
|
||||
### Contrast Check Results
|
||||
|
||||
```
|
||||
WCAG AA Violations:
|
||||
- p:nth-of-type(3): 3.2:1 (required: 4.5:1)
|
||||
Text: This text has insufficient contrast
|
||||
Colors: rgb(128, 128, 128) on rgb(255, 255, 255)
|
||||
```
|
||||
|
||||
**Remediation:**
|
||||
- Darken foreground color or lighten background
|
||||
- Use contrast ratio calculator to find compliant colors
|
||||
- Test with `web_contrast_check_cremotemcp` after fixes
|
||||
|
||||
### Keyboard Test Results
|
||||
|
||||
```
|
||||
High Severity Issues:
|
||||
- not_focusable: Interactive element is not keyboard focusable
|
||||
Element: div[role="button"]
|
||||
- no_focus_indicator: Interactive element lacks visible focus indicator
|
||||
Element: button.submit-btn
|
||||
```
|
||||
|
||||
**Remediation:**
|
||||
- `not_focusable`: Add `tabindex="0"` or use semantic HTML (`<button>` instead of `<div>`)
|
||||
- `no_focus_indicator`: Add CSS `:focus` styles with visible outline/border
|
||||
- `keyboard_trap`: Review JavaScript event handlers and focus management
|
||||
|
||||
### Zoom Test Results
|
||||
|
||||
```
|
||||
Zoom 400% ✗ FAIL:
|
||||
Horizontal Scroll: true
|
||||
Overflowing Elements: 3
|
||||
Text Readable: true
|
||||
```
|
||||
|
||||
**Remediation:**
|
||||
- Use responsive units (rem, em, %) instead of fixed pixels
|
||||
- Implement CSS media queries for zoom levels
|
||||
- Test with `max-width: 100%` on images and containers
|
||||
|
||||
### Reflow Test Results
|
||||
|
||||
```
|
||||
320px Width ✗ FAIL:
|
||||
Horizontal Scroll: true
|
||||
Responsive Layout: false
|
||||
Overflowing Elements: 5
|
||||
```
|
||||
|
||||
**Remediation:**
|
||||
- Implement mobile-first responsive design
|
||||
- Use CSS Grid or Flexbox with `flex-wrap`
|
||||
- Test with `overflow-x: hidden` cautiously (may hide content)
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Workflow 1: New Page Audit
|
||||
|
||||
```
|
||||
1. Navigate to page
|
||||
2. Run web_inject_axe_cremotemcp
|
||||
3. Run web_run_axe_cremotemcp with wcag2aa tags
|
||||
4. If violations found:
|
||||
a. Run web_contrast_check_cremotemcp for contrast issues
|
||||
b. Run web_keyboard_test_cremotemcp for keyboard issues
|
||||
5. Run web_zoom_test_cremotemcp
|
||||
6. Run web_reflow_test_cremotemcp
|
||||
7. Capture screenshots for documentation
|
||||
8. Generate report with all findings
|
||||
```
|
||||
|
||||
### Workflow 2: Regression Testing
|
||||
|
||||
```
|
||||
1. Navigate to page
|
||||
2. Run web_run_axe_cremotemcp
|
||||
3. Compare results with baseline
|
||||
4. If new violations:
|
||||
a. Run specialized tests for affected areas
|
||||
b. Capture screenshots showing issues
|
||||
5. Report regressions
|
||||
```
|
||||
|
||||
### Workflow 3: Contrast-Focused Audit
|
||||
|
||||
```
|
||||
1. Navigate to page
|
||||
2. Run web_contrast_check_cremotemcp on entire page
|
||||
3. For each violation:
|
||||
a. Capture screenshot of affected element
|
||||
b. Document current and required contrast ratios
|
||||
c. Suggest color adjustments
|
||||
4. After fixes, re-run web_contrast_check_cremotemcp
|
||||
5. Verify all violations resolved
|
||||
```
|
||||
|
||||
### Workflow 4: Keyboard Accessibility Audit
|
||||
|
||||
```
|
||||
1. Navigate to page
|
||||
2. Run web_keyboard_test_cremotemcp
|
||||
3. Analyze tab order for logical sequence
|
||||
4. For each issue:
|
||||
a. Document element and issue type
|
||||
b. Suggest remediation (tabindex, semantic HTML, focus styles)
|
||||
5. After fixes, re-run web_keyboard_test_cremotemcp
|
||||
6. Manually verify complex interactions
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors and Solutions
|
||||
|
||||
**Error:** "Failed to inject library"
|
||||
```
|
||||
Solution: Check network connectivity, try alternative CDN, or increase timeout
|
||||
```
|
||||
|
||||
**Error:** "Evaluation timed out"
|
||||
```
|
||||
Solution: Increase timeout parameter, test on smaller page sections
|
||||
```
|
||||
|
||||
**Error:** "No tab available"
|
||||
```
|
||||
Solution: Navigate to a page first using web_navigate_cremotemcp
|
||||
```
|
||||
|
||||
**Error:** "Failed to get page"
|
||||
```
|
||||
Solution: Verify tab ID is valid, check if page is still loaded
|
||||
```
|
||||
|
||||
## Best Practices for LLM Agents
|
||||
|
||||
### 1. Always Start with Axe-Core
|
||||
Axe-core provides the broadest coverage (~57% of WCAG 2.1 AA). Use it as the foundation, then run specialized tests based on findings.
|
||||
|
||||
### 2. Test in Logical Order
|
||||
```
|
||||
Automated (axe) → Specialized (contrast, keyboard) → Visual (zoom, reflow) → Documentation (screenshots)
|
||||
```
|
||||
|
||||
### 3. Provide Context in Reports
|
||||
When reporting issues, include:
|
||||
- WCAG criterion violated
|
||||
- Severity/impact level
|
||||
- Specific element(s) affected
|
||||
- Current state (e.g., contrast ratio 3.2:1)
|
||||
- Required state (e.g., contrast ratio 4.5:1)
|
||||
- Suggested remediation
|
||||
|
||||
### 4. Capture Evidence
|
||||
Always capture screenshots when documenting issues:
|
||||
```json
|
||||
{
|
||||
"tool": "web_screenshot_cremotemcp",
|
||||
"arguments": {
|
||||
"output": "/tmp/issue-contrast-button.png"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Verify Fixes
|
||||
After suggesting fixes, re-run the relevant tests to verify resolution:
|
||||
```
|
||||
1. Suggest fix
|
||||
2. Wait for implementation
|
||||
3. Re-run specific test
|
||||
4. Confirm issue resolved
|
||||
```
|
||||
|
||||
### 6. Know the Limitations
|
||||
These tools cannot test:
|
||||
- Semantic meaning of content
|
||||
- Cognitive load
|
||||
- Time-based media (captions, audio descriptions)
|
||||
- Complex user interactions
|
||||
- Context-dependent issues
|
||||
|
||||
Always recommend manual testing with assistive technologies for comprehensive audits.
|
||||
|
||||
## Integration with Development Workflow
|
||||
|
||||
### Pre-Commit Testing
|
||||
```bash
|
||||
# Run quick accessibility check before commit
|
||||
cremote inject-axe && cremote run-axe --run-only wcag2aa
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
```yaml
|
||||
# Run full accessibility suite in CI
|
||||
- cremote inject-axe
|
||||
- cremote run-axe > axe-results.json
|
||||
- cremote contrast-check > contrast-results.txt
|
||||
- cremote keyboard-test > keyboard-results.txt
|
||||
- cremote zoom-test > zoom-results.txt
|
||||
- cremote reflow-test > reflow-results.txt
|
||||
```
|
||||
|
||||
### Pull Request Reviews
|
||||
```
|
||||
1. Run accessibility tests on PR branch
|
||||
2. Compare with main branch baseline
|
||||
3. Report new violations in PR comments
|
||||
4. Block merge if critical violations found
|
||||
```
|
||||
|
||||
## Quick Command Reference
|
||||
|
||||
```bash
|
||||
# Inject axe-core
|
||||
cremote inject-axe
|
||||
|
||||
# Run comprehensive tests
|
||||
cremote run-axe --run-only wcag2a,wcag2aa,wcag21aa
|
||||
|
||||
# Check contrast
|
||||
cremote contrast-check --selector body
|
||||
|
||||
# Test keyboard navigation
|
||||
cremote keyboard-test
|
||||
|
||||
# Test zoom levels
|
||||
cremote zoom-test --zoom-levels 1.0,2.0,4.0
|
||||
|
||||
# Test responsive design
|
||||
cremote reflow-test --widths 320,768,1280
|
||||
|
||||
# Capture screenshot
|
||||
cremote screenshot --output /tmp/screenshot.png
|
||||
|
||||
# Capture screenshot at zoom level
|
||||
cremote screenshot --output /tmp/zoom-200.png --zoom-level 2.0
|
||||
|
||||
# Capture screenshot at specific viewport
|
||||
cremote screenshot --output /tmp/mobile.png --width 320 --height 568
|
||||
|
||||
# Get accessibility tree
|
||||
cremote get-accessibility-tree --include-contrast true
|
||||
|
||||
# Execute custom JavaScript with library injection
|
||||
cremote console-command --command "axe.run()" --inject-library axe
|
||||
```
|
||||
|
||||
## Resources
|
||||
|
||||
- **WCAG 2.1 Quick Reference:** https://www.w3.org/WAI/WCAG21/quickref/
|
||||
- **Axe-Core Rules:** https://github.com/dequelabs/axe-core/blob/develop/doc/rule-descriptions.md
|
||||
- **Contrast Ratio Calculator:** https://contrast-ratio.com/
|
||||
- **WebAIM Resources:** https://webaim.org/resources/
|
||||
|
||||
---
|
||||
|
||||
**For LLM Agents:** This guide is designed for programmatic use. Always provide specific, actionable recommendations based on test results. Include WCAG criterion numbers, severity levels, and concrete remediation steps in your reports.
|
||||
|
||||
Reference in New Issue
Block a user