bump
This commit is contained in:
Josh at WLTechBlog
486
NEW_FEATURES_TESTING_GUIDE.md
Normal file
486
NEW_FEATURES_TESTING_GUIDE.md
Normal file
@@ -0,0 +1,486 @@
|
||||
# New Features Testing Guide
|
||||
|
||||
**Date:** 2025-10-02
|
||||
**Version:** 1.0
|
||||
**Status:** Ready for Testing
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This guide provides specific test cases for the **8 new automated accessibility testing tools** added to cremote. These tools increase WCAG 2.1 Level AA coverage from 70% to 93%.
|
||||
|
||||
---
|
||||
|
||||
## Testing Prerequisites
|
||||
|
||||
### 1. Deployment
|
||||
- [ ] cremote daemon restarted with new binaries
|
||||
- [ ] MCP server updated with new tools
|
||||
- [ ] All 8 new tools visible in MCP tool list
|
||||
|
||||
### 2. Dependencies
|
||||
- [ ] ImageMagick installed (for gradient contrast)
|
||||
- [ ] Tesseract OCR 5.5.0+ installed (for text-in-images)
|
||||
|
||||
### 3. Test Pages
|
||||
Prepare test pages with:
|
||||
- Gradient backgrounds with text
|
||||
- Video/audio elements with and without captions
|
||||
- Tooltips and hover content
|
||||
- Images containing text
|
||||
- Multiple pages with navigation
|
||||
- Instructional content with sensory references
|
||||
- Animated content (CSS, GIF, video)
|
||||
- Interactive elements with ARIA attributes
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 Tools Testing
|
||||
|
||||
### Tool 1: Gradient Contrast Check
|
||||
|
||||
**Tool:** `web_gradient_contrast_check_cremotemcp`
|
||||
**WCAG:** 1.4.3, 1.4.6, 1.4.11
|
||||
|
||||
#### Test Cases
|
||||
|
||||
**Test 1.1: Linear Gradient with Good Contrast**
|
||||
```json
|
||||
{
|
||||
"tool": "web_gradient_contrast_check_cremotemcp",
|
||||
"arguments": {
|
||||
"selector": ".good-gradient",
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
**Expected:** WCAG AA pass, worst_case_ratio ≥ 4.5:1
|
||||
|
||||
**Test 1.2: Linear Gradient with Poor Contrast**
|
||||
```json
|
||||
{
|
||||
"tool": "web_gradient_contrast_check_cremotemcp",
|
||||
"arguments": {
|
||||
"selector": ".bad-gradient",
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
**Expected:** WCAG AA fail, worst_case_ratio < 4.5:1, specific recommendations
|
||||
|
||||
**Test 1.3: Multiple Elements with Gradients**
|
||||
```json
|
||||
{
|
||||
"tool": "web_gradient_contrast_check_cremotemcp",
|
||||
"arguments": {
|
||||
"selector": "body",
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
**Expected:** Analysis of all gradient backgrounds, list of violations
|
||||
|
||||
**Test 1.4: Element without Gradient**
|
||||
```json
|
||||
{
|
||||
"tool": "web_gradient_contrast_check_cremotemcp",
|
||||
"arguments": {
|
||||
"selector": ".solid-background",
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
**Expected:** No gradient detected message or fallback to standard contrast check
|
||||
|
||||
---
|
||||
|
||||
### Tool 2: Media Validation
|
||||
|
||||
**Tool:** `web_media_validation_cremotemcp`
|
||||
**WCAG:** 1.2.2, 1.2.5, 1.4.2
|
||||
|
||||
#### Test Cases
|
||||
|
||||
**Test 2.1: Video with Captions**
|
||||
```json
|
||||
{
|
||||
"tool": "web_media_validation_cremotemcp",
|
||||
"arguments": {
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
**Expected:** Video detected, captions present, no violations
|
||||
|
||||
**Test 2.2: Video without Captions**
|
||||
**Expected:** Missing captions violation, recommendation to add track element
|
||||
|
||||
**Test 2.3: Video with Autoplay**
|
||||
**Expected:** Autoplay violation if no controls, recommendation to add controls or disable autoplay
|
||||
|
||||
**Test 2.4: Audio Element**
|
||||
**Expected:** Audio detected, check for transcript or captions
|
||||
|
||||
**Test 2.5: Inaccessible Track File**
|
||||
**Expected:** Track file error, recommendation to fix URL or file
|
||||
|
||||
---
|
||||
|
||||
### Tool 3: Hover/Focus Content Testing
|
||||
|
||||
**Tool:** `web_hover_focus_test_cremotemcp`
|
||||
**WCAG:** 1.4.13
|
||||
|
||||
#### Test Cases
|
||||
|
||||
**Test 3.1: Native Title Tooltip**
|
||||
```json
|
||||
{
|
||||
"tool": "web_hover_focus_test_cremotemcp",
|
||||
"arguments": {
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
**Expected:** Native title tooltip detected, violation flagged
|
||||
|
||||
**Test 3.2: Custom Tooltip (Dismissible)**
|
||||
**Expected:** Tooltip can be dismissed with Escape key, passes
|
||||
|
||||
**Test 3.3: Custom Tooltip (Not Dismissible)**
|
||||
**Expected:** Violation - cannot dismiss with Escape
|
||||
|
||||
**Test 3.4: Tooltip (Not Hoverable)**
|
||||
**Expected:** Violation - tooltip disappears when hovering over it
|
||||
|
||||
**Test 3.5: Tooltip (Not Persistent)**
|
||||
**Expected:** Warning - tooltip disappears too quickly
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 Tools Testing
|
||||
|
||||
### Tool 4: Text-in-Images Detection
|
||||
|
||||
**Tool:** `web_text_in_images_cremotemcp`
|
||||
**WCAG:** 1.4.5, 1.4.9, 1.1.1
|
||||
|
||||
#### Test Cases
|
||||
|
||||
**Test 4.1: Image with Text and Good Alt**
|
||||
```json
|
||||
{
|
||||
"tool": "web_text_in_images_cremotemcp",
|
||||
"arguments": {
|
||||
"timeout": 30
|
||||
}
|
||||
}
|
||||
```
|
||||
**Expected:** Text detected, alt text adequate, passes
|
||||
|
||||
**Test 4.2: Image with Text and No Alt**
|
||||
**Expected:** Violation - missing alt text, detected text shown
|
||||
|
||||
**Test 4.3: Image with Text and Insufficient Alt**
|
||||
**Expected:** Violation - alt text doesn't include all detected text
|
||||
|
||||
**Test 4.4: Decorative Image with No Text**
|
||||
**Expected:** No text detected, no violation
|
||||
|
||||
**Test 4.5: Complex Infographic**
|
||||
**Expected:** Multiple text elements detected, recommendation for detailed alt text
|
||||
|
||||
---
|
||||
|
||||
### Tool 5: Cross-Page Consistency
|
||||
|
||||
**Tool:** `web_cross_page_consistency_cremotemcp`
|
||||
**WCAG:** 3.2.3, 3.2.4, 1.3.1
|
||||
|
||||
#### Test Cases
|
||||
|
||||
**Test 5.1: Consistent Navigation**
|
||||
```json
|
||||
{
|
||||
"tool": "web_cross_page_consistency_cremotemcp",
|
||||
"arguments": {
|
||||
"urls": [
|
||||
"https://example.com/",
|
||||
"https://example.com/about",
|
||||
"https://example.com/contact"
|
||||
],
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
**Expected:** Common navigation detected, all pages consistent, passes
|
||||
|
||||
**Test 5.2: Inconsistent Navigation**
|
||||
**Expected:** Violation - missing navigation links on some pages
|
||||
|
||||
**Test 5.3: Multiple Main Landmarks**
|
||||
**Expected:** Violation - multiple main landmarks without labels
|
||||
|
||||
**Test 5.4: Missing Header/Footer**
|
||||
**Expected:** Warning - inconsistent landmark structure
|
||||
|
||||
---
|
||||
|
||||
### Tool 6: Sensory Characteristics Detection
|
||||
|
||||
**Tool:** `web_sensory_characteristics_cremotemcp`
|
||||
**WCAG:** 1.3.3
|
||||
|
||||
#### Test Cases
|
||||
|
||||
**Test 6.1: Color-Only Instruction**
|
||||
```json
|
||||
{
|
||||
"tool": "web_sensory_characteristics_cremotemcp",
|
||||
"arguments": {
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
**Text:** "Click the red button to continue"
|
||||
**Expected:** Violation - color-only instruction detected
|
||||
|
||||
**Test 6.2: Shape-Only Instruction**
|
||||
**Text:** "Press the round icon to submit"
|
||||
**Expected:** Violation - shape-only instruction detected
|
||||
|
||||
**Test 6.3: Location-Only Instruction**
|
||||
**Text:** "See the information above"
|
||||
**Expected:** Warning - location-based instruction detected
|
||||
|
||||
**Test 6.4: Multi-Sensory Instruction**
|
||||
**Text:** "Click the red 'Submit' button on the right"
|
||||
**Expected:** Pass - multiple cues provided
|
||||
|
||||
**Test 6.5: Sound-Only Instruction**
|
||||
**Text:** "Listen for the beep to confirm"
|
||||
**Expected:** Violation - sound-only instruction detected
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 Tools Testing
|
||||
|
||||
### Tool 7: Animation/Flash Detection
|
||||
|
||||
**Tool:** `web_animation_flash_cremotemcp`
|
||||
**WCAG:** 2.3.1, 2.2.2, 2.3.2
|
||||
|
||||
#### Test Cases
|
||||
|
||||
**Test 7.1: CSS Animation (Safe)**
|
||||
```json
|
||||
{
|
||||
"tool": "web_animation_flash_cremotemcp",
|
||||
"arguments": {
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
**Expected:** Animation detected, no flashing, passes
|
||||
|
||||
**Test 7.2: Rapid Flashing Content**
|
||||
**Expected:** Violation - flashing > 3 times per second
|
||||
|
||||
**Test 7.3: Autoplay Animation > 5s without Controls**
|
||||
**Expected:** Violation - no pause/stop controls
|
||||
|
||||
**Test 7.4: Animated GIF**
|
||||
**Expected:** GIF detected, check for controls if > 5s
|
||||
|
||||
**Test 7.5: Video with Flashing**
|
||||
**Expected:** Warning - video may contain flashing (manual review needed)
|
||||
|
||||
---
|
||||
|
||||
### Tool 8: Enhanced Accessibility Tree
|
||||
|
||||
**Tool:** `web_enhanced_accessibility_cremotemcp`
|
||||
**WCAG:** 1.3.1, 4.1.2, 2.4.6
|
||||
|
||||
#### Test Cases
|
||||
|
||||
**Test 8.1: Button with Accessible Name**
|
||||
```json
|
||||
{
|
||||
"tool": "web_enhanced_accessibility_cremotemcp",
|
||||
"arguments": {
|
||||
"timeout": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
**Expected:** Button has accessible name, passes
|
||||
|
||||
**Test 8.2: Button without Accessible Name**
|
||||
**Expected:** Violation - missing accessible name
|
||||
|
||||
**Test 8.3: Interactive Element with aria-hidden**
|
||||
**Expected:** Violation - aria-hidden on interactive element
|
||||
|
||||
**Test 8.4: Invalid Tabindex**
|
||||
**Expected:** Violation - tabindex value not 0 or -1
|
||||
|
||||
**Test 8.5: Multiple Nav Landmarks without Labels**
|
||||
**Expected:** Violation - multiple landmarks need distinguishing labels
|
||||
|
||||
**Test 8.6: Broken aria-labelledby Reference**
|
||||
**Expected:** Warning - referenced ID does not exist
|
||||
|
||||
---
|
||||
|
||||
## Integration Testing
|
||||
|
||||
### Test Suite 1: Complete Page Audit
|
||||
|
||||
Run all 8 new tools on a single test page:
|
||||
|
||||
```bash
|
||||
1. web_gradient_contrast_check_cremotemcp
|
||||
2. web_media_validation_cremotemcp
|
||||
3. web_hover_focus_test_cremotemcp
|
||||
4. web_text_in_images_cremotemcp
|
||||
5. web_sensory_characteristics_cremotemcp
|
||||
6. web_animation_flash_cremotemcp
|
||||
7. web_enhanced_accessibility_cremotemcp
|
||||
8. web_cross_page_consistency_cremotemcp (with multiple URLs)
|
||||
```
|
||||
|
||||
**Expected:** All tools complete successfully, results are actionable
|
||||
|
||||
### Test Suite 2: Performance Testing
|
||||
|
||||
Measure processing time for each tool:
|
||||
|
||||
| Tool | Expected Time | Acceptable Range |
|
||||
|------|---------------|------------------|
|
||||
| Gradient Contrast | 2-5s | < 10s |
|
||||
| Media Validation | 3-8s | < 15s |
|
||||
| Hover/Focus Test | 5-15s | < 30s |
|
||||
| Text-in-Images | 10-30s | < 60s |
|
||||
| Cross-Page (3 pages) | 6-15s | < 30s |
|
||||
| Sensory Chars | 1-3s | < 5s |
|
||||
| Animation/Flash | 2-5s | < 10s |
|
||||
| Enhanced A11y | 3-8s | < 15s |
|
||||
|
||||
### Test Suite 3: Error Handling
|
||||
|
||||
Test error conditions:
|
||||
|
||||
1. **Invalid selector:** Should return clear error message
|
||||
2. **Timeout exceeded:** Should return partial results or timeout error
|
||||
3. **Missing dependencies:** Should return dependency error (ImageMagick, Tesseract)
|
||||
4. **Network errors:** Should handle gracefully (cross-page, text-in-images)
|
||||
5. **Empty page:** Should return "no elements found" message
|
||||
|
||||
---
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
### Functionality
|
||||
- [ ] All 8 tools execute without errors
|
||||
- [ ] Results are accurate and actionable
|
||||
- [ ] Violations are correctly identified
|
||||
- [ ] Recommendations are specific and helpful
|
||||
- [ ] WCAG criteria are correctly referenced
|
||||
|
||||
### Performance
|
||||
- [ ] Processing times are within acceptable ranges
|
||||
- [ ] No memory leaks or resource exhaustion
|
||||
- [ ] Concurrent tool execution works correctly
|
||||
- [ ] Large pages are handled gracefully
|
||||
|
||||
### Accuracy
|
||||
- [ ] Gradient contrast calculations are correct
|
||||
- [ ] Media validation detects all video/audio elements
|
||||
- [ ] Hover/focus testing catches violations
|
||||
- [ ] OCR accurately detects text in images
|
||||
- [ ] Cross-page consistency correctly identifies common elements
|
||||
- [ ] Sensory characteristics patterns are detected
|
||||
- [ ] Animation/flash detection identifies violations
|
||||
- [ ] ARIA validation catches missing names and invalid attributes
|
||||
|
||||
### Documentation
|
||||
- [ ] Tool descriptions are clear
|
||||
- [ ] Usage examples are correct
|
||||
- [ ] Error messages are helpful
|
||||
- [ ] WCAG references are accurate
|
||||
|
||||
---
|
||||
|
||||
## Known Issues and Limitations
|
||||
|
||||
Document any issues found during testing:
|
||||
|
||||
1. **Gradient Contrast:**
|
||||
- Complex gradients (radial, conic) may not be fully analyzed
|
||||
- Very large gradients may take longer to process
|
||||
|
||||
2. **Media Validation:**
|
||||
- Cannot verify caption accuracy (only presence)
|
||||
- May not detect dynamically loaded media
|
||||
|
||||
3. **Hover/Focus:**
|
||||
- May miss custom implementations using non-standard patterns
|
||||
- Timing-dependent, may need adjustment
|
||||
|
||||
4. **Text-in-Images:**
|
||||
- OCR struggles with stylized fonts, handwriting
|
||||
- Low contrast text may not be detected
|
||||
- CPU-intensive, takes longer
|
||||
|
||||
5. **Cross-Page:**
|
||||
- Requires 2+ pages
|
||||
- May flag intentional variations as violations
|
||||
- Network-dependent
|
||||
|
||||
6. **Sensory Characteristics:**
|
||||
- Context-dependent, may have false positives
|
||||
- Pattern matching may miss creative phrasing
|
||||
|
||||
7. **Animation/Flash:**
|
||||
- Simplified flash rate estimation
|
||||
- Cannot analyze video frame-by-frame
|
||||
- May miss JavaScript-driven animations
|
||||
|
||||
8. **Enhanced A11y:**
|
||||
- Simplified reference validation
|
||||
- Doesn't check all ARIA states (expanded, selected, etc.)
|
||||
- May miss complex widget issues
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Testing is complete when:
|
||||
|
||||
- [ ] All 8 tools execute successfully on test pages
|
||||
- [ ] Accuracy is ≥ 75% for each tool (compared to manual testing)
|
||||
- [ ] Performance is within acceptable ranges
|
||||
- [ ] Error handling is robust
|
||||
- [ ] Documentation is accurate and complete
|
||||
- [ ] Known limitations are documented
|
||||
- [ ] User feedback is positive
|
||||
|
||||
---
|
||||
|
||||
## Next Steps After Testing
|
||||
|
||||
1. **Document findings** - Create test report with results
|
||||
2. **Fix critical issues** - Address any blocking bugs
|
||||
3. **Update documentation** - Refine based on testing experience
|
||||
4. **Train users** - Create training materials and examples
|
||||
5. **Monitor production** - Track accuracy and performance in real use
|
||||
6. **Gather feedback** - Collect user feedback for improvements
|
||||
7. **Plan enhancements** - Identify areas for future improvement
|
||||
|
||||
---
|
||||
|
||||
**Ready for Testing!** 🚀
|
||||
|
||||
Use this guide to systematically test all new features and validate the 93% WCAG 2.1 Level AA coverage claim.
|
||||
|
||||
Reference in New Issue
Block a user