{"article":{"slug":"all-you-need-is-accessibility","title":"All you need is accessibility","subtitle":null,"summary":"Building FynPDF on macOS, Maheep Kumar walks through failed AI UI-testing approaches (screenshots, VNC, XCUITest, generic computer-use) and why a small AXUIElement test API finally let agents drive the app in the background.","content_type":"blog_post","language":"en","canonical_url":"https://maheepk.net/posts/accessibility/","author":{"name":"Maheep Kumar","url":"https://maheepk.net","person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Maheep's Webspace","url":"https://maheepk.net","listing_slug":null,"listing":null},"topics":[{"name":"Developer Tools","slug":"developer-tools","url":"https://listedarticles.com/topics/developer-tools"},{"name":"AI Agents","slug":"ai-agents","url":"https://listedarticles.com/topics/ai-agents"},{"name":"Programming","slug":"programming","url":"https://listedarticles.com/topics/programming"},{"name":"User Experience","slug":"user-experience","url":"https://listedarticles.com/topics/user-experience"},{"name":"Software Engineering","slug":"software-engineering","url":"https://listedarticles.com/topics/software-engineering"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":443,"reading_minutes":2,"published_at":"2026-09-27T00:00:00.000Z","added_at":"2026-09-27T12:14:51.260Z","updated_at":"2026-09-27T12:14:51.260Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/all-you-need-is-accessibility","markdown_url":"https://listedarticles.com/articles/all-you-need-is-accessibility.md","example":false,"citation":"Maheep Kumar, Maheep's Webspace. \"All you need is accessibility.\" 27 Sept 2026. https://maheepk.net/posts/accessibility/ (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://maheepk.net/posts/accessibility/"},"body_markdown":"# All you need is accessibility\n\n*Maheep Kumar — 27 September 2026*\n\nI’ve been building FynPDF, a PDF reader and editor for macOS, with its own PDF engine written from scratch. The engine is easy to test headlessly. The user-facing viewer/editor is not: stale thumbnails, 200 ms blank saves, stuck popovers, mis-hit clicks — none of that shows up in a unit test.\n\nI wanted AI agents to exercise the application: open, scroll, zoom, annotate, save, reopen — and tell me when something went wrong. The interesting part wasn’t deciding accessibility was the answer up front; the requirements emerged one failed approach at a time.\n\n## Attempt 1: Screenshots + System Events\n\nA skill file had the agent drive FynPDF via menus/shortcuts, screenshot, and repeat. Problems: the app stole focus; clicks landed in the wrong window; screenshots burn tokens; every step was slow. A runaway render needed a watchdog. Deleted the skill. Lesson: **the test must not interfere with me.**\n\n## Attempt 2: Separate macOS account + VNC\n\nIsolation fixed desktop risk but still forced pixel reasoning and expensive observations. Lesson: talk to the application, don’t continuously stare at a desktop; keep the model out of every micro-interaction.\n\n## Attempt 3: XCUITest\n\nProper identifiers and a regression suite — but synthesized HID events take over the machine, fight the Debug build, and couple tightly to UI architecture (title-bar tabs broke “the window” assumptions).\n\n## Attempt 4: General computer-use agent\n\nBackground accessibility-tree interaction was closer, but pixel clicks failed under overlays, snapshots were expensive, focusing fields activated the app, and a person-like agent is the wrong abstraction for a deterministic test runner.\n\n## What finally worked: accessibility as a test API\n\nA ~900-line Swift CLI, `fynpdf-ax`, drives a Debug build that never comes to the front via `AXUIElement`: press actions (no pointer), set values (no focus/activation), find by identifier, walk menus. First day found real bugs (popover never dismissed; text box too small).\n\nTests are JSON **flows** — fixture PDF + steps (menu, action, set, press, capture, same/differ pixel assertions, parity, relaunch). The agent writes the flow; the runner executes hundreds of actions without burning model tokens per click. Making the app testable also made it more usable with VoiceOver and Switch Control.\n\n## Where accessibility isn’t enough\n\nVisual checks still need pixels; some bugs need in-process tile inspection; real drags/ink still matter. The point isn’t that accessibility replaces every UI test — it’s a semantic control surface that doesn’t require owning the desktop.\n\n## Advice\n\n1. Give every control an accessibility identifier.\n2. Add accessibility actions for anything that currently needs a mouse.\n3. Keep the flows in the repo — a repro that runs once is a demo; a checked-in flow is a test.\n\n*Source: [maheepk.net/posts/accessibility](https://maheepk.net/posts/accessibility/)*\n","body_html":"<h1 id=\"all-you-need-is-accessibility\">All you need is accessibility</h1>\n<p><em>Maheep Kumar — 27 September 2026</em></p>\n<p>I’ve been building FynPDF, a PDF reader and editor for macOS, with its own PDF engine written from scratch. The engine is easy to test headlessly. The user-facing viewer/editor is not: stale thumbnails, 200 ms blank saves, stuck popovers, mis-hit clicks — none of that shows up in a unit test.</p>\n<p>I wanted AI agents to exercise the application: open, scroll, zoom, annotate, save, reopen — and tell me when something went wrong. The interesting part wasn’t deciding accessibility was the answer up front; the requirements emerged one failed approach at a time.</p>\n<h2 id=\"attempt-1-screenshots-system-events\">Attempt 1: Screenshots + System Events</h2>\n<p>A skill file had the agent drive FynPDF via menus/shortcuts, screenshot, and repeat. Problems: the app stole focus; clicks landed in the wrong window; screenshots burn tokens; every step was slow. A runaway render needed a watchdog. Deleted the skill. Lesson: <strong>the test must not interfere with me.</strong></p>\n<h2 id=\"attempt-2-separate-macos-account-vnc\">Attempt 2: Separate macOS account + VNC</h2>\n<p>Isolation fixed desktop risk but still forced pixel reasoning and expensive observations. Lesson: talk to the application, don’t continuously stare at a desktop; keep the model out of every micro-interaction.</p>\n<h2 id=\"attempt-3-xcuitest\">Attempt 3: XCUITest</h2>\n<p>Proper identifiers and a regression suite — but synthesized HID events take over the machine, fight the Debug build, and couple tightly to UI architecture (title-bar tabs broke “the window” assumptions).</p>\n<h2 id=\"attempt-4-general-computer-use-agent\">Attempt 4: General computer-use agent</h2>\n<p>Background accessibility-tree interaction was closer, but pixel clicks failed under overlays, snapshots were expensive, focusing fields activated the app, and a person-like agent is the wrong abstraction for a deterministic test runner.</p>\n<h2 id=\"what-finally-worked-accessibility-as-a-test-api\">What finally worked: accessibility as a test API</h2>\n<p>A ~900-line Swift CLI, <code>fynpdf-ax</code>, drives a Debug build that never comes to the front via <code>AXUIElement</code>: press actions (no pointer), set values (no focus/activation), find by identifier, walk menus. First day found real bugs (popover never dismissed; text box too small).</p>\n<p>Tests are JSON <strong>flows</strong> — fixture PDF + steps (menu, action, set, press, capture, same/differ pixel assertions, parity, relaunch). The agent writes the flow; the runner executes hundreds of actions without burning model tokens per click. Making the app testable also made it more usable with VoiceOver and Switch Control.</p>\n<h2 id=\"where-accessibility-isn-t-enough\">Where accessibility isn’t enough</h2>\n<p>Visual checks still need pixels; some bugs need in-process tile inspection; real drags/ink still matter. The point isn’t that accessibility replaces every UI test — it’s a semantic control surface that doesn’t require owning the desktop.</p>\n<h2 id=\"advice\">Advice</h2>\n<ol><li>Give every control an accessibility identifier.</li><li>Add accessibility actions for anything that currently needs a mouse.</li><li>Keep the flows in the repo — a repro that runs once is a demo; a checked-in flow is a test.</li></ol>\n<p><em>Source: <a href=\"https://maheepk.net/posts/accessibility/\" rel=\"nofollow ugc noopener\">maheepk.net/posts/accessibility</a></em></p>","headings":[{"level":1,"text":"All you need is accessibility","id":"all-you-need-is-accessibility"},{"level":2,"text":"Attempt 1: Screenshots + System Events","id":"attempt-1-screenshots-system-events"},{"level":2,"text":"Attempt 2: Separate macOS account + VNC","id":"attempt-2-separate-macos-account-vnc"},{"level":2,"text":"Attempt 3: XCUITest","id":"attempt-3-xcuitest"},{"level":2,"text":"Attempt 4: General computer-use agent","id":"attempt-4-general-computer-use-agent"},{"level":2,"text":"What finally worked: accessibility as a test API","id":"what-finally-worked-accessibility-as-a-test-api"},{"level":2,"text":"Where accessibility isn’t enough","id":"where-accessibility-isn-t-enough"},{"level":2,"text":"Advice","id":"advice"}]}}