Software, Accessibility, and Immersive Tech

Accessibility Tooling for Agentic Coding Loops

2026-02-23T00:41:23+00:00

Coding agents are writing and modifying front-end code at scale. If your team maintains a design system or owns accessibility on a product, this changes the calculus: code that once went through a human review loop is now generated in seconds, often without any accessibility check at all.

Existing tools weren’t designed for this. They assume a human in the loop; run a scan, read the report, interpret the findings, figure out the fix. That workflow requires domain expertise at every step.

An agent operating in a coding loop needs something different. Not a report to interpret, but structured diagnostics it can act on directly: machine-executable fix instructions, DOM context for reasoning, a fixability classification for triage, and a verification mechanism to confirm fixes landed.

@accesslint/mcp is an MCP server built on @accesslint/core, a rule engine designed from the ground up for agent consumption. It exposes tools for auditing HTML (as a string, file, or URL), diffing before-and-after audits, and listing rules; for Claude Code, Cursor, Windsurf, or any MCP-compatible agent.

This post walks through the design decisions behind the tool: how violations are structured, how fixability classification works, what context collection looks like per rule, and how the diff loop closes the audit-fix-verify cycle.

Does it actually help?

Before getting into the design, here’s the evidence. Both approaches - agent with MCP tools vs. agent alone - were benchmarked across 25 HTML test cases covering 67 fixable WCAG violations (3 runs each, Claude Opus):

	With @accesslint/mcp	Agent alone
Violations fixed	99.5% (200/201)	93.5% (188/201)
Regressions	1.7 / run	2.0 / run
Cost	$0.56 / run	$0.62 / run
Duration	270s / run	377s / run
Timeouts	0 / 63 tasks	2 / 63 tasks

The MCP-assisted path uses 23% fewer output tokens per run. Without tools, the agent has to recall WCAG rules from training data, reason about which rules apply to which elements, and then fix them. The MCP replaces that open-ended reasoning with structured output: specific rule IDs, CSS selectors pointing to exact elements, and concrete fix suggestions. The agent skips straight to applying fixes. Fewer reasoning steps, fewer tokens, less time, lower cost.

The largest gains are on complex cases. A test case with 6 violations across nested landmark structures completed in 25-38 seconds with MCP tooling. The agent alone timed out at 90 seconds in 2 of 3 runs.

Anatomy of a violation

When an agent calls audit_html, each violation includes:

1. [CRITICAL] labels-and-names/button-name
   Button has no discernible text.
   Element: button.icon-search
   HTML: <button class="icon-search" onclick="openSearch()"><svg aria-hidden="true">...</svg></button>
   Fix: add-text-content
   Fixability: contextual
   Browser hint: Screenshot the button to identify its icon or visual label,
   then add a matching aria-label.
   Context: Classes: icon-search
   Guidance: Screen reader users need to know what a button does. Add visible
   text content, aria-label, or aria-labelledby. For icon buttons, use
   aria-label describing the action (e.g., aria-label='Close'). If the button
   contains an image, ensure the image has alt text describing the button's
   action.

Each field is deliberate:

Fix is a structured instruction from a closed set: add-attribute, set-attribute, remove-attribute, add-element, remove-element, add-text-content, or suggest. The first six are mechanically executable. suggest is the escape hatch for violations where the fix depends on intent. About 75% of rules provide a mechanical fix; the remaining 25% use suggest.

Fixability classifies the violation, not the fix. The button-name rule provides a mechanical fix type (add-text-content) that satisfies the rule. But its contextual classification signals that the agent should use the collected context to determine what text to add. More on this below.

Context is collected per-rule from the DOM. Guidance provides the remediation principle behind the rule, written for direct LLM consumption. Browser hint, present on 26 of 92 rules, tells agents with browser access how to verify or improve a fix using screenshots or DevTools.

Determining “Fixability”

Every rule carries a fixability classification that the MCP server surfaces on each violation:

Mechanical (20 rules): Deterministic. A positive tabindex gets set to "0". A non-valid ARIA role gets flagged with the correct value. No ambiguity.
Contextual (65 rules): Requires surrounding context, but an LLM can reason about it. The violation’s context and guidance fields provide the inputs. The structured fix provides a safe floor.
Visual (4 rules): Requires rendered output. Color contrast, primarily. Browser hints tell the agent how to inspect computed styles or screenshot the element.

The interaction between fixability and the structured fix is the core design decision. Take button-name: the fix type is add-text-content, which is mechanically executable. But what text? The contextual classification is the signal to use the collected context. The violation reports Classes: icon-search. That’s developer intent that never made it into the accessible name. The agent reads the class, infers the action, and adds aria-label="Search".

This also maps to list_rules filtering. An agent or workflow can query rules by fixability to scope an audit pass: mechanical-only for automated batch remediation, contextual for agent-assisted passes, visual for flagging to human review.

Gathering context

Each rule gathers the specific context its violation type needs. This is where the design diverges most from existing tools, which tend to report the element and stop.

button-name reports CSS class names (btn-close, icon-search), the enclosing form’s label, and the nearest heading. Class names are the key signal. They encode developer intent that never made it into the accessible name. A button with class icon-search inside a form labeled “Site search” gives the agent two independent signals pointing to aria-label="Search".

img-alt checks whether the image is inside a link (and captures the href), looks for a figcaption, and captures adjacent text. If a figcaption already describes the image, alt="" avoids redundant adjacent text. If the image is a standalone link, the href helps the agent infer purpose.

form-label reports the input’s type, name, placeholder, and id, plus the full accessible name computation chain: aria-labelledby resolution, aria-label, associated <label>, title, and placeholder fallback.

link-name captures the href, nearby heading text, and parent element context. For a link wrapping only an icon, the href and surrounding headings are often enough to infer purpose.

The goal is to front-load enough information that the agent can reason about the fix in a single pass, without a round-trip to read more of the document.

Looping on diffs

The diff loop is the verification mechanism. The workflow:

The agent calls audit_html with name: "before" to audit and store the result.
The agent applies fixes.
The agent calls diff_html with the updated markup and before: "before".

The server audits the new HTML, diffs against the stored result, and returns:

Summary: 2 fixed, 1 new, 3 remaining

FIXED:
  - [CRITICAL] text-alternatives/img-alt at img[src="photo.jpg"]
  - [CRITICAL] labels-and-names/button-name at button.icon-search

NEW:
  - [SERIOUS] aria/aria-roles at div[role="buton"]
    ARIA role "buton" is not a valid role value.
    Fix: set-attribute role="button"

REMAINING:
  - [MODERATE] navigable/heading-order at h4
  - [MODERATE] distinguishable/link-in-text-block at a.subtle
  - [MINOR] text-alternatives/image-alt-words at img[alt="image of logo"]

Violations are matched by ruleId + selector. The agent gets a clear signal: what was fixed, what regressed (with the diagnosis and fix instruction for self-correction), and what remains.

The NEW category is critical. An agent that adds role="buton" (a typo) gets the regression surfaced immediately, with a structured fix to correct it. The loop is: audit, fix, diff, self-correct. No human in the middle.

Opening up the browser

26 rules carry a browser hint: an instruction for agents with browser access (screenshots, DevTools MCP) on how to verify or improve a fix.

For a visual rule like color-contrast:

Violation context includes computed colors and ratio. After changing colors, use JavaScript to read getComputedStyle() on the element and recalculate the contrast ratio. Screenshot the element to verify the fix looks correct in context.

For a contextual rule like button-name:

Screenshot the button to identify its icon or visual label, then add a matching aria-label.

These are opt-in. An agent without browser tools ignores them. An agent with browser MCP tools (Chrome DevTools, Playwright) can use them to bridge the gap between static analysis and rendered output, particularly for the 4 visual rules where static analysis alone can’t fully verify the fix.

Handling page fragments

When the input HTML lacks <!DOCTYPE html> or <html>, the server auto-enables component mode, suppressing 22 page-level rules (document title, landmarks, lang attribute, etc.) that would produce false positives on isolated markup. The agent can override this with the component_mode parameter.

This means an agent auditing a React component, a partial template, or a code snippet gets relevant results without noise.

Architecture

The MCP server is a thin integration layer. The rule engine is @accesslint/core: 92 rules, zero runtime dependencies, synchronous execution. The API is runAudit(doc: Document): AuditResult. The server handles HTML parsing (happy-dom), fragment detection, audit state for diffing, and violation enrichment (joining each violation with its rule’s fixability, browser hint, and guidance).

The core library covers 23 WCAG 2.1 success criteria across Level A and AA, scoped to what static DOM analysis can meaningfully check. Rules that throw during execution are caught and skipped; the audit always completes.

The library also exports lower-level primitives (getAccessibleName, getComputedRole, isAriaHidden) and a declarative rule engine for authoring rules as JSON with validateDeclarativeRule and compileDeclarativeRule. An agent can write, validate, and register new rules at runtime.

A note on the core engine

Every MCP tool call is a round-trip through the protocol, and an agent running audit-fix-diff in a loop may make dozens of them. Latency per call matters. Engines like axe-core were designed at the outset for the browser; the execution model is async and demonstrably slow¹. The latency accrues.

The rule engine is @accesslint/core: 92 rules, 23 WCAG 2.1 success criteria (Level A and AA), zero runtime dependencies, synchronous execution. The MCP server parses HTML with happy-dom and calls runAudit(doc): AuditResult. A typical component audit completes in single-digit milliseconds. At that speed, the bottleneck is the LLM, not the tooling.

The structured output described throughout this post (fix suggestions, fixability classifications, per-rule context, browser hints) are first-class fields on every violation, not bolted on after the fact. The rules are authored with agent consumption in mind.

To add the MCP server to Claude Code:

claude mcp add accesslint -- npx -y @accesslint/mcp

Or add it to any MCP client configuration:

{
  "mcpServers": {
    "accesslint": {
      "command": "npx",
      "args": ["-y", "@accesslint/mcp"]
    }
  }
}

What’s next

Testing in the DOM is important for many accessibility tests, and doesn’t have to be slow. Keep an eye out for improvements to the AccessLint StoryBook Addon soon that will help.

I’d love to hear from you! Please share questions and comments, or drop me a note, I’m always happy to nerd out on accessibility.

@accesslint/core vs axe-core benchmarks ↩

Gesture Control in Adobe XD

2019-06-25T00:00:00+00:00

Adobe XD launched key command support recently, that allows you to trigger actions based on gamepad or keyboard input. You can use this feature to control your prototype with other inputs, by mapping values to keyboard events on the web.

I created a Chrome Extension (code on GitHub) that takes in a webcam feed and translates hand gestures to key commands.

It uses TensorFlow.js and the PoseNet model to identify your hand position relative to your body.
Then, the extension checks if your hands are in one of a few orientations to your shoulders and face.
If the gesture matches a command, the plugin creates a keyboard event that is sent to the canvas, like so:

const ev = new KeyboardEvent("keydown", { key: "w" }); // replace "w" with the keyboard trigger value
document.querySelector("#root-1").dispatchEvent(ev);

The XD web application handles the rest!

Combine this with voice commands, that are built into XD, to prototype a multimodal experience.

Auto Mute Video Calls using Head Rotation

2019-03-30T00:00:00+00:00

When you look away from your screen during a conference call, it’s probably because someone interrupted you, or you need to have a sidebar. It can be anywhere from distracting to embarrassing to have these interactions get broadcast to your meeting room. Microsoft recently released a background blurring feature for these situations:

This weekend I wrote some code with a similar goal. It disables your video in a conference call when you turn away from the screen. It does that by connecting to Bose AR wearable devices – in the demo my Bose Frames – to tell what direction your head is turning. Then it disables the camera if you’re looking away.

The bookmarklet connects using the Web Bluetooth API, via a JavaScript port of the Bose AR SDK. I made it for fun and to learn more about spatial audio and audio AR. If you want to try it out: in Google Chrome, drag this link to your bookmarks bar:

Enable Auto Mute

Then open a meeting in the Zoom.us web client, and click the bookmark to load the code. Then, when you turn on the camera, you’ll get a prompt to connect your Bose Frames or QC35 II headphones.

Create a GitHub PR via a link

2019-03-20T00:00:00+00:00

It’s possible to auto-populate a PR with parameters in a link, like you can with a mailto: link.

Parameters

https://github.com/[account]/[repo]/new/master \
  ?filename=filename \
  &value=file contents \
  &message=PR title \
  &description=PR description \
  &quick_pull=base-branch-name \
  &target_branch=new-branch-name

Demo

Try it out. Open a test PR.

Ruby Helper

    def link_to_test_pr(project:, options: {})
      hostname = "https://www.github.com/"
      url_params = File.join(project, *%w(new master))
      uri = URI.join(hostname, url_params)
      query_params = %w(
        filename=accesslint-test.html
        value=<img\ src="logo.png"\ />
        message=[DO\ NOT\ MERGE]\ Testing\ AccessLint
        description=Expect\ comments\ from\ AccessLint
        quick_pull=master
        target_branch=accesslint-test
      ).join("&")
      uri.query = query_params

      link_to(t(".test"), uri.to_s, class: options[:class])
    end

Background

I uncovered this while building AccessLint GitHub App, to open demo PR to test the build integration.

Bose Frames Prescription Lenses

2019-03-18T00:00:00+00:00

I’ve been wearing the new Bose Frames full time since January. They’re audio AR glasses that are low profile, inexpensive, have solid sensor hardware, and an application platform via iOS, Android, and Unity developer SDKs.

The Bose Frames are sold as sunglasses, and Bose doesn’t offer a prescription option. Despite that, I took them to my local optometrist, James Leonard, and they agreed to try fitting them with my prescription. Turned out it was not an issue, and mostly covered under my vision insurance plan.

All in, Frames and lenses, I spent $280. That’s pretty good for some slick prescription AR glasses.

Talking Web - Voice Control using Semantic HTML and JavaScript

2018-09-24T11:04:00+00:00

An HTML element and its attributes that carry intrinsic meaning. When you compose these elements into a page or web application, in a way that is semantic (and accessible), support for Voice control falls into place.

Searching for Meaning

Let’s look an example using Search. Search is a pretty easy choice, because it’s such ubiquitous, consistent, and standardized feature of web pages. This is a Chrome extension I’ve dubbed “Say11y”, to demonstrate using website search with Voice, with no added HTML or back-end APIs.

The plugin uses the Speech Recognition API to access the microphone and capture utterances. Once the recognizer detects a discernible phrase, we look up the command. In this case, if the phrase starts with “Search”, we find a HTML element on the page that meets one of these selector criteria:

input type of search
aria label that starts with “search”
value, name, or class attribute that starts with “search”

If we find a matching element, we set the form value for that field to the content of the utterance that following the command, and submit the form.

Leaning on Standards

It took me a couple of hours to make this work fairly consistently across a dozen different sites that I tried. That’s because, when web content creators adhere to standards, it’s trivial to make Voice control work.

This can apply to many different features. Here are a couple off the top of my head:

“Search for hard cheese.” (input[type='search'])
“Found Cheddar, Manchego, and Swiss” ([role='main'] > ul)
“Describe cheddar”
“Aged 2 years, from Vermont. $4.99” (dd)
“Add cheddar”
“How much?”
“3 ounces”
“Added.” (form input[type^='number'])
“Where am I?”
“Shopping cart” (h1)

Accessibility Innovations

Web standards advocates have long seen the benefits of semantic markup for accessibility and screen reader support. Now, with the advent of Voice Assistants, it should be clear that this is a universal principal. I’m excited to see how we can take other lessons from accessible technology to set us up for the future.

Assistive Tech with AR Way-finding

2018-09-24T11:04:00+00:00

I joined a mini-batch at the Recurse Center last week, and spent my time researching computer vision for people with visual impairments. I came up with an app to find empty seats in a crowded space, like a classroom or conference hall. Here’s a breakdown of what I did and why.

This clip shows me panning the device around the room. Sighted users will see a box outlining the chairs in the viewport.

The Problem

A person with a visual impairment may find it annoying and uncomfortable to find a seat at an event. From talking with friends of mine who are blind, this is especially true because

Blind folks get unwanted attention when way-finding.

This can lead to disruptions, and sometimes unwelcome guidance.

Hypothesis

Mobile phones now have enough power to run real-time object recognition, using their built in cameras and processing power. We can use these advancements to augment traditional way-finding devices like canes and service animals.

Challenges

Recording in public is still taboo and conspicuous.
People with disabilities don’t have a lot of cash to spend, due to high unemployment rates.

Design

I built the initial design using Adobe XD, using auto-animate and audio output features. In the process I discovered that audio cues needed to be very succinct and in some cases replaced with vibration, and that I’d need to signal obstacles, distance, and placement, as well as just identifying the object.

Approach

There are many computer vision models for neural networks. Some have existing ports to mobile phones. iOS also has it’s own native object recognition in ARKit.

iOS is more popular than Android with people using screen readers, so I started there. I tried ARKit’s object classifier first, but it does not identify more that one object at once. It’s also limited to newer devices.

Instead of ARKit 2, I turned to Forge, an iOS toolkit for neural networks. It includes the YOLO neural network. This implementation is limited to 20 object categories, but they include chairs and sofas. Recognition is super fast, on the order of milliseconds, and works reliably in a handful of conditions. It’s also not restricted to newer devices.

There’s still work left to do, specifically adding placement and distance indication.

Credit

Matthijs Hollemans’s work porting YOLO to iOS made this possible. Also, thanks to Gus and Chancey for the usability feedback, and to the Recurse Center for being a great place to learn and experiment.