Do design systems deliver accessibility at scale?

Design systems promise consistency, and their maintainers increasingly promise accessibility: components ship with semantics, focus management, and contrast decisions already made. This study quantifies that promise at estate scale under a pre-registered protocol. On a frozen snapshot of 12,252 live US federal websites from the GSA Site Scanning programme, sites with strong US Web Design System adoption signals show roughly half the automatically detectable accessibility violations of sites without them - incidence-rate ratio 0.50 (95% CI 0.38-0.65) - after agency fixed effects and digital-maturity controls, with a monotone dose-response across graded adoption bands.

With the specification frozen before a single UK site was scanned, the same model was then applied once to a held-out replication sample collected for this paper: 6,295 UK public-sector websites (central government, local authorities, parish and town councils, NHS) scanned with a govuk-frontend detector built as a structural mirror of GSA's USWDS score, plus the same axe-core ruleset. The association replicated in both direction and magnitude: IRR 0.56 (95% CI 0.47-0.67), within the pre-registered +/-0.20 window of the US estimate - consistent, at the level automated checks can measure, with the Government Digital Service's claim that the GOV.UK Design System helps services ship more accessible front ends.

Pre-registered diagnostics point to a site-level association rather than a pure good-teams artifact: the gradient survives within-agency comparison with only 20% attenuation, exceeds placebo gradients on performance metrics, and is stable under Oster-style selection bounds. Not everything replicated: USWDS v3 shows no improvement over v2, the predicted component-vs-template category specificity did not appear, and the UK partial-adoption band - where detection is noisiest - reverses sign; all three are reported as found. The design is observational and the outcome is the machine-detectable floor of accessibility, not the whole of it. All data, code, results, and the locked pre-registration are public and explorable in the browser.

Design systems are the largest coordinated investment in front-end quality that governments and large organisations make. Their maintainers increasingly sell them on accessibility: components ship with semantics, focus management, labelled controls, and contrast decisions already made, so teams “inherit” conformance instead of re-deriving it. The US Web Design System (USWDS) is mandated guidance for federal agencies under 21st Century IDEA; the GOV.UK Design System tells UK service teams it helps them “get to WCAG 2.2 faster”. That promise is plausible, widely repeated — and has never been quantified at estate scale.

This paper asks a narrow, answerable version of the question: is graded design-system adoption associated with fewer automatically detectable accessibility violations on production government websites? We estimate the association twice, in two countries, under a pre-registered protocol: first on ~12,000 US federal websites whose USWDS adoption and axe-core violations are scanned daily by GSA’s Site Scanning programme, then — with the specification frozen — once on a held-out replication sample of UK public-sector websites that we scanned ourselves with an openly specified govuk-frontend detector and the same violation taxonomy.

The design follows the discovery/confirmation split used in quantitative social science. Descriptive patterns were explored on the US snapshot; hypotheses, exact model specifications, decision rules, and the analyst’s priors were then locked in a public pre-registration before any confirmatory estimate was computed, and before a single UK site was scanned. The UK scan recipe — domain sources, detection weights, band boundaries, violation mapping — was committed before data collection, so the replication could not be tuned to the data it would meet. Automated checks (axe-core) capture only a machine-detectable subset of accessibility; that floor is precisely what design systems claim to raise, which makes it the right yardstick for this question.

2.1 United States — GSA Site Scanning

The discovery sample is a frozen snapshot of the GSA Site Scanning programme, which scans every known live federal .gov/.mil website daily. One row per live primary website with a completed accessibility scan and non-missing agency: 12,252 sites across 100+ agencies. The exposure is uswds_count, GSA’s graded USWDS detection score (CSS class prefixes, asset paths, fonts, favicon, version strings), banded as none (0), trace (1–24), partial (25–49), likely (50–99), definite (100+); “strong” adoption means a score ≥ 50. The outcome is violations_total, the sum of axe-core violations across GSA’s tracked categories on the scanned page, with 11 category-level counts retained.

2.2 United Kingdom — a new scan, collected for this paper

No UK equivalent of Site Scanning publishes per-site data, so we built one. Following the frozen UK scan recipe, we scanned the homepage of every reachable site in a universe assembled from the official .gov.uk domains register, the mySociety local-authority register, NHS trust websites (via Wikidata), and a curated devolved-government list. Each site was loaded in headless Chromium, classified into one of five organisation types (central, local authority, parish & town, NHS, devolved), scored with a govuk-frontend detector that mirrors GSA’s uswds_count formula component-for-component (same weights, caps, and band boundaries), and audited with the same axe-core ruleset mapped to GSA’s category names. Detection correctness was verified against a pre-declared 24-site calibration set, excluded from analysis.

The frozen UK artifact covers 6,295 completed, deduplicated sites (of 8,136 scan attempts), scanned 2026-06-12T16:07:13Z – 2026-06-12T18:35:40Z.

All confirmatory models are Poisson quasi-maximum-likelihood (PPML) regressions of violation counts, with standard errors clustered at the organisation level. PPML is consistent for the conditional mean under overdispersion and handles the long right tail of violation counts without dropping the roughly one-third of sites that have zero violations.

• H1 (dose–response, primary, US): violations_total ~ ln(1 + uswds_count) + controls + agency FE, β < 0 at one-sided p < .05; in the banded form, IRR(strong vs none) < 1 with band IRRs weakly monotone non-increasing from none → likely.
• H2 (version contrast, US): among the ~900 sites with a detected semantic version, v3 vs v2 within adopters, same controls and fixed effects.
• H3 (category specificity, US): the H1 model estimated separately per violation category; prediction that the mean standardized gradient is more negative for component-mediated categories (contrast, ARIA, form labels, control names, link purpose, keyboard access) than template/content-mediated ones (language, page titles, images, lists, frames). Tested with a 2,000-draw cluster bootstrap over agencies.
• H4 (held-out UK replication, gating): the frozen specification applied once to the UK artifact — exposure govuk_count, organisation-type fixed effects, HTTPS-hygiene and page-complexity controls, SEs clustered by registrant organisation. Prediction: IRR(strong vs none) < 1, one-sided p < .05, and the UK IRR within ±0.20 of the US IRR.

Controls in the US models: Digital Analytics Program participation (maturity proxy), security-hygiene index (HTTPS enforcement + HSTS), viewport meta tag, asinh of the third-party service count, and CMS indicators. The paper’s core claim stands only if H1 and H4 both hold; H2 and H3 enrich the mechanism story but do not gate it. Exact decision rules, priors, and six pre-registered diagnostics are in the locked pre-registration.

H1 — Dose–response

The continuous gradient is negative: each unit of ln(1 + uswds_count) is associated with an IRR of 0.88 (95% CI 0.82–0.94) (p < .001, one-sided), holding agency and maturity controls fixed. Strong adoption (score ≥ 50) versus everything below that threshold gives IRR 0.50 (95% CI 0.38–0.65) — -50% detected violations — p < .001; the none-referenced band contrasts say the same thing (likely 0.46, definite 0.49; see appendix D3 on this operationalization). Band IRRs are weakly monotone non-increasing from none to likely. H1 is supported under the pre-registered decision rule.

H2 — Version contrast

Among sites with a detected semantic version, USWDS v3 vs v2 gives IRR 1.50 (95% CI 0.95–2.37) (p = .958, one-sided; N = 877). H2 is not supported; as pre-registered, we interpret the IRR and CI rather than the p-value alone given the small contrast sample.

H3 — Category specificity

Across the 11 violation categories, the mean standardized adoption gradient is -0.181 for component-mediated categories and -0.268 for template/content-mediated ones (difference 0.087, one-sided p = .767 from the agency-cluster bootstrap). H3 is not supported.

1. Who adopts? Regressing adoption on the maturity proxies shows the expected self-selection: dap +0.182 (p < .001); hygiene +0.014 (p = .448); viewport_meta_tag +0.096 (p = .002); asinh_tp +0.012 (p = .447) — within-agency R² 0.234. Adoption is not random; the remaining diagnostics ask whether that selection explains the H1 gradient.
2. Placebo outcomes. The standardized accessibility gradient (-0.197) is larger in magnitude than the placebo gradients on layout shift (0.092, p = .028) and load time (-0.066, p = .051). A generic good-teams confound that improves everything equally does not explain the accessibility-specific gradient, though one of the bootstrap comparisons is borderline and we flag it as such.
3. Pooled vs within-agency. The pooled IRR (0.37) attenuates to 0.50 under agency fixed effects — 20% of the distance to the null. Below the pre-registered 75% collapse threshold: a within-agency, site-level association survives.
4. Oster bounds. Moving from no controls (β = -0.092, R² = 0.037) to the full specification (β = -0.094, R² = 0.205) gives δ = 1.00 and a bias-adjusted β* = -0.095 at Rmax = 0.27. Unobservables would need to be implausibly stronger than the observed controls to drive the estimate to zero.
5. DAP subset. Within Digital Analytics Program participants — a uniformly “engaged” subsample — the continuous gradient is IRR 0.87 (95% CI 0.80–0.95) (p < .001, N = 5,909).
6. Functional form. A negative-binomial fit of the strong-vs-none contrast gives IRR 0.35 (95% CI 0.33–0.38). Winsorizing the outcome at the 99th percentile gives 0.87 (95% CI 0.83–0.92) for the continuous term. The banded and continuous forms agree in direction and ordering.

Strong govuk-frontend adoption versus everything below the strong threshold gives IRR 0.56 (95% CI 0.47–0.67) (p < .001, one-sided), with organisation-type fixed effects and the pre-registered controls — the same contrast construction as the US headline, keeping the comparison window like-for-like. The continuous gradient is 0.92 (95% CI 0.88–0.97) per unit of ln(1 + govuk_count). The direction prediction replicates.

The pre-registered magnitude window asked whether the UK IRR lands within ±0.20 of the US estimate. US: 0.50; UK: 0.56; absolute difference 0.07 — within the window. H4 is supported under the locked decision rule, so the paper's gating claim (H1 ∧ H4) stands.

One band breaks the pattern, and it is the one the pre-registration flagged as the riskiest: partial adoption (score 25–49) shows IRR 4.05 (95% CI 2.35–6.97) — more violations than the none band. This is the “council CMS noise” stratum: platforms that embed fragments of govuk-frontend styling without its components land here, disproportionately local authorities and parish-council CMS products whose underlying templates differ from genuine adopters in every other way too. The dose–response claim in the UK is carried by the likely and definite bands — sites where detection is unambiguous — exactly as the structural score construction intends. We report the reversal rather than smooth it; it is the clearest single illustration of why band-level detection noise is listed as a limitation.

• Observational throughout. No agency or council was randomized into adoption. The diagnostics bound, but cannot eliminate, selection: teams that adopt a design system differ from teams that don’t in ways the maturity proxies only partly capture.
• Automated detection is a floor, not a ceiling. axe-core catches roughly a third to a half of WCAG failures and almost nothing about task completion, cognitive load, or assistive-technology usability. “Fewer detected violations” is the claim design systems make mechanically; it is not the same as “accessible”.
• Homepage-only sampling (UK). GSA scans primary URLs; our UK scan mirrors that. Homepages are disproportionately design-system shells; deep transactional pages may behave differently.
• Detection noise. uswds_count and govuk_count measure observable adoption signals, not true component usage; JS-heavy sites and councils on CMS platforms with partial govuk styling blur the bands. Measurement error of this kind generally attenuates gradients toward zero.
• Two estates, one regime each. Both samples are government estates under accessibility law (Section 508; PSBAR 2018). Generalisation to commercial design systems is conjecture until tested.

• D1 — Calibration-list repair (before any analysis). Eight of the 24 pre-declared calibration entries were wrong about the world, not about the detector: dead hostnames, WAF-blocked sites, and two sites whose adoption status was mislabelled. They were replaced with verified equivalents; no detector weights, thresholds, or band boundaries changed. The detector then passed 12/12 known adopters and 12/12 non-adopters.
• D2 — www fallback for apex connection failures (during the scan). Many UK estates serve only the www. host while the apex domain refuses connections. Where an apex attempt failed at connection level, the scanner retried once with the www. prefix and flagged the row (used_www_fallback). This mirrors the redirect-following behaviour GSA’s scanner gets for free and was adopted before any outcome data was examined.
• D3 — Labelling of the headline contrast (after the US run, before any UK estimation). The single-number “strong” contrast was implemented as a binary ≥ 50 dummy, whose reference group is every site below 50 — not literally the none band. The pre-registered banded model, estimated alongside, gives the none-referenced IRRs directly and they bracket the dummy estimate, so nothing turns on the choice. Labels and prose were corrected to say “vs below-50”; no model, sample, or decision rule changed.

Full deviations log.