Run a11y tests in CI - #310
Conversation
rwood-moz
left a comment
There was a problem hiding this comment.
That's cool that you're using the playwright container in CI!
Were you able to get the new a11y tests to run locally? Please update the main README Testing section to add how to run the a11y tests. I ran the tests locally :) I tried with:
npm ci
npx playwright install
npm run test:a11y
But I get this error stack:
rwood@Roberts-MacBook-Pro services-ui % npm run test:a11y
> @thunderbirdops/services-ui@1.6.7 test:a11y
> vitest run --project=storybook
(node:35382) [DEP0205] DeprecationWarning: `module.register()` is deprecated. Use `module.registerHooks()` instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
RUN v4.1.10 /Users/rwood/dev/services-ui
Failed to resolve dependency: react-dom/client, present in client 'optimizeDeps.include'
Failed to resolve dependency: react-dom/client, present in client 'optimizeDeps.include'
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Vitest caught 1 unhandled error during the test run.
This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected.
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Error ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Error: browserType.launch: Executable doesn't exist at /Users/rwood/Library/Caches/ms-playwright/chromium_headless_shell-1228/chrome-headless-shell-mac-arm64/chrome-headless-shell
╔════════════════════════════════════════════════════════════╗
║ Looks like Playwright was just installed or updated. ║
║ Please run the following command to download new browsers: ║
║ ║
║ npx playwright install ║
║ ║
║ <3 Playwright Team ║
╚════════════════════════════════════════════════════════════╝
❯ node_modules/@vitest/browser-playwright/dist/index.js:941:55
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { log: [] }
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Test Files (23)
Tests no tests
Errors 1 error
Start at 17:00:43
Duration 200ms (transform 0ms, setup 0ms, import 0ms, tests 0ms, environment 0ms)
(!) Failed to run dependency scan. Skipping dependency pre-bundling. Error: Failed to scan for dependencies from entries:
/Users/rwood/dev/services-ui/.storybook/preview.ts
/Users/rwood/dev/services-ui/src/foundation/CopyIcon.stories.ts
/Users/rwood/dev/services-ui/src/foundation/NoticeIcon.stories.ts
/Users/rwood/dev/services-ui/src/foundation/RefreshIcon.stories.ts
/Users/rwood/dev/services-ui/src/foundation/StatusIcon.stories.ts
/Users/rwood/dev/services-ui/src/patterns/CopyToClipboard.stories.ts
/Users/rwood/dev/services-ui/src/patterns/StandardFooter.stories.ts
/Users/rwood/dev/services-ui/src/components/BaseBadge.stories.ts
/Users/rwood/dev/services-ui/src/components/BaseButton.stories.ts
/Users/rwood/dev/services-ui/src/components/BubbleSelect.stories.ts
/Users/rwood/dev/services-ui/src/components/CheckboxInput.stories.ts
/Users/rwood/dev/services-ui/src/components/ExpiryBadge.stories.ts
/Users/rwood/dev/services-ui/src/components/LoadingSkeleton.stories.ts
/Users/rwood/dev/services-ui/src/components/ModalDialog.stories.ts
/Users/rwood/dev/services-ui/src/components/NoticeBar.stories.ts
/Users/rwood/dev/services-ui/src/components/SegmentedControl.stories.ts
/Users/rwood/dev/services-ui/src/components/SelectInput.stories.ts
/Users/rwood/dev/services-ui/src/components/SwitchToggle.stories.ts
/Users/rwood/dev/services-ui/src/components/SyncCard.stories.ts
/Users/rwood/dev/services-ui/src/components/TextArea.stories.ts
/Users/rwood/dev/services-ui/src/components/TextInput.stories.ts
/Users/rwood/dev/services-ui/src/components/ToolTip.stories.ts
/Users/rwood/dev/services-ui/src/components/UserAvatar.stories.ts
/Users/rwood/dev/services-ui/src/components/VisualDivider.stories.ts
Build failed with 24 errors:
[plugin vite:dep-scan:resolve]
Error: The server is being restarted or closed. Request is outdated
at throwClosedServerError (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30141:30)
at EnvironmentPluginContainer.resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30276:65)
at resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30983:38)
at resolve (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30995:22)
at PluginContextImpl.resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:31117:29)
at plugin (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/shared/bindingify-input-options-ClrST5Xx.mjs:1092:30)
at plugin.<computed> (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/shared/bindingify-input-options-ClrST5Xx.mjs:1625:18)
[plugin vite:dep-scan:resolve]
Error: The server is being restarted or closed. Request is outdated
at throwClosedServerError (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30141:30)
at EnvironmentPluginContainer.resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30276:65)
at resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30983:38)
at resolve (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30995:22)
at PluginContextImpl.resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:31117:29)
at plugin (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/shared/bindingify-input-options-ClrST5Xx.mjs:1092:30)
at plugin.<computed> (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/shared/bindingify-input-options-ClrST5Xx.mjs:1625:18)
[plugin vite:dep-scan:resolve]
Error: The server is being restarted or closed. Request is outdated
at throwClosedServerError (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30141:30)
at EnvironmentPluginContainer.resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30276:65)
at resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30983:38)
at resolve (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30995:22)
at PluginContextImpl.resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:31126:29)
at plugin (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/shared/bindingify-input-options-ClrST5Xx.mjs:1092:30)
at plugin.<computed> (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/shared/bindingify-input-options-ClrST5Xx.mjs:1625:18)
[plugin vite:dep-scan:resolve]
Error: The server is being restarted or closed. Request is outdated
at throwClosedServerError (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30141:30)
at EnvironmentPluginContainer.resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30276:65)
at resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30983:38)
at resolve (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30995:22)
at PluginContextImpl.resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:31117:29)
at plugin (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/shared/bindingify-input-options-ClrST5Xx.mjs:1092:30)
at plugin.<computed> (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/shared/bindingify-input-options-ClrST5Xx.mjs:1625:18)
[plugin vite:dep-scan:resolve]
Error: The server is being restarted or closed. Request is outdated
at throwClosedServerError (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30141:30)
at EnvironmentPluginContainer.resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30276:65)
at resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30983:38)
at resolve (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30995:22)
at PluginContextImpl.resolveId (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:31117:29)
at plugin (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/shared/bindingify-input-options-ClrST5Xx.mjs:1092:30)
at plugin.<computed> (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/shared/bindingify-input-options-ClrST5Xx.mjs:1625:18)
...
at aggregateBindingErrorsIntoJsError (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/shared/error-BuvQYXuZ.mjs:48:18)
at unwrapBindingResult (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/shared/error-BuvQYXuZ.mjs:18:128)
at scan (file:///Users/rwood/dev/services-ui/node_modules/rolldown/dist/experimental-index.mjs:181:3)
at Object.build (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30943:3)
at scan (file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:30887:4)
at file:///Users/rwood/dev/services-ui/node_modules/vite/dist/node/chunks/node.js:23353:15
error during close browserType.launch: Executable doesn't exist at /Users/rwood/Library/Caches/ms-playwright/chromium_headless_shell-1228/chrome-headless-shell-mac-arm64/chrome-headless-shell
╔════════════════════════════════════════════════════════════╗
║ Looks like Playwright was just installed or updated. ║
║ Please run the following command to download new browsers: ║
║ ║
║ npx playwright install ║
║ ║
║ <3 Playwright Team ║
╚════════════════════════════════════════════════════════════╝
at /Users/rwood/dev/services-ui/node_modules/@vitest/browser-playwright/dist/index.js:941:55 {
log: [],
name: 'Error',
type: 'Unhandled Error',
stacks: [
{
method: '',
file: '/Users/rwood/dev/services-ui/node_modules/@vitest/browser-playwright/dist/index.js',
line: 941,
column: 55
}
]
}
rwood@Roberts-MacBook-Pro services-ui %
|
@rwood-moz ah good point, thanks! I've ran it locally by manually installing playwright chromium but forgot to add a setup script in |
Thank you for the updates, appreciated. Unfortunately I'm still seeing the error above, mainly:
@devmount Are you able to run the a11y tests successfully on your machine? I'm on a MacBook M3 Pro w/Tahoe 26 |
NVM, I found the issue. Two different versions of Playwright, so the wrong browser version was installed. I changed package.json:41 to
and did |
There was a problem hiding this comment.
Interesting! I've updated it here and tests still seems to pass. Thanks!
|
@rwood-moz I'm currently testing on my machine (TuxedoOS, a Ubuntu 24 derivate), but it already told me my OS is not officially supported by Playwright. Let's see |
It's ok, I found the issue (see above) thank you anyway @devmount appreciated! |
|
@rwood-moz No worries, for some reason the chromium installation won't finish on my machine. But good if you found it! |
When using playwright 1.59.1 or below and Node 24:16+ the playwright browser install hangs at 100%, it's a known playwright issue. |
rwood-moz
left a comment
There was a problem hiding this comment.
Thanks for the update @davinotdavid. Yep works great now!
Test Files 23 passed (23)
Tests 113 passed (113)
Start at 17:12:39
Duration 4.71s (transform 0ms, setup 18.22s, import 6.83s, tests 6.88s, environment 0ms)
What changed?
test:a11yscript to run a11y tests.validate.ymlCI workflow to run the tests.BaseButtonandSyncCardhad colour contrast issues. For now, keeping the same colour variable but doing acolor-mixto darken it a bit.UserAvatarcomponent used an<aside />but changed to a normal<div />so that we can fix the "Landmarks should have a unique role or role/label/title (i.e. accessible name) combination (landmark-unique)" issue.SelectInputcomponent stories need to have a label.Why?
Limitations and Notes
Applicable Issues
Closes #309
How to test