Semantic Component Testing

An Alternative to Visual Regression that Puts Usability First

Familiar Component Testing Approaches

• DOM Property Testing
• DOM Snapshots
• Visual Regression

DOM Property Testing

When I call the open() method, the expanded DOM property should be true.

DOM Property Testing PROs

• Easy-to-write assertions
• Focused on public state and methods

DOM Property Testing CONs

• Low degree of confidence
• Can only assert one aspect at a time
• No insight into the DOM, accessibility tree, or styles

DOM snapshots

When I call the open() method, the shadow root's HTML should match the last known-good snapshot.

DOM Snapshot PROs

• High degree of confidence
• Validates the entire DOM, holistically
• Assertions are even simpler to write

DOM Snapshot CONs

• Tightly coupled to (private) DOM structure
• Manual snapshot validation
• Can only validate serializable state

// setting default semantics
this.#internals.ariaLabel = "Pick a card"

this.#comboboxElement // shadow
  .ariaActiveDescendantElement =
    this.querySelector('x-option[active]'); // light

Some of these issues can be resolved by normalizing the actual and expected HTML with pkgs like semantic-dom-diff

Visual Regression

When I call the open() method, after all animation stops, the element should match the last known-good screenshot to within a 1% margin of error.

Visual Regression PROs

• Extremely high confidence
• Direct visual feedback is great for reviewers

Visual Regression CONs

• No insight into DOM or AX tree
• Can be flaky esp. cross-platform
• Same maintenance issues as DOM snapshots

Accessibility Audits

Does the content of the page pass all automated aXe audits?

Accessibility Audits - PROS

• Good for whole pages / apps
• Provides comprehensive reports

Accessibility Audits - CONS

• Performance issues
  • Runs all tests every time
• May not have access to in-memory state
  • Potential for false positives

The Accessibility Tree

The Accessibility Tree - Playwright

• Playwright provides access to the browser's AX Tree
• It's not perfect, but it provides lots of info

{
  "role": "WebArea",
  "name": "",
  "children": [{
    "role": "combobox",
    "name": "options",
    "expanded": true,
    "focused": true,
    "autocomplete": "both",
    "haspopup": "listbox"
  }, {
    "role": "button",
    "name": "options",
    "expanded": true
  }, {
    "role": "listbox",
    "name": "options",
    "orientation": "vertical",
    "children": [{
      "role": "option",
      "name": "Select an Option",
      "disabled": true
    }, {
      "role": "option",
      "name": "1"
    }, {
      "role": "option",
      "name": "2"
    }, {
      "role": "option",
      "name": "3"
    }, {
      "role": "option",
      "name": "4"
    }, {
      "role": "option",
      "name": "5"
    }, {
      "role": "option",
      "name": "6"
    }, {
      "role": "option",
      "name": "7"
    }, {
      "role": "option",
      "name": "8"
    }, {
      "role": "option",
      "name": "9"
    }, {
      "role": "option",
      "name": "10"
    }]
  }]
}

Semantic Assertions

Instead of asserting on the state of the DOM, or on the state of the visual renderer, assert on the state of the accessibility tree

 {
   "role": "combobox",
   "name": "options",
-  "expanded": false,
+  "expanded": true,
   "focused": true,
   "autocomplete": "both",
   "haspopup": "listbox"
 }

When I click the combobox toggle, assistive technology should have access to the listbox

Semantic Assertions - Snapshots

Same issues as other kinds of snapshot testing

describe('clicking the first button', function() {
  beforeEach(clickButton1);
  it('remains closed', async function() {
    expect(await a11ySnapshot()).to.deep.equal({
      name: '',
      role: 'WebArea',
      children: [
        { role: 'button', name: 'Toggle 1', focused: true },
        { role: 'button', name: 'Toggle 2' },
      ],
    });
  });
});

Semantic Assertions - Mocha / Chai

describe('click combobox button', function() {
  beforeEach(() => clickElementAtCenter(element));
  it('expands the listbox', async function() {
    expect(await a11ySnapshot())
      .to.axContainRole('listbox');
  });
  it('focuses on the first item', async function() {
    expect(await a11ySnapshot())
      .axTreeFocusedNode
      .to.have.axName('1');
  });
});

Semantic Assertions - PROS

• Puts usability first
• The curb cut effect

the curb cut here is that we can test for correctness based on effects, rather than by asserting on the private DOM structure

Semantic Assertions - CONS

• Playwright's AX Tree may not deliver all features
• Playwright wants to deprecate 😨

Tradeoffs

Thanks