How we test our tools

The rule

A test that compares a tool against its own earlier output only proves the tool is consistent. It cannot tell you whether the tool was right in the first place. So every expected value in our suites comes from somewhere other than our code: the test vectors a standard publishes, another project's published test data, or arithmetic worked out by hand and written into the test. A few habits follow from that:

• Properties where a convention is arguable.Rather than hard-coding our own answer, a test checks something that has to hold — that VLSM allocations come out the same in any order, or that the JSON formatter's layout is byte-for-byte what JSON.stringify produces on thousands of random documents.
• Negative controls. A checker that never fails is worse than none, so new checks are shown to fail on deliberately broken input before they are trusted.
• No skipping on our own output. An early version of one sweep skipped cases where our output looked odd, and that hid a real bug. A strange result is a finding, not something to filter out.

What is tested, and against what

The exhaustive one

The domain-name converter has one more check that is too slow to run on every change: every Unicode code point, pushed through both our implementation and ICU — the library browsers use for the address bar. The last run compared 1,112,042 code points and agreed on 99.9862% of them, leaving 154 differences. All 154 are characters Unicode assigned in version 14.0 or later, which the ICU build used for comparison does not yet classify as right-to-left. None go the other way: there is no name the tool accepts that ICU refuses. The full account, including the two earlier runs and what changed between them, is in most punycode converters disagree with your browser.

What testing has found

Not every bug was found by a test, and it would be misleading to imply otherwise:

• Timestamp Converter: 90 minutes ago read “1 hour ago” while 90 minutes ahead read “in 2 hours”, because Math.round breaks ties upwards. Found by the test suite.
• Timestamp Converter: An unrecognised time zone threw an error during render instead of degrading, which would have blanked the tool. Found by the test suite.
• Punycode / IDN Converter: U+FE70 maps to a space, which UTS #46 calls valid, so the tool briefly produced domain labels with a space in them. Found by the ICU sweep.
• Punycode / IDN Converter: xn--7ba.com, which every browser resolves, was rejected, and an invisible U+200B was dropped with no warning. Found by the test suite.
• JSON Formatter: Formatting rounded integers past 2^53 — 1753524000123456789 came back as 1753524000123456800 — reordered numeric keys and dropped duplicate keys. Found by research for our post on nanosecond timestamps.

What is not tested

The automated suite covers the logic underneath the tools, not the screens on top of them. Interface behaviour — a slider, a copy button, a drag-and-drop zone — is checked by hand, and a bug in that layer has reached production at least once: dragging the password generator's length slider threw an error until it was reported and fixed. Tools whose work is done by the browser itself, such as image conversion through the canvas API or PDF handling through pdf-lib, are tested by using them rather than by vectors, because there is no published expected output for re-encoding a JPEG.

Check it yourself

Every source above is public. Take any vector from them — an RFC 4648 example, a Base58 pair from Bitcoin Core's file, a sample string from RFC 3492 — paste it into the matching tool, and compare. Because the tools run entirely in your browser, you can do that offline, and confirm in your browser's network panel that nothing you type is sent anywhere. A tool that disagrees with a published vector is wrong, whatever this page says.