offlineutils
Glossary
Plain English
Cross-linked to tools

Unicode normalization

Also known as: Unicode normalisation, NFC, NFD, NFKC, NFKD, UAX #15, Unicode equivalence

Unicode normalization rewrites text into one standard sequence of code points, so strings that look identical — é stored as a single precomposed character, or as e followed by a combining accent — compare as equal. Unicode defines four normalization forms: NFC, NFD, NFKC and NFKD.

Overview

Unicode often has more than one way to write the same thing. é can be the single code point U+00E9, or U+0065 followed by the combining acute accent U+0301. The two render identically and mean the same, but they are different bytes, so a plain string comparison, a hash, a database lookup or a domain-name conversion treats them as different strings. Normalization picks one representation so that equal text is also equal data.

Unicode recognises two kinds of sameness, and the four forms come from combining them with a direction. Canonical equivalence covers genuinely identical characters encoded differently: precomposed é and e plus its accent, the Ångström sign U+212B and the letter Å, a Hangul syllable like 한 and its three component jamo. NFD decomposes everything into base characters and combining marks; NFC decomposes and then recomposes wherever a precomposed character exists, which makes it the shorter form, the one most text is already in, and the default for JavaScript's normalize(). Compatibility equivalence is looser: it also equates characters that differ only in formatting, such as the fi ligature and 'fi', fullwidth Ａ and A, superscript ² and 2, or circled ① and 1. NFKD and NFKC fold those as well, which is why they suit matching rather than storage — x² becomes x2, and nothing can turn it back.

The forms show up wherever text is compared. Domain names: UTS #46 maps a name and then normalises it to NFC, so a decomposed café still reaches the same domain as a precomposed one. URLs: percent-encoding the two spellings gives caf%C3%A9 and cafe%CC%81, two different paths to a server that does not normalise. Filenames: HFS+, the Mac file system until 2017, stored every name decomposed and APFS keeps whichever form a name was created in, so a file copied off an older Mac may not match the same name typed elsewhere. Identifiers: Python normalises identifiers with NFKC, so a variable written with the fi ligature is the same variable as 'file'. Passwords and usernames: RFC 8265, the PRECIS profile for both, requires NFC; its predecessor SASLprep used NFKC.

Normalization changes length, which catches out validation code. é is one code point in NFC and two in NFD, and NFKC can expand a single character dramatically — U+FDFA, an Arabic ligature, becomes 18 characters. Normalise first, then measure, truncate or enforce limits. The results are stable across versions: Unicode's stability policy guarantees that a string normalised under one version of the standard stays normalised under every later one, so normalised data never has to be migrated when Unicode is updated.

Common questions about Unicode normalization

What is the difference between NFC and NFD?
NFD splits precomposed characters into a base character plus combining marks, so é becomes e followed by U+0301 and the Hangul syllable 한 becomes three jamo. NFC does the same and then recombines wherever a precomposed character exists. Both are lossless and represent exactly the same text. NFC is shorter, is what most software produces and expects, and is JavaScript's default, so use it unless a particular system requires NFD.
When should I use NFKC instead of NFC?
When you are matching or validating identifiers and want formatting variants to count as the same thing: usernames, search keys, programming-language identifiers (Python uses NFKC), and the character mapping behind domain names. Do not use it to store what people wrote. NFKC is deliberately lossy — x² becomes x2, ① becomes 1, the fi ligature becomes fi — and the original cannot be recovered.
Why don't two identical-looking strings match?
Most often because one contains a precomposed character and the other a base letter plus a combining mark, which is common in text copied from macOS filenames and from some keyboards. Normalising both to NFC before comparing, hashing or using them as keys fixes it. Normalization does not fix two other causes: invisible characters such as the zero-width space U+200B, and letters from different scripts that merely look alike, like Cyrillic а and Latin a. Those are confusables, a separate problem.
Does normalization change the length of a string?
Yes. é is one code point in NFC and two in NFD, and NFKC can expand a single character into many: U+FDFA, one Arabic ligature, becomes 18 characters. Normalise before enforcing length limits or truncating, and truncate on character boundaries rather than code units, or you can cut a base letter off from its accent.
Should passwords be normalised before hashing?
Yes, and consistently. Without it, the same password typed on two devices can produce different bytes, and therefore a different hash, so a correct password fails to match. RFC 8265, the PRECIS standard for usernames and passwords, applies NFC to passwords; the older SASLprep applied NFKC. What matters most is using the same form at registration and at every login, before the password is hashed.

Tools that work with Unicode normalization

Punycode / IDN Converter

Convert between Unicode domain names and xn-- Punycode, with full UTS #46 and homograph checks.

URL Slug Generator

Turn any title into a clean, URL-safe slug.

URL Encoder / Decoder

Percent-encode and decode URL components instantly.

Related concepts

External references