Engineering

Unicode normalisation and search in a rich text editor

The same café can be stored two ways and still fail to match. Normalise once at the boundary, build a search key, and never rewrite what the writer typed.

7 min read

A writer searches for “café” and the document that contains it is not found. The two look identical on screen, and they are different strings. One uses a single character for é, and the other uses an “e” followed by a combining accent. Text copied from a PDF, a Mac file name or a keyboard layout can produce either form, and a search that compares raw strings will miss matches in ways that are hard to explain to users.

The same letter, two encodings

Unicode allows several ways to write the same visible text. The composed form stores é as one code point, U+00E9. The decomposed form stores the letter e, U+0065, and then a combining acute accent, U+0301. Both render the same way, and === treats them as different:

Code example
const composed = "caf\u00E9";       // "café" with one code point
const decomposed = "cafe\u0301";   // "café" with a combining accent

composed === decomposed;            // false
composed.length;                    // 4
decomposed.length;                  // 5

composed.normalize("NFC") === decomposed.normalize("NFC"); // true

Normalising both sides to NFC, the composed form, makes them equal. The rule is simple: normalise text once when it enters the system, and compare normalised values everywhere else. A document saved from a Mac keyboard, a filename from macOS and a string pasted from a PDF can all arrive decomposed, so normalise at the boundary, not at the point of the search.

Store one form, search with a key

Normalising the stored content to NFC is safe, because it does not change how the text looks. Search needs more than that. Users expect a search for “cafe” to find “Café”, and a search for “2500” to find “2 500” with a non-breaking space. Build a separate search key from the stored text, and keep the stored text as the writer wrote it:

Code example
// NFKC maps compatibility characters to their plain forms: no-break spaces become
// regular spaces, the ligature "fi" becomes "fi", superscript "²" becomes "2".
function searchKey(text: string): string {
  return text
    .normalize("NFKC")
    .replace(/\s+/g, " ")
    .trim()
    .toLocaleLowerCase("en");
}

searchKey("Café\u00A0Latte") === searchKey("cafe latte"); // false: the accent still differs

That example shows a limit. NFKC normalises compatibility forms, but it does not remove accents, so “café” and “cafe” still differ. For accent-insensitive matching, use a collator, which knows about the letters of a language:

Code example
const baseCollator = new Intl.Collator("en", { sensitivity: "base" });

baseCollator.compare("café", "cafe") === 0;   // true: letters match, accents ignored
baseCollator.compare("cafe", "cafes") === 0;  // false

A collator is slower than a plain string comparison, so use it on the results of a search, not across the whole document on every keystroke. For a live search box, normalise the query once and run the comparison over the text blocks as the user types.

Whitespace is the sneakiest difference

Text in a document uses many spaces that look the same. The regular space, U+0020, is the one a keyboard types. The no-break space, U+00A0, keeps two words on one line, and French typography uses the narrow no-break space, U+202F, before a colon or inside a number. A search for “2 500” with a regular space will miss “2 500”. The normalisation above handles most of these. Tabs, line breaks and other control characters need their own rule, depending on whether they should count as separators.

Zero-width characters are the hardest to see. A soft hyphen (U+00AD) or a zero-width space (U+200B) can sit inside a word and break a match, and it can be invisible in the editor. Strip them from the search key, but leave them in the stored text unless you have a reason to remove them, because some of them are there on purpose.

Do not normalise away what the writer typed

There is a difference between normalising for storage and normalising for display. The NFC form is what the writer typed on most keyboards, so storing it is harmless. NFKC is different: it changes characters such as superscripts, ligatures and no-break spaces into plain ones. Applying it to the stored document rewrites the writer’s text. A footnote marked with a superscript “²” should still be a superscript after the document is saved.

Keep the two uses apart in the code. Store NFC. Build the search key with NFKC and case folding at query time, and discard it after the search.

A short test list

RichTextEditor is a perpetual-licence JavaScript editor — one purchase, self-hosted, no metered editor loads. Download the evaluation or see how it compares.