Text normalization

Short description: Process of transforming text into a single canonical form

Not to be confused with word normalization or Unicode normalization.

Text normalization is the process of transforming text into a single canonical form that it might not have had before. Normalizing text before storing or processing it allows for separation of concerns, since input is guaranteed to be consistent before operations are performed on it. Text normalization requires being aware of what type of text is to be normalized and how it is to be processed afterwards; there is no all-purpose normalization procedure.

Applications

Text normalization is frequently used when converting text to speech. Numbers, dates, acronyms, and abbreviations are non-standard "words" that need to be pronounced differently depending on context. For example:

• "$200" would be pronounced as "two hundred dollars" in English, but as "lua selau tālā" in Samoan.
• "vi" could be pronounced as "vie," "vee," or "the sixth" depending on the surrounding words.

Text can also be normalized for storing and searching in a database. For instance, if a search for "resume" is to match the word "résumé," then the text would be normalized by removing diacritical marks; and if "john" is to match "John", the text would be converted to a single case. To prepare text for searching, it might also be stemmed (e.g. converting "flew" and "flying" both into "fly"), canonicalized (e.g. consistently using American or British English spelling), or have stop words removed.

Techniques

For simple, context-independent normalization, such as removing non-alphanumeric characters or diacritical marks, regular expressions would suffice. For example, the sed script sed ‑e "s/\s+/ /g"  inputfile would normalize runs of whitespace characters into a single space. More complex normalization requires correspondingly complicated algorithms, including domain knowledge of the language and vocabulary being normalized. Among other approaches, text normalization has been modeled as a problem of tokenizing and tagging streams of text and as a special case of machine translation.

Unicode-aware text normalization in software

Text normalization in software involves several distinct operations that should not be confused with one another. Unicode normalization addresses equivalence between certain encoded character sequences, while natural-language normalization may involve case conversion, spelling variants, tokenization, abbreviation expansion, or other language-dependent transformations.

An implementation should distinguish these operations because they preserve different kinds of information and establish different notions of equality.

Encoding conversion versus normalization

An encoding converts character data between a representation of Unicode characters and a sequence of code units or bytes.

For example, decoding a UTF-8 file produces a sequence of Unicode characters. Re-encoding those characters as UTF-8 does not, by itself, perform NFC or NFD normalization.

Consider the character é:

Precomposed form:
U+00E9
LATIN SMALL LETTER E WITH ACUTE

Decomposed form:
U+0065 U+0301
LATIN SMALL LETTER E
COMBINING ACUTE ACCENT

Both sequences can represent the same abstract text under Unicode canonical equivalence.

However, their UTF-8 representations are different:

U+00E9:
C3 A9

U+0065 U+0301:
65 CC 81

A program comparing raw UTF-8 bytes will find these sequences different.

Converting both strings to UTF-8 does not resolve this difference, because UTF-8 is an encoding form rather than a Unicode normalization algorithm.

Unicode normalization provides standardized transformations for dealing with such equivalent representations.

Canonical and compatibility normalization

Unicode defines four principal normalization forms:

Form	Transformation	Main property
NFC	Canonical decomposition followed by canonical composition	Preserves compatibility distinctions
NFD	Canonical decomposition and canonical ordering	Preserves compatibility distinctions
NFKC	Compatibility decomposition followed by canonical composition	May remove compatibility distinctions
NFKD	Compatibility decomposition and canonical ordering	May remove compatibility distinctions

Canonical normalization can bring canonically equivalent strings into the same representation.

For example:

NFC(U+0065 U+0301) = U+00E9

NFD(U+00E9) = U+0065 U+0301

Compatibility normalization is broader.

Consider:

U+0041
LATIN CAPITAL LETTER A

U+1D400
MATHEMATICAL BOLD CAPITAL A

These are distinct Unicode characters.

NFC preserves the distinction, whereas NFKC maps mathematical bold capital A to ordinary capital A.

This behavior is useful for some search and matching applications, but it can destroy distinctions intended by the original text.

For example, mathematical typography can use different alphabetic styles to distinguish variables or classes of objects. Removing that distinction may alter the interpretation of a formula.

Therefore, compatibility normalization should not automatically be applied to all stored text.

The choice of normalization form is part of an application's data and comparison policy.

Normalization and case folding

Case conversion is another operation that is distinct from Unicode normalization.

Lowercasing is not always sufficient to implement general Unicode caseless matching.

Unicode case folding provides mappings intended for caseless comparison.

For example, the German word:

Straße

and the uppercase spelling:

STRASSE

can produce equal comparison keys under Unicode default case folding.

Case folding may expand one character into multiple characters. Consequently, a normalized or case-folded string can have a different number of code points from the original.

A case-folded search index should not automatically replace the original display text.

Locale-sensitive text matching may require additional rules beyond default Unicode case folding.

For example, handling Turkish dotted and dotless I can require a language-specific policy.

Text normalization for search and indexing

A text-search system may maintain separate representations of the same original input.

One representation preserves the source text, while another is constructed for searching or comparison.

A possible processing pipeline is:

Input bytes
    |
    v
Decode using the specified encoding
    |
    v
Preserve original Unicode text
    |
    v
Apply selected normalization policy
    |
    v
Apply case-folding policy if needed
    |
    v
Segment or tokenize for the application
    |
    v
Construct search index

The operations should not be treated as universally interchangeable.

For example:

• NFC can reconcile canonically equivalent representations.
• Case folding can support caseless matching.
• Accent-insensitive matching may require a separate, explicitly defined transformation.
• Tokenization identifies units such as words or terms.
• Stemming and lemmatization are language-processing operations.
• Transliteration changes the writing-system representation.
• Spelling correction changes text according to a linguistic or statistical model.

An application may need several of these operations, but their ordering and consequences should be documented.

The original source text should be preserved where exact reconstruction, auditing, digital signatures, provenance, or faithful display is required.

Python implementation

Python provides Unicode normalization functions through the standard-library unicodedata module.

The following example separates canonical equality from a simple caseless comparison policy.

import unicodedata as ud


def canonical_key(text: str) -> str:
    return ud.normalize("NFC", text)


def caseless_key(text: str) -> str:
    canonical = ud.normalize("NFC", text)
    folded = canonical.casefold()
    return ud.normalize("NFC", folded)


precomposed = "\u00e9"
decomposed = "e\u0301"

print(precomposed == decomposed)

print(
    canonical_key(precomposed)
    == canonical_key(decomposed)
)

print(
    caseless_key("Straße")
    == caseless_key("STRASSE")
)

bold_a = "\U0001D400"

print(
    ud.normalize("NFC", bold_a) == "A"
)

print(
    ud.normalize("NFKC", bold_a) == "A"
)

Expected output:

False
True
True
False
True

The example demonstrates three separate properties:

1. Canonically equivalent strings may have different original code-point sequences.
2. Default case folding can provide caseless comparison keys.
3. Compatibility normalization can remove distinctions preserved by NFC.

The functions above define particular comparison policies. They do not constitute a complete locale-sensitive collation algorithm or a security-oriented Unicode identifier profile.

Accent-insensitive search is not canonical equivalence

Accent-insensitive matching is sometimes described informally as normalization, but it is not equivalent to Unicode canonical normalization.

For example, a search system may intentionally match:

resume
résumé

This is broader than canonical equivalence.

NFC does not remove accents from résumé.

A system wishing to ignore accents must define an additional matching policy.

Removing combining marks after decomposition can be useful in some contexts, but it is not a universal multilingual solution.

Some letters do not decompose into a simple base letter plus removable combining marks. In other languages, marks are essential to the identity or pronunciation of text.

Therefore, indiscriminate removal of combining marks can produce incorrect results or discard meaningful linguistic distinctions.

An accent-insensitive index should generally be distinguished from the source text used for display.

Grapheme clusters and text segmentation

A Unicode code point is not necessarily equivalent to one user-perceived character.

Some characters are represented using multiple code points, including:

• Letters followed by combining accents.
• Emoji with skin-tone modifiers.
• Emoji zero-width-joiner sequences.
• Certain Indic-script conjunct sequences.
• Regional-indicator pairs used for flags.

Unicode Standard Annex #29 defines default grapheme-cluster boundaries and other text-segmentation rules.

Grapheme segmentation is different from normalization.

For example, NFC may combine a base letter and an accent into one precomposed code point when such a composition is defined.

However, NFC does not guarantee that every grapheme cluster consists of exactly one code point.

A text-processing pipeline that counts, truncates, or reverses user-perceived characters may require grapheme-aware algorithms even after normalization.

This distinction matters for:

• Text editors.
• Cursor movement.
• Search-result highlighting.
• String-length validation.
• Truncation of user-generated content.
• Unicode-aware approximate string matching.

The correct unit of processing depends on the application's requirements.

Normalization across concatenation boundaries

Normalization introduces an important implementation consideration when processing text incrementally.

Even if two strings are individually normalized, their concatenation is not guaranteed to remain normalized.

For example:

NFC("e") = "e"

NFC(U+0301) = U+0301

Concatenating those results gives:

U+0065 U+0301

However, NFC applied to the complete concatenated sequence produces:

U+00E9

A Python demonstration is:

import unicodedata as ud

left = ud.normalize("NFC", "e")
right = ud.normalize("NFC", "\u0301")

combined = left + right

print(
    combined == ud.normalize("NFC", combined)
)

print(
    ud.normalize("NFC", combined) == "\u00e9"
)

Expected output:

False
True

Therefore, separately normalized input chunks should not simply be concatenated with the assumption that the result is still in the same normalization form.

Streaming normalization implementations must account for boundaries between chunks.

The Unicode normalization standard specifies the relevant rules and conformance requirements.

Database consistency and preservation of source text

Database equality depends partly on the database's collation and comparison settings.

Two canonically equivalent Unicode strings can compare differently under binary comparison.

Conversely, accent-insensitive or case-insensitive collations may consider strings equal even when they are not canonically equivalent.

A robust data model may store:

• The original submitted text.
• A separately computed canonical comparison key.
• An optional caseless or search-oriented key.
• Metadata identifying the transformation policy.

This design allows search behavior to evolve without necessarily losing the original content.

It is also useful for investigating differences between software components that use different Unicode library versions or collation settings.

For applications involving digital signatures, the exact encoded byte sequence must be defined by the relevant protocol. Unicode normalization must not be inserted into a signing pipeline unless that transformation is explicitly specified.

Security and confusable characters

Unicode normalization does not solve every problem involving visually similar text.

Characters from different scripts may resemble one another without being canonically or compatibly equivalent.

For example, Latin and Cyrillic alphabets contain distinct characters with similar visible forms.

Likewise, some invisible Unicode characters can affect comparison, rendering, or text direction.

Security-sensitive applications may require additional policies for:

• Permitted character repertoires.
• Mixed-script identifiers.
• Confusable-character detection.
• Bidirectional controls.
• Default-ignorable code points.
• Identifier-specific validation.

Such policies should be kept separate from ordinary text normalization.

See Unicode Technical Standard #39: Unicode Security Mechanisms.

Testing and reproducibility

Unicode-aware normalization should be tested using exact code-point sequences rather than screenshots alone.

Useful test cases include:

Input or operation	Expected result	Purpose
NFC applied to U+0065 U+0301	U+00E9	Canonical composition
NFD applied to U+00E9	U+0065 U+0301	Canonical decomposition
NFC applied to U+1D400	U+1D400	Preservation of compatibility distinctions
NFKC applied to U+1D400	U+0041	Compatibility mapping
Case folding of Straße and STRASSE	Equal comparison keys under default folding	Caseless matching
Separately normalized e and U+0301 followed by concatenation	Combined result requires NFC normalization	Stream-boundary behavior

For conformance testing, implementations should use the version-matched Unicode Character Database and the normalization test data supplied by the Unicode Consortium.

The version of the Unicode data used by the runtime should also be recorded when reproducibility matters.

Unicode normalization guarantees specific equivalence transformations. It does not guarantee identical font rendering, language-sensitive spelling equivalence, or complete protection against visually confusable identifiers.

Further technical reading

The Unicode Consortium's normalization specification is the normative reference for NFC, NFD, NFKC and NFKD.

A supplementary technical explanation, including implementation examples, canonical combining order, database comparison and normalization test cases, is available in Unicode Normalization: NFC, NFD and Canonical Equivalence.

Textual scholarship

In the field of textual scholarship and the editing of historic texts, the term "normalization" implies a degree of modernization and standardization – for example in the extension of scribal abbreviations and the transliteration of the archaic glyphs typically found in manuscript and early printed sources. A normalized edition is therefore distinguished from a diplomatic edition (or semi-diplomatic edition), in which some attempt is made to preserve these features. The aim is to strike an appropriate balance between, on the one hand, rigorous fidelity to the source text (including, for example, the preservation of enigmatic and ambiguous elements); and, on the other, producing a new text that will be comprehensible and accessible to the modern reader. The extent of normalization is therefore at the discretion of the editor, and will vary. Some editors, for example, choose to modernize archaic spellings and punctuation, but others do not.

An edition of a text might be normalized based on internal criteria, where orthography is standardized according to the language of the original, or external criteria, where the norms of a different time period are applied. For an example of the latter, a published edition of a medieval Icelandic manuscript might be normalized to the conventions of modern Icelandic, or it might be normalized to Classical Old Icelandic. Standards of normalization vary based on language of the edition as well as the specific conventions of the publisher.

See also

• Canonicalization – Process for converting data into a "standard", "normal", or canonical form
• Text simplification – Automated process
• Unicode equivalence – Aspect of the Unicode standard