supernormalize

supernormalize is a JavaScript library that agressively normalizes text to a standard form. Use cases include:

• Mitigating homoglyph attacks
• Normalizing text for comparison
• Preparation for indexing text in a search engine
• Preparation for blacklisting text

Steps

The library performs the following steps:

1. Remove all marks (i.e. diacritics) and perform compatibility normalization
2. Convert the text to lowercase
3. Normalize homoglyphs using a mapping based on this list from the Unicode Consortium in version 15.1.0 (the used list does not include homoglyphs that are already normalized in steps 1 and 2)
4. Replace all whitespace characters with a single space and trim the text

Installation

npm install supernormalize

Usage

import { supernormalize } from "supernormalize";

const text = "⋿╳⍺rñ⍴lé";
const normalizedText = supernormalize(text);
console.log(normalizedText); // 'examp1e'

Examples

Input	Output	Note
⋿╳⍺rñ⍴lé	examp1e	Below rules can be combined
𝕋𝕙𝕚𝕤 𝕚𝕤 𝕒 𝕥𝕖𝕤𝕥!	th1s 1s a test!	Homoglyphs are normalized to a common form
D̴̝̼̅i̴̱̐͊́a̵̢͎͒͝ĉ̵͓̈́̽r̶͂͝ͅi̷͔͜͝ṭ̴͋͆͘i̵͔̅c̷̛͉̪͂͊s̵̞̝̲͊	d1acr1t1cs	Diacritics are removed
AАΑ	aaa	Latin, Cyrillic, and Greek characters are normalized to the same form
rn	m	Multiletter homoglyphs are normalized
ffi…	ff1...	Ligatures are normalized to letters
\tHELLO WORLD \n	he110 w0r1d	Whitespace and casing is normalized

Functions

supernormalize(text: string): string

Normalizes the given text performing the steps described above.

supernormalize.normalizeCase(text: string): string

Converts the given text to lowercase.

supernormalize.normalizeMarks(text: string): string

Removes all marks (i.e. diacritics) and performs compatibility normalization on the given text.

supernormalize.normalizeHomoglyphs(text: string): string

Normalizes homoglyphs using a mapping based on this list from the Unicode Consortium.

supernormalize.normalizeWhitespace(text: string): string

Replaces all whitespace characters with a single space. Trim the text.

License

This project is licensed under the MIT License - see the LICENSE file for details.