4. Unicode¶
Unicode is a character set. It is a superset of all the other character sets. In the version 6.0, Unicode has 1,114,112 code points (the last code point is U+10FFFF). Unicode 1.0 was limited to 65,536 code points (the last code point was U+FFFF), the range U+0000—U+FFFF called BMP (Basic Multilingual Plane). I call the range U+10000—U+10FFFF as non-BMP characters.
4.1. Unicode Character Set¶
The Unicode Character Set (UCS) contains 1,114,112 code points: U+0000—U+10FFFF. Characters and code point ranges are grouped by categories. Only encodings of the UTF family are able to encode the UCS.
4.2. Categories¶
Unicode 6.0 has 7 character categories, and each category has subcategories:
Letter (L): lowercase (Ll), modifier (Lm), titlecase (Lt), uppercase (Lu), other (Lo)
Mark (M): spacing combining (Mc), enclosing (Me), non-spacing (Mn)
Number (N): decimal digit (Nd), letter (Nl), other (No)
Punctuation (P): connector (Pc), dash (Pd), initial quote (Pi), final quote (Pf), open (Ps), close (Pe), other (Po)
Symbol (S): currency (Sc), modifier (Sk), math (Sm), other (So)
Separator (Z): line (Zl), paragraph (Zp), space (Zs)
Other (C): control (Cc), format (Cf), not assigned (Cn), private use (Co), surrogate (Cs)
There are 3 ranges reserved for private use (Co subcategory): U+E000—U+F8FF (6,400 code points), U+F0000—U+FFFFD (65,534) and U+100000—U+10FFFD (65,534). Surrogates (Cs subcategory) use the range U+D800—U+DFFF (2,048 code points).
4.3. Statistics¶
On a total of 1,114,112 possible code points, only 248,966 code points are assigned: 77.6% are not assigned. Statistics excluding not assigned (Cn), private use (Co) and surrogate (Cs) subcategories:
Letter: 100,520 (91.8%)
Symbol: 5,508 (5.0%)
Mark: 1,498 (1.4%)
Number: 1,100 (1.0%)
Punctuation: 598 (0.5%)
Other: 205 (0.2%)
Separator: 20 (0.0%)
On a total of 106,028 letters and symbols, 101,482 are in “other” subcategories (Lo and So): only 4.3% have well defined subcategories:
Letter, lowercase (Ll): 1,759
Letter, uppercase (Lu): 1,436
Symbol, math (Sm): 948
Letter, modifier (Lm): 210
Symbol, modifier (Sk): 115
Letter, titlecase (Lt): 31
Symbol, currency (Sc): 47
4.4. Normalization¶
Unicode standard explains how to decompose a character. For example, the precomposed character ç (U+00C7, Latin capital letter C with cedilla) can be written as the sequence of two characters: {¸ (U+0327, Combining cedilla), c (U+0043, Latin capital letter C)}. This decomposition can be useful when searching for a substring in a text, e.g. removing the diacritic is pratical for the user. The decomposed form is called Normal Form D (NFD) and the precomposed form is called Normal Form C (NFC).
Form |
String |
Unicode |
---|---|---|
NFC |
ç |
U+00C7 |
NFD |
¸c |
{U+0327, U+0043} |
Unicode database also contains a compatibility layer: if a character cannot be rendered (no font contain the requested character) or encoded to a specific encoding, Unicode proposes a replacment character sequence which looks like the character, but may have a different meaning.
For example, ij (U+0133, Latin small ligature ij) is replaced by the two characters {i (U+0069, Latin small letter I), j (U+006A, Latin small letter J)}. ij character cannot be encoded to ISO 8859-1, whereas ij characters can.
Two extra normal forms use this compatibility layer: NFKD (decomposed) and NFKC (precomposed).
Note
The precomposed forms (NFC and NFKC) begin by a canonical decomposition before recomposing pre-combined characters again.