5. Charsets and encodings

5.1. Encodings

There are many encodings around the world. Before Unicode, each manufacturer invented its own encoding to fit its client market and its usage. Most encodings are incompatible on at least one code, with some exceptions. A document stored in ASCII can be read using ISO 8859-1 or UTF-8, because ISO-8859-1 and UTF-8 are supersets of ASCII. Each encoding can have multiple aliases, examples:

  • ASCII: US-ASCII, ISO 646, ANSI_X3.4-1968, …
  • ISO-8859-1: Latin-1, iso88591, …
  • UTF-8: utf8, UTF_8, …

Unicode is a charset and it requires a encoding. Only encodings of the UTF family are able to encode and decode all Unicode code points. Other encodings only support a subset of Unicode codespace. For example, ISO-8859-1 are the first 256 Unicode code points (U+0000—U+00FF).

This book presents the following encodings: ASCII, cp1252, GBK, ISO 8859-1, ISO 8859-15, JIS, UCS-2, UCS-4, UTF-8, UTF-16 and UTF-32.

5.2. Popularity

The three most common encodings are, in chronological order of their creation: ASCII (1968), ISO 8859-1 (1987) and UTF-8 (1996).

Google posted an interesting graph of the usage of different encodings on the web: Unicode nearing 50% of the web (Mark Davis, january 2010). Because Google crawls a huge part of the web, these numbers should be reliable. In 2001, the most used encodings were:

  • 1st (56%): ASCII
  • 2nd (23%): Western Europe encodings (ISO 8859-1, ISO 8859-15 and cp1252)
  • 3rd (8%): Chinese encodings (GB2312, ...)
  • and then come Korean (EUC-KR), Cyrillic (cp1251, KOI8-R, ...), East Europe (cp1250, ISO-8859-2), Arabic (cp1256, ISO-8859-6), etc.
  • (UTF-8 was not used on the web in 2001)

In december 2007, for the first time: UTF-8 becomes the most used encoding (near 25%). In january 2010, UTF-8 was close to 50%, and ASCII and Western Europe encodings were near 20%. The usage of other encodings doesn’t change.

5.3. Encodings performances

Complexity of getting the n th character in a string, and of getting the length in character of a string:

  • O(1) for 7 and 8 bit encodings (ASCII, ISO 8859 family, ...), UCS-2 and UCS-4
  • O(n) for variable length encodings (e.g. the UTF family)

5.4. Examples

Encoding A (U+0041) é (U+00E9) € (U+20AC) U+10FFFF
ASCII 0x41
ISO-8859-1 0x41 0xE9
UTF-8 0x41 0xC3 0xA9 0xE2 0x82 0xAC 0xF4 0x8F 0xBF 0xBF
UTF-16-LE 0x41 0x00 0xE9 0x00 0xAC 0x20 0xFF 0xDB 0xFF 0xDF
UTF-32-BE 0x00 0x00 0x00 0x41 0x00 0x00 0x00 0xE9 0x00 0x00 0x20 0xAC 0x00 0x10 0xFF 0xFF

— indicates that the character cannot be encoded.

5.5. Handle undecodable bytes and unencodable characters

5.5.1. Undecodable byte sequences

When a byte string is decoded, the decoder may fail to decode a specific byte sequence. For example, 0x61 0x62 0x63 0xE9 is not decodable from ASCII nor UTF-8, but it is decodable from ISO 8859-1.

Some encodings are able to decode any byte sequences. All encodings of the ISO-8859 family have this property, because all of the 256 code points of these 8 bits encodings are assigned.

5.5.2. Unencodable characters

When a character string is encoded to a character set smaller than the Unicode character set (UCS), a character may not be encodable. For example, € (U+20AC) is not encodable to ISO 8859-1, but it is encodable to ISO 8859-15 and UTF-8.

5.5.3. Error handlers

There are different choices to handle undecodable byte sequences and unencodable characters:

  • strict: raise an error
  • ignore
  • replace by ? (U+003F) or � (U+FFFD)
  • replace by a similar glyph
  • escape: format its code point
  • etc.

Example of the “abcdé” string encoded to ASCII, é (U+00E9) is not encodable to ASCII:

Error handler Output
strict raise an error
ignore "abcd"
replace by ? "abcd?"
replace by a similar glyph "abcde"
escape as hexadecimal "abcd\xe9"
escape as XML entities "abcdé"

5.5.4. Replace unencodable characters by a similar glyph

By default, WideCharToMultiByte() replaces unencodable characters by similarly looking characters. The normalization to NFKC and NFKD does also such operation. Examples:

Character Replaced by
U+0141, latin capital letter l with stroke Ł L U+004C, latin capital letter l
U+00B5, micro sign µ μ U+03BC, greek small letter mu
U+221E, infinity 8 U+0038, digit eight
U+0133, latin small ligature ij ij ij {U+0069, U+006A}
U+20AC, euro sign EUR {U+0045, U+0055, U+0052}

∞ (U+221E) replaced by 8 (U+0038) is the worst example of the method: these two characters have completely different meanings.

5.5.5. Escape the character

Python “backslashreplace” error handler uses \xHH, \uHHHH or \UHHHHHHHH where HHH...H is the code point formatted in hexadecimal. PHP “long” error handler uses U+HH, U+HHHH or encoding+HHHH (e.g. JIS+7E7E).

PHP “entity” and Python “xmlcharrefreplace” error handlers escape the code point as an HTML/XML entity. For example, when U+00E9 is encoded to ASCII: it is replaced by é in PHP and é in Python.

5.6. Other charsets and encodings

There are much more charsets and encodings, but it is not useful to know them. The knowledge of a good conversion library, like iconv, is enough.