5. Charsets and encodings

5.1. Encodings

There are many encodings around the world. Before Unicode, each manufacturer invented its own encoding to fit its client market and its usage. Most encodings are incompatible on at least one code, with some exceptions. A document stored in ASCII can be read using ISO 8859-1 or UTF-8, because ISO-8859-1 and UTF-8 are supersets of ASCII. Each encoding can have multiple aliases, examples:

  • ASCII: US-ASCII, ISO 646, ANSI_X3.4-1968, …

  • ISO-8859-1: Latin-1, iso88591, …

  • UTF-8: utf8, UTF_8, …

Unicode is a charset and it requires a encoding. Only encodings of the UTF family are able to encode and decode all Unicode code points. Other encodings only support a subset of Unicode codespace. For example, ISO-8859-1 are the first 256 Unicode code points (U+0000—U+00FF).

This book presents the following encodings: ASCII, cp1252, GBK, ISO 8859-1, ISO 8859-15, JIS, UCS-2, UCS-4, UTF-8, UTF-16 and UTF-32.

5.2. Popularity

The three most common encodings are, in chronological order of their creation: ASCII (1968), ISO 8859-1 (1987) and UTF-8 (1996).

Google posted an interesting graph of the usage of different encodings on the web: Unicode nearing 50% of the web (Mark Davis, january 2010). Because Google crawls a huge part of the web, these numbers should be reliable. In 2001, the most used encodings were:

  • 1st (56%): ASCII

  • 2nd (23%): Western Europe encodings (ISO 8859-1, ISO 8859-15 and cp1252)

  • 3rd (8%): Chinese encodings (GB2312, …)

  • and then come Korean (EUC-KR), Cyrillic (cp1251, KOI8-R, …), East Europe (cp1250, ISO-8859-2), Arabic (cp1256, ISO-8859-6), etc.

  • (UTF-8 was not used on the web in 2001)

In december 2007, for the first time: UTF-8 becomes the most used encoding (near 25%). In january 2010, UTF-8 was close to 50%, and ASCII and Western Europe encodings were near 20%. The usage of other encodings doesn’t change.

5.3. Encodings performances

Complexity of getting the n th character in a string, and of getting the length in character of a string:

  • \(O(1)\) for 7 and 8 bit encodings (ASCII, ISO 8859 family, …), UCS-2 and UCS-4

  • \(O(n)\) for variable length encodings (e.g. the UTF family)

5.4. Examples

Encoding

A (U+0041)

é (U+00E9)

€ (U+20AC)

U+10FFFF

ASCII

0x41

ISO-8859-1

0x41

0xE9

UTF-8

0x41

0xC3 0xA9

0xE2 0x82 0xAC

0xF4 0x8F 0xBF 0xBF

UTF-16-LE

0x41 0x00

0xE9 0x00

0xAC 0x20

0xFF 0xDB 0xFF 0xDF

UTF-32-BE

0x00 0x00 0x00 0x41

0x00 0x00 0x00 0xE9

0x00 0x00 0x20 0xAC

0x00 0x10 0xFF 0xFF

— indicates that the character cannot be encoded.

5.5. Handle undecodable bytes and unencodable characters

5.5.1. Undecodable byte sequences

When a byte string is decoded, the decoder may fail to decode a specific byte sequence. For example, 0x61 0x62 0x63 0xE9 is not decodable from ASCII nor UTF-8, but it is decodable from ISO 8859-1.

Some encodings are able to decode any byte sequences. All encodings of the ISO-8859 family have this property, because all of the 256 code points of these 8 bits encodings are assigned.

5.5.2. Unencodable characters

When a character string is encoded to a character set smaller than the Unicode character set (UCS), a character may not be encodable. For example, € (U+20AC) is not encodable to ISO 8859-1, but it is encodable to ISO 8859-15 and UTF-8.

5.5.3. Error handlers

There are different choices to handle undecodable byte sequences and unencodable characters:

  • strict: raise an error

  • ignore

  • replace by ? (U+003F) or � (U+FFFD)

  • replace by a similar glyph

  • escape: format its code point

  • etc.

Example of the “abcdé” string encoded to ASCII, é (U+00E9) is not encodable to ASCII:

Error handler

Output

strict

raise an error

ignore

"abcd"

replace by ?

"abcd?"

replace by a similar glyph

"abcde"

escape as hexadecimal

"abcd\xe9"

escape as XML entities

"abcdé"

5.5.4. Replace unencodable characters by a similar glyph

By default, WideCharToMultiByte() replaces unencodable characters by similarly looking characters. The normalization to NFKC and NFKD does also such operation. Examples:

Character

Replaced by

U+0141, latin capital letter l with stroke

Ł

L

U+004C, latin capital letter l

U+00B5, micro sign

µ

μ

U+03BC, greek small letter mu

U+221E, infinity

8

U+0038, digit eight

U+0133, latin small ligature ij

ij

ij

{U+0069, U+006A}

U+20AC, euro sign

EUR

{U+0045, U+0055, U+0052}

∞ (U+221E) replaced by 8 (U+0038) is the worst example of the method: these two characters have completely different meanings.

5.5.5. Escape the character

Python “backslashreplace” error handler uses \xHH, \uHHHH or \UHHHHHHHH where HHH…H is the code point formatted in hexadecimal. PHP “long” error handler uses U+HH, U+HHHH or encoding+HHHH (e.g. JIS+7E7E).

PHP “entity” and Python “xmlcharrefreplace” error handlers escape the code point as an HTML/XML entity. For example, when U+00E9 is encoded to ASCII: it is replaced by é in PHP and é in Python.

5.6. Other charsets and encodings

There are much more charsets and encodings, but it is not useful to know them. The knowledge of a good conversion library, like iconv, is enough.