CESU-8
The Compatibility Encoding Scheme for UTF-16: 8-Bit (CESU-8) is a variant of UTF-8 that is described in Unicode Technical Report #26.[1] A Unicode code point from the Basic Multilingual Plane (BMP), i.e. a code point in the range U+0000 to U+FFFF, is encoded in the same way as in UTF-8. A Unicode supplementary character, i.e. a code point in the range U+10000 to U+10FFFF, is first represented as a surrogate pair, like in UTF-16, and then each surrogate code point is encoded in UTF-8. Therefore, CESU-8 needs six bytes (3 bytes per surrogate) for each Unicode supplementary character while UTF-8 needs only four. Though not specified in the technical report, unpaired surrogates are also encoded as 3 bytes each, and CESU-8 is exactly the same as applying an older UCS-2 to UTF-8 converter to UTF-16 data.
The encoding of Unicode non-BMP characters works out to 11101101 1010yyyy 10xxxxxx 11101101 1011xxxx 10xxxxxx (yyyy represents the top five bits of the character minus one). The byte values 0xF0—0xF4 will not appear in CESU-8, as they start the 4-byte encodings used by UTF-8.
CESU-8 is not an official part of the Unicode Standard, because Unicode Technical Reports are informative documents only.[2] It should be used exclusively for internal processing and never for external data exchange.
Supporting CESU-8 in HTML documents is prohibited by the W3C[3][4] and WHATWG[5] HTML standards, as it would present a cross-site scripting vulnerability.[6]
Java's Modified UTF-8 is CESU-8 with a special overlong encoding of the NUL character (U+0000) as the two-byte sequence C0 80.[7]
The Oracle database uses CESU-8 for its "UTF8" character set. Standard UTF-8 can be obtained using the character set "AL32UTF8" (since Oracle version 9.0).
Examples
| Code point | U+0045 | U+0205 | U+10400 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Character | E | ȅ | 𐐀 | ||||||||||||
| UTF-8 | 45 | C8 | 85 | F0 | 90 | 90 | 80 | ||||||||
| UTF-16 | 0045 | 0205 | D801 | DC00 | |||||||||||
| CESU-8 | 45 | C8 | 85 | ED | A0 | 81 | ED | B0 | 80 | ||||||
References
- ↑ McGowan, Rick. "Unicode Technical Report #26 - Compatibility Encoding Scheme for UTF-16: 8-Bit (CESU-8)". Unicode Consortium. https://www.unicode.org/reports/tr26/.
- ↑ "About Unicode Technical Reports - Types of Unicode Technical Reports: UAX, UTS, UTR". Unicode Consortium. https://www.unicode.org/reports/about-reports.html#Types.
- ↑ "8.2.2.3. Character encodings". W3C. https://www.w3.org/TR/html51/syntax.html#character-encodings.
- ↑ "8.2.2.3. Character encodings". W3C. https://www.w3.org/TR/html5/syntax.html#character-encodings.
- ↑ "12.2.3.3 Character encodings". WHATWG. https://html.spec.whatwg.org/multipage/parsing.html#character-encodings.
- ↑ "<meta> - HTML". Mozilla. https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta.
- ↑ "Java SE documentation for Interface java.io.DataInput, subsection on Modified UTF-8". Oracle Corporation. 2015. https://docs.oracle.com/javase/8/docs/api/java/io/DataInput.html#modified-utf-8.
External links
- Unicode Technical Report #26
- Modified UTF-8 definition
- Graphical View of CESU-8 in ICU's Converter Explorer
 |  |
