UTF-8

While natural languages encoded in traditional encodings are far from random byte sequences they are also unlikely to produce byte sequences that would pass a UTF-8 validity test and then be misinterpreted (obviously pure ASCII text would pass a UTF-8 validity test but provided the legacy encodings under consideration are also ASCII based this is not a problem). For example, for ISO-8859-1 text to be misrecognized as UTF-8, the only non-ASCII characters in it would have to be in sequences starting with either an accented letter or the multiplication symbol and ending with a symbol.

The bit patterns can be used to identify UTF-8 characters. If the byte's first hex code begins with 0–7, it is an ASCII character. If it begins with C or D, it is an 11 bit character (expressed in two bytes). If it begins with E, it is 16 bit (expressed in 3 bytes), and if it begins with F, it is 21 bits (expressed in 4 bytes). 8 through B cannot be first hex codes, but all following bytes must begin with a hex code between 8 through B. Thus, at a glance, it can be seen that "0xA9" is not a valid UTF-8 character, but that "0x54" or "0xE3 0xB4 0xB1" is a valid UTF-8 character.

The exact response required of a UTF-8 decoder on invalid input is not uniformly defined by the standards. In general, there are several ways a UTF-8 decoder might behave in the event of an invalid byte sequence:

1. Insert a replacement character (e.g. '?', '�').
2. Ignore the bytes.
3. Interpret the bytes according to a different character encoding (often the ISO-8859-1 character map).
4. Not notice and decode as if the bytes were some similar bit of UTF-8.
5. Stop decoding and report an error (possibly giving the caller the option to continue).

It is possible for a decoder to behave in different ways for different types of invalid input.

RFC 3629 only requires that UTF-8 decoders must not decode "overlong sequences" (where a character is encoded in more bytes than needed but still adheres to the forms above). The Unicode Standard requires a Unicode-compliant decoder to "…treat any ill-formed code unit sequence as an error condition. This guarantees that it will neither interpret nor emit an ill-formed code unit sequence."

Overlong forms are one of the most troublesome types of UTF-8 data. The current RFC says they must not be decoded but older specifications for UTF-8 only gave a warning and many simpler decoders will happily decode them. Overlong forms have been used to bypass security validations in high profile products including Microsoft's IIS web server. Therefore, great care must be taken to avoid security issues if validation is performed before conversion from UTF-8, and it is generally much simpler to handle overlong forms before any input validation is done.

To maintain security in the case of invalid input, there are two options. The first is to decode the UTF-8 before doing any input validation checks. The second is to use a decoder that, in the event of invalid input, returns either an error or text that the application considers to be harmless. Another possibility is to avoid conversion out of UTF-8 altogether but this relies on any other software that the data is passed to safely handling the invalid data.

Another consideration is error recovery. To guarantee correct recovery after corrupt or lost bytes, decoders must be able to recognise the difference between lead and trail bytes, rather than just assuming that bytes will be of the type allowed in their position.

• 与单字节(过时的编码)比较
  • 优点
    • UTF-8 can encode any Unicode character, avoiding the need to figure out and set a "code page" or otherwise indicate what character set is in use, and allowing output in multiple languages at the same time.
  • 缺点
    • UTF-8 is larger than the appropriate legacy encoding for everything except diacritic-free, Latin-alphabet text.
    • Legacy encodings using a single byte per character make string cutting and joining easy even with simple-minded APIs.

• 与多字节(过时的编码)比较
  • 优点
    • UTF-8 can encode any Unicode character. In most cases, legacy encodings can be converted to Unicode and back with no loss and—as UTF-8 is an encoding of Unicode—this applies to it too.
    • Character boundaries are easily found from anywhere in an octet stream (scanning either forwards or backwards). This implies that if a stream of bytes is scanned starting in the middle of a multibyte sequence, only the information represented by the partial sequence is lost and decoding can begin correctly on the next character. Similarly, if a number of bytes are corrupted or dropped then correct decoding can resume on the next character boundary. Many legacy multi-byte encodings are much harder to resynchronise.
    • A byte sequence for one character never occurs as part of a longer sequence for another character as it did in older variable-length encodings like Shift-JIS (see the previous section on this). For instance, US-ASCII octet values do not appear otherwise in a UTF-8 encoded character stream. This provides compatibility with file systems or other software (e.g., the printf() function in C libraries) that parse based on US-ASCII values but are transparent to other values.
    • The first byte of a multibyte sequence is enough to determine the length of the multibyte sequence. This makes it extremely simple to extract a substring from a given string without elaborate parsing. This was often not the case in legacy multibyte encodings.
    • Efficient to encode using simple bit operations. UTF-8 does not require slower mathematical operations such as multiplication or division (unlike the obsolete UTF-1 encoding).
  • 缺点
    • UTF-8 is generally larger than the appropriate legacy encoding for everything except diacritic-free, Latin-alphabet text. East Asian scripts generally had two bytes per character in their legacy encodings yet take three bytes per character in UTF-8.

• 与UTF-7比较
  • 优点
    • UTF-8 uses significantly fewer bytes per character for all non-ASCII characters.
    • UTF-8 encodes "+" as itself whereas UTF-7 encodes it as "+-".
  • 缺点
    • UTF-8 requires the transmission system to be eight-bit clean. In the case of e-mail this means it has to be further encoded using quoted printable or base64 in some cases. This extra stage of encoding carries a significant size penalty. However, this disadvantage is not so important an issue anymore because most mail transfer agents in modern use are eight-bit clean and support 8BITMIME SMTP extension as specified in RFC 1869.

• 与UTF-16比较
  • 优点
    • Byte values of 0 (The ASCII NUL character) do not appear in the encoding unless U+0000 (the Unicode NUL character) is represented. This means that legacy C library string functions (such as strcpy()) that use a null terminator will not incorrectly truncate strings.
    • Since ASCII characters can be represented in a single byte, text consisting of mostly diacritic-free Latin letters will be around half the size in UTF-8 than it would be in UTF-16. Text in many other alphabets will be slightly smaller in UTF-8 than it would be in UTF-16 because of the presence of spaces.
    • Most existing computer programs (including operating systems) were not written with Unicode in mind, and using UTF-16 with them would create major compatibility issues, as it is not a superset of ASCII. UTF-8 allows programs to treat ASCII as they always did, and changes behaviour only for non-ASCII characters that were different by location anyway.
    • In UTF-8, characters outside the basic multilingual plane are not a special case.
    • UTF-8 uses a byte as its atomic unit while UTF-16 uses a 16-bit word which is generally represented by a pair of bytes. This representation raises a couple of potential problems of its own.
      • When representing a word in UTF-16 as two bytes, the order of those two bytes becomes an issue. A variety of mechanisms can be used to deal with this issue (for example, the Byte Order Mark), but they still present an added complication for software and protocol design.
      • If an odd number of bytes are removed from the beginning of UTF-16-encoded text, the result will be either invalid UTF-16 or completely meaningless text. In UTF-8, if part of a multi-byte character is removed, only that character is affected and not the rest of the text.
    • UTF-16 is often mistaken to be constant-length, leading to code that works for most text but suddenly fails for non-BMP characters. Retrofitting code tends to be hard, so it's better to implement support for the entire range of Unicode from the start.
  • 缺点
    • Characters above U+0800 in the BMP use three bytes in UTF-8, but only two in UTF-16. As a result, text in [for example] Chinese, Japanese or Hindi takes up more space when represented in UTF-8. However, this advantage is partly offset by the fact that characters below U+0080 (Latin letters, numbers and punctuation marks, space, carriage return and line feed) that frequently appear in those text take only one byte in UTF-8 while they take two bytes in UTF-16.