Re: xml/rapidxml UTF-8 validity assurance (was: [ANN] lub, lut, xml, yaml, dub and osc for Lua 5.3)

[Tim Hill <drtimhill@...>]

On May 11, 2015, at 11:54 AM, Coda Highland <chighland@gmail.com> wrote:

— UTF-8 is sometimes used to encode UTF-16 values (such as BOM), some of which are now accepted. Reject/accept?

ZWNBSP (that is, BOM) is a perfectly legit character in UTF-8. Accept
it. I get pissed at decoders that choke on it.

Well that’s true of the ZWNBSP *codepoint* U+FEFF, which of course encodes to 0xEF/0xBB/0xBF. But what about dumb encoders that encode a big-endian UTF-16 sequence into UTF-8 and emit a byte-swapped encoding for the BOM?

The problem is UTF-8 *should* be used to decode: UTF-8 -> codepoint array. Instead its (shudder) often used to decode UTF-8 -> UTF-16 -> (byte-swap based on BOM) -> codepoint array.

It’s one reason I detest Unicode.

—Tim