Re: Managing Unicode (UTF-8 and UTF-16) data in Lua

> BTW, I don't know if it is possible to display arbitrary Unicode symbol in
> Windows console,
It is. But you have to use the Unicode APIs, not the C stdlib.

> so maybe the wish of making Unicode version of print() is unattainable
> anyway.
It's perfectly possible, but not with ANSI C only (which is what core
Lua requires).

The problem is not in choosing right API functions.

Try to write some Arabic or Japanese text in console on your European Windows using Unicode WinAPI.

Probably, you will see question marks instead of readable symbols on the screen.

Only symbols from current Windows locale will be displayed correctly.
There is no "universal locale" in Windows, so, Unicode version of print() is unattainable.

Welcome to Windows world! Some thing here are quite weird.

Why do that when the standard Lua string type is UTF-8 safe? Better
surely to use UTF-8 via Lua strings, and only use UTF-16 for
interfacing to the Windows APIs?

Conversion between UTF-16 and UTF-8 is expensive.

Basic string operations on UTF-8 are supported by Lua 5.3 utf8 library only partially, and they are expensive.
Operations on fixed width character strings (such as UTF-16) are processed faster.

UTF-8 on Windows is a handicap, not a helper.