Re: Adding utf-8 handling support to Lua 5.1 in 2017

[Vadim Peretokin <vperetokin@...>]

I've had a closer look at utf8 in 5.3 and unfortunately it does not enable all of string.* to work with utf-8 which is what I need, so that is a no-go. I think the alternatives are starwing/luautf8, Stepets/utf8.lua and Mediawiki's ustring.

Has anyone had experience with any of those or other libraries I've missed to provide equivalent utf-8 support of the string.* library?

On Fri, Apr 14, 2017 at 12:23 AM Philipp Janda <siffiejoe@gmx.net> wrote:

Am 13.04.2017 um 19:01 schröbte Vadim Peretokin:
> Hey, thanks for that link, this kind of stuff is why I came to ask here. I
> read that your module does changes to the global environment (in the
> default load manner), are any of those changes backwards incompatible with
> code written for plain 5.1?

It's mostly extensions/enhancements and the old functions like
`getfenv`, `loadstring`, `unpack`, etc. are still there, but if you look
hard enough you'll find some incompatibilities (e.g. the table library
in 5.3 respects metamethods while the library in Lua 5.1 uses raw
accesses, or the return values of `os.execute`). You can
`require"compat53.module"` avoid this, or if you are only interested in
utf8, there's also `require"compat53.utf8"` which only loads the utf8
backport. The last two options don't modify the global environment.


Philipp


>
> On Thu, 13 Apr 2017 6:47 pm Philipp Janda, <siffiejoe@gmx.net> wrote:
>
>> Am 13.04.2017 um 16:28 schröbte Luiz Henrique de Figueiredo:
>>>> I'd like to add support for dealing with utf-8 text in Lua 5.1 - for
>>>> functions such as string.gsub, string.len and etc. I'm aware 5.3 comes
>> with
>>>> a utf8 library but I'm not considering breaking capability for the huge
>>>> ecosystem of code that's been developed for 5.1 for my application just
>> yet.
>>>>
>>>> I know there's starwing/luautf8, are there any other options that I
>> should
>>>> be considering?
>>>
>>> With the simple changes below, lutf8lib.c from 5.3 compiles fine in 5.1.
>>> I haven't tested it though.
>>
>> It won't work because of `lua_pushfstring(L, "%U", ?)`. You can #define
>> it to something reasonable, though. Compat-5.3[1] has lutf8lib.c
>> backported to Lua 5.1, btw.
>>
>>>
>>>     10d9
>>>     < #include "lprefix.h"
>>>     250,251c249,250
>>>     < LUAMOD_API int luaopen_utf8 (lua_State *L) {
>>>     <   luaL_newlib(L, funcs);
>>>     ---
>>>     > LUALIB_API int luaopen_utf8 (lua_State *L) {
>>>     >   luaL_register(L, "utf8", funcs);
>>>
>>
>> Philipp
>>
>>    [1]: https://github.com/keplerproject/lua-compat-5.3
>>
>>
>>
>>
>>
>>
>