Lua Locales

wiki

Here are a few notes on [locales] in Lua.

Lua is heavily influenced by Standard C, so much of the C treatment of locales carries over info Lua.

The current locale can be get or set using the [os.setlocale] function.

Warning: os.setlocale internally calls the C [setlocale] function, which globally sets the locale for all Lua states and all OS threads in the current process. However, some C implementations provide non-standard functions (e.g. Microsoft's [_configthreadlocale]) so that only the current OS thread is affected.

A brief background on locales in [PiL 22.2] (warning: see Historical notes below).

Lua syntax

In Lua 5.1, identifiers in the Lua language were locale dependent [1]. In 5.2, "Lua identifiers cannot use locale dependent characters [2].

Numbers in Lua code must be written in the C locale style:

print(1.23) -- always valid
print(1,23) -- always interpreted as two numbers: "1" and "23".

Tests on Numbers

> assert(os.setlocale('fr_FR'))
> = 1,23 , 1.23
1	23	1,23
> =loadstring("return 1,23 , 1.23")()
1	23	1,23
> print(1.23, tostring(1.23), string.format("%0.2f",1.23))
1,23	1,23	1,23
> 
> ="1.23" + 0
stdin:1: attempt to perform arithmetic on a string value
stack traceback:
	stdin:1: in main chunk
	[C]: in ?
> ="1,23" + 0
1,23
> =tonumber("1.23"), tonumber("1,23")
nil	1,23

The above will depend on luaconf.h settings, which in 5.2.0rc2 mentions this:

/*
@@ lua_str2number converts a decimal numeric string to a number.
@@ lua_strx2number converts an hexadecimal numeric string to a number.
** In C99, 'strtod' do both conversions. C89, however, has no function
** to convert floating hexadecimal strings to numbers. For these
** systems, you can leave 'lua_strx2number' undefined and Lua will
** provide its own implementation.
*/
#define lua_str2number(s,p)	strtod((s), (p))

#if defined(LUA_USE_STRTODHEX)
#define lua_strx2number(s,p)	strtod((s), (p))
#endif

If you disable LUA_USE_STRTODHEX, then Lua's own lua_strx2number implementation is used. This relies on locale-independent functions (e.g. lisspace from lctype.c). Observe the curious behavior:

Lua 5.2.0  Copyright (C) 1994-2011 Lua.org, PUC-Rio
> assert(os.setlocale'fr_FR')
> return tonumber'1.5', tonumber'1,5'
nil	1,5
> return tonumber'0x1.5', tonumber'0x1,5'
1,3125	nil

Tests on Character Classes

function findrange(pat)
  for i=0,255 do
    if string.char(i):match(pat) then
      print(i, string.char(i))
    end
  end
end

assert(os.setlocale'C')
findrange'%a' --> A-Z,a-z
assert(os.setlocale'en_US.ISO-8859-1')
findrange'%a' --> A-Z,a-z,\170,\181,\186,\192-\255
findrange'%l' --> a-z,\181,\223-\255

Other classes like isspace (%w) and isdigit (%d) potentially might return more than the C locale [3].

string.lower and string.upper are also locale dependent.

String Comparisons

> assert(os.setlocale'C')
> return "é" < "e"
false
> assert(os.setlocale('fr_FR'))
> return "é" < "e"
true

Tests on Dates

> assert(os.setlocale('C'))
> =os.date()
Sat Nov 26 13:22:56 2011
> assert(os.setlocale('fr_FR'))
> =os.date()
sam. 26 nov. 2011 13:22:56 EST

Historical Notes

-- This error occurred in 5.0 (but not 5.1 or above)
-- http://www.lua.org/pil/22.2.html
print(os.setlocale('pt_BR'))    --> pt_BR (Portuguese-Brazil)
print(3,4)                      --> 3    4
print(3.4)       --> stdin:1: malformed number near `3.4'