In C, 0 is an _expression_ of type int; writing ~0 creates an unary _expression_ taking an int: the ~ operator only uses the standard type promotion (to int) by default and no other longer type (if you pass a short, that short will be promoted to an int), then it computes the bit-inversion (i.e. its a simple arithmetic addition: substract the value from the maximum of its input type and adds the minimum of that type); as this input type is an int, , it returns an int equal to (INTMAX - 0 + INTMIN). This gives the value you use in the initializer of your declared variable: It's only at that time that there will be a conversion from int to lua_Unsigned ! This conversion being possibly lossy if the target type cannot contain the full range for you initializer value (and because you did not provide an, explicit typecast to the result, the compiler should warn you.
As well converting a signed int to an unsigned number will use sign propagation before truncating the extra high bit: this is not lossy if all discarded high bits are the equal to the highest bit kept in the target type (this preserves the value), otherwise you have an overflow and the compiler will also warn you (this happens when the implicit conversion of the initializer value takes a value whose range is reduced in range or precision.
Using a static typecast however will silently remove that warning.
So
Lua_Unsigned umax = ~ 0;
is almost equivalent to:
Lua_Unsigned umax = (Lua_Unsigned)(~ 0);
except that you will not be warned by the compiler in case of loss of range or precision (static typecasts force the compiler to remain silent and discard all bits in excess, even if this changes the numeric value).
So yes you need to place a typecast like you said:
lua_Unsigned umax = ~(lua_Unsigned)0;
This static typecast is always safe because 0 and 1 are the only two numeric constants that can be typecasted without any loss of range or precision. Now the ~operator will correctly work on a long unsigned int (Lus_Unsigned) as expected and will return a value of that type, which can can safely used to initialize a variable with exactly the same type.