Was the argument 0xFFFFFFFF originally derived from a literal?
If so, I'll hazard a guess that the problem happens in the Lua compiler,
compile time, when the parser converts the string token 0xFFFFFFFF into a
number.
If your code contains the string "0xFFFFFFFF", try replacing it with
"4294967295" or "2^32-1" and see what happens. If it then works, you know
that the problem isn't in bitlib itself.
I was using a numeric literal. I tried the above constants and here
are the results:
require "bit"
print(bit.band(3, 0xFFFFFFFF))
-- displays 0
print(bit.band(3, 4294967295))
-- displays 0
print(bit.band(3, 2^32 -1))
-- displays 0