Re: lua_newuserdata() and integer overflow

[Roberto Ierusalimschy <roberto@...>]

> Traditionally you allocate memory using malloc() and Lua environments you might use lua_newuserdata() to get garbage collection. Now, when you allocate memory for more than one element, usually the idiom malloc(nelem * size) or lua_newuserdata(nelem * size) is used.
> 
> The integer multiplication, however, can overflow and lead to buffer overflows.  Try e.g. malloc(65536 * 65536). In C libraries a function calloc(nelem, size) exists, but unfortunately it does not guarantee to not overflow either.  On some operating systems, e.g. FreeBSD, it detects overflow and returns NULL.
> 
> I am suggesting to add a function to the Lua C API that is like lua_newuserdata(), but takes two parameters, a size and a number of elements, and that checks for overflow and returns NULL in this case:
> 
> lua_newuserdatas(size_t count, size_t size)
> 
> Thoughts on this?

Overflows in C can occurr for any arithmetic operation. Usually we use
the program logic to avoid that, not explicit checks for each operation.

Moreover, the only portable way to check multiplication overflow is by
doing a division, which is not very efficient. More often than not,
at the call site, 'size' would be a constant, and so the division
there could be much more efficient. Another drawback is that often
a program would want to handle an overflow quite differently from a
memory-allocation error.

-- Roberto