lua_newuserdata() and integer overflow

[Marc Balmer <marc@...>]

Traditionally you allocate memory using malloc() and Lua environments you might use lua_newuserdata() to get garbage collection. Now, when you allocate memory for more than one element, usually the idiom malloc(nelem * size) or lua_newuserdata(nelem * size) is used.

The integer multiplication, however, can overflow and lead to buffer overflows.  Try e.g. malloc(65536 * 65536). In C libraries a function calloc(nelem, size) exists, but unfortunately it does not guarantee to not overflow either.  On some operating systems, e.g. FreeBSD, it detects overflow and returns NULL.

I am suggesting to add a function to the Lua C API that is like lua_newuserdata(), but takes two parameters, a size and a number of elements, and that checks for overflow and returns NULL in this case:

lua_newuserdatas(size_t count, size_t size)

Thoughts on this?