possible bug in lua.c main report lua_tostring malloc

[David Manura <dm.lua@...>]

In Lua 5.1.4, in lua.h:main, the call to "report" invokes
lua_tostring, which according to the Lua Reference Manual may raise a
memory error, presumably in the condition where the error value is a
number that must be translated into a string and therefore allocate
memory for the string.  In this context, the call to lua_tostring is
unprotected.

I was able to cause "lua -e 'error(5)'" to crash by triggering an
allocation failure in lmem.c:

--- src/lmem.c~
+++ src/lmem.c
@@ -74,9 +74,11 @@
 ** generic allocation routine.
 */
 void *luaM_realloc_ (lua_State *L, void *block, size_t osize, size_t nsize) {
+  static int count = 0;
   global_State *g = G(L);
   lua_assert((osize == 0) == (block == NULL));
   block = (*g->frealloc)(g->ud, block, osize, nsize);
+  if (nsize > osize && ++count > 450) luaD_throw(L, LUA_ERRMEM);
   if (block == NULL && nsize > 0)
     luaD_throw(L, LUA_ERRMEM);
   lua_assert((nsize == 0) == (block == NULL));

BTW, the following type of code is probably fairly common for users to
include at the top level C stack:

  if (lua_cpcall(L, f, NULL) != 0) {
    fprintf(stderr, "%s\n", lua_tostring(L, -1) ? lua_tostring(L, -1)
: "(not a string)");
  }

The hidden bug is that lua_tostring needs to be protected itself or
simply checked against lua_type(L, -1) == LUA_TSTRING, which
presumably cannot raise a memory error.  Ideally, we want to handle
the __tostring metamethod properly here too.  The correct solution to
that, accounting to errors as well, is not obvious.