Bug: Segfault calling tostring on a result from debug.getlocal

[Dan Tull <dtull@...>]

If I run the below script in a stock (make macosx) Lua 5.1.5 interpreter, it 
crashes with a segmentation fault. Lua 5.2.0 appears to have the same bug, 
but it does not manifest as a crash.

I included some gory details below, but essentially one of the value results 
from debug.getlocal is a special type that lua_getmetatable (called by 
tostring) isn't anticipating, so it indexes past the end of G(L)->mt.

Note that the "name" of this result is the special name "(*temporary)" and 
those temporary values are only still present because I'm coming in from a
hook callback, so this is a pretty rare edge case.

Dan Tull
Developer ( Lua Tools)
Adobe Systems

--[[
Gory analysis details:

With assertions enabled in 5.1.5 I get:
	lapi.c:607: failed assertion `!(((i_o)->tt) >= 4) || 
		((((i_o)->tt) == (i_o)->value.gc->gch.tt) && 
		!(((i_o)->value.gc)->gch.marked & 
		((L->l_G)->currentwhite ^ ((1<<(0)) | (1<<(1)))) 
		& ((1<<(0)) | (1<<(1)))))'

This clause is what produces a false result:
	((((i_o)->tt) == (i_o)->value.gc->gch.tt)

ttype(obj) is 9 (LUA_TPROTO), and the switch default does this:
	mt = G(L)->mt[ttype(obj)];

G(L)->mt is statically sized at 9 elements (types 0-8) and as luck would have
it, mt[9] happens to contain a pointer to the TString for "__index", so when
sethvalue sets the i_o->tt to LUA_TTABLE, (i_o)->value.gc->gch.tt is still 4.

In Lua 5.2.0, G(L)->mt[9] happens to contain NULL, so the tostring function
does not crash and returns "proto: 0x0".

The temporary local value of type proto is left behind by the loadstring call.
Usually values of that type are completely internal, but debug.getlocal at
the right moment allowed it to leak out.
]]

local function caller()
	local result, message = loadstring( "return" )
	return result, message
end

local function locals()
	local i = 1
	local inf, n, v
	
	repeat
		inf = debug.getinfo( i, "fn" )
		if inf and inf.func == caller then
			break
		end
		i = i + 1
	until not inf

	if inf then
		local j = 1
		repeat 
			n, v  = debug.getlocal( i, j )
			if n then
				if type( v ) == "proto" and tostring( v ) then
					print "BOOM" -- never printed on 5.1.5
				end
			end
			j = j + 1
		until not n
	end
end

debug.sethook( locals, "", 1 )
caller()