Yes, if you want a richer enum, you'd need to cut into the error generation pipeline earlier. I think the two functions to consider patching are luaG_runerror and luaL_error -- making versions of both that accept an optional enum argument would seem a fairly straightforward way of getting the behavior you're asking for.
I understand but my employer agreed to open-source the debugger with the promise it'd give them visibility and help recruiting Lua scripters. Patching the VM would make the debugger useless to anyone using pre-built Lua libs.
Reading between the lines, it sounds like this is a custom debugger commissioned to work with a particular C++ project, but, it might also be distributed as a standalone open source app?
In that case, you could consider modding the VM, and posting the diff on the power patches page. That way other projects that embed different versions of Lua could potentially patch in the hooks needed by your debugger. And if the debugger is awesome enough, there's an outside chance that the Lua author's might take notice, and roll the diffs into 5.3 :)
But if you're serious about supporting a wide range of Lua VM's, pattern matching is probably you're only option. If you go this route, I'd also grab a copy of the LuaJIT sources, as that's the other version of the VM you're likely to encounter in the wild.
-Sven
