Is it possible you have more than one heap in release and you are getting cross-heap problems?
When you run in debug, is absolutely all of your code compiled in debug with the same compilation settings? Now when you run in release is this still the case?
In some cases mixing debug/release versions causes the C runtime to effectively run 2 different heaps. It sounds to me like the one of two things.
First possibility, the cleanup of your first instance is bad. Possibly you are mangling the lua stack and then in release lua is corrupting the heap upon exit because its stack is wrong.
Another one is that you are overwriting memory while running and this has no effect in debug because things are allocated with more boundary bytes but in release you are thrashing part of the lua system.
In any case nothing in lua conflicts with malloc; the error is a corrupt heap due to a problem.
Chris