That attack based on excessive usage of memory is not impossible to prevent, as long as Lua provides a correct resource manager checking allowed quotas, and then properly generates an error for any scripts attempting to bypass the quota.
Here it is just the lack of quota checking in the currently deployed implementation that is the only problem. It could happen to any project. And even Linux provides per-process quotas, so it should not hang the machine (at worst, that process will be killed with a fatal error: this may not be desirable, but it's up to the process to take preventive measures and implement them, if that process must continue working (for example if it runs concurrent threads for other sessions that should continue working in an unaltered way, you need to make Lua kill the offending thread only, garbage collect its resources, if the thread does not want to die; as the thread may be running a full Lua instance with its coroutines, it's then to the Lua instance to take the preventive measures on quotas, and gracefully send a standard Lua error to the current offending coroutine; then Lua scripts will have to adapt if they want to resist to these errors, so the call to string.rep() can return an error and reject the request to allocate more than what is permitted by the quotas set in the Lua instance, and not just assume everything to be handled by the standard C allocator, which may be more permissive and not checking the quotas for each thread, and certainly not for each Lua coroutine).
In summary, fix the functions in the string.* library to not use the C allocator blindly...
