If the documentation does not define the behavior, then the detection is also undefined as you may just test one case for which you would get one result or another case where you would get the opposite answer.
The detection will ever be fullproof. And it is not reliable to depend on this detection, except by not writing ANY code that depends on an undefined behavior.
This means that ANY code that uses table constructors mixing implicit (autoincremented) integer keys and other explicit integer keys IS UNRELIALE. and it will remain unreliable and will cause unexpected bugs on all versions of LUA that still don't specify the behavior and checks/enforces the rules.
It's very strange then that Lua accepts a simple syntax like {1, [1]=2} without being able to say what will be the value stored at index [1]. Your test may give an answer correct only for some sets of keys, where the order of assignments will be in one way, but for other sets of keys, you could as well get the inverse result, and you would NOT detect LuaJIT, and your code would make false decisions.
In summary, table constructors should ONLY be pure sequences with implicit keys, OR all members should be explicitly keyed.
I see it as a major defect of the language that should be solved by enforcing the rules to define the expected behavior. This should not break existing applications more than when they already use today the undefined behavior assuming that it is one way when it could still be the other way even on exactly the same platform instance.
Given that, this means that Lua is unable to say that it is portable, and every developer is forced to test their apps on many implementations, and some will be never be tested, notably newer LUA engines that could appear at any time or could be upgraded.
This means that for critical applications, YOU MUST NOT upgrade Lua, but this application Must provide its own copy of the engine. And reusability is also compromised notably for external Lua libraries used that would depend also on a behavior which is defined only for some versions of implementations.
And this can cause serious security bugs that could be exploited later.
Undefined behavior has already caused lot of damages in many languages and platforms. Inclicluding hardware platforms like CPUs. LUA devopers should make efforts to define the behavior and enforce them by conformance and coverage tests so that platforms can fix them and then be safely upgradable and serviceable.
Until then, do not upgrade your existing Lua engine, and do not use any third party library that was not explicitly tested fie exactly the same version of the implementation.
Anlther way to solve this problem would be to develop a minter that will detect the code using undefined behavior, so that this code can be rewritten.
The assumptions that Lua tables are capable of mixing sequences and associative keyed arrays is then wrong, it has never been specified correctly.