the overhead of storing an extra integer (size) in each table (which imho is usually zero overhead since most allocators are sufficiently course-grained this is lost in the allocation).
Most frequent usage of tables with user-defined length would be for creating a wrapper for tuples, and tuples are usually short.
So, overhead could be reduced to 1 byte in most use cases.
If byte = 0, then "user-specified length" is nil.
If byte = 1..254, then "user-specified length" is an integer in the range 0..253
If byte = 255, then "user-specified length" is arbitrary Lua value, and this value is stored in hash part of table under some unique key (that key must be non-constructible by user, and next() must skip that key)
In general, "user-specified length" could be arbitrary user object supporting arithmetics with numbers (__add, __sub, __lt)
