You're right that the JS-counterpart of 'nil' is 'undefined'. It's the result if a key is queried of a table ("object" in JS) is not existent.
null in JS or somewhat isn't the issue, a good counterpart in Lua is created simply by doing defining the global "null = { }". Maybe add a metatable for it, to hinter key adding/reading of it. The only difference by itself it defaults to false in JS, while {} defaults to true. Anyway, writing a lot of _javascript_, I can tell you "null" isn't the issue and one shouldn't use it in _javascript_ ever anyway (except some backward API demands it).
However, the difference is, _javascript_ does have a state of keys in objects that is "this is a known key of the object, but it is set to undefined". Normally one wouldn't notice, unless you either use the Object.keys( o ) function to get all keys, where it will show up, or use the "'key' in o" operator or use for( let key in Object ) etc. I suppose in some implementations it might also make a performance difference due to virtual classing.
I can tell you only of a recent experience of mine
sync{ table.optiom, key1 = 'foo', key2 = 'bar' }
now "optiom" is clearly a spelling mistake, it thus defaulted to nil and I had no way in sync() to check for it and it simply resulted in assuming it wasn't given at all. I think, I'll add a metatable to 'table' to error on non-existing keys as workaround, but anyway, that was a typical paper cut due to the language used.
And yes I'm aware a calling syntax like this wouldn't be available in JS anyway. There I would have to do it like
sync( table.option, 'key1', 'foo', 'key2', 'bar' ).. or sync( [ table.option ], { key1: 'foo', key2: 'bar' } )
so yes it's a complaint about an issue that couldn't arise in JS anyway due to less elegant calling syntax to start with.
In Lua upvalues can be the 'it is there but nil', as well function parameters and function return values, I always considered it inconsistent that it wasn't available for tables.
PS: I'm still happy with the Lua 5.x line having booleans. Especially since Lua 5.3, the question would otherwise be, would "not nil" result in an integer or floating 1? :-) And yes it also helps greatly for people having to work with JSON.