I have tried to understand your problem, but something does not click.
What I have understood is that when the user code wants to push a shared_ptr holding a nullptr, your code pushes sol::nil, which is not Lua's nil. Correct so far?
Then the user pops something assuming it is a shared_ptr pushing earlier, and your code also assumes it is shared_ptr, just casting it as such. But in this case it casts sol::nil as shared_ptr, and that is bad. Is that correct?
The disconnect I have here is why, since you know that in certain cases you push shared_ptr as sol::nil, the popping code has no check for this case. That sounds like a very silly bug, easily fixable, so I must be missing something.