& v_shared_ref = binding_magic["v"]; // ok
2d. std::shared_ptr<some_type> v_shared_value = binding_magic["v"]; // ok
2e. sol::optional<std::shared_ptr<some_type>> maybe_v_shared_value = binding_magic["v"]; // ok
Now, let's assume "v" is "nil". Of those 5 forms:
[ 2a ] We can handle the "nil" case, and return a nullptr safely
[ 2b ] We cannot convert "nil" to a reference: errors
[ 2c ] We cannot convert "nil" to a shared_ptr (lua_touserdata returns nullptr): errors
[ 2d ] Same error as 2c (even though it is a value, the conversion function returns a reference and the user makes a copy when it reaches the top-level and gets converted to a value by the user specifying a type)
[ 2e ] We can handle the "nil" case, and return a "this optional does not have stuff"
When "nil" is present, only the forms [ 2a ] and [ 2e ] are valid ways of working with it. I am a little disappointed about [ 2d ], because it's a value and we should _reasonably_ be able to create a nullptr-containing object (default-constructed, as you say), but I cannot differentiate between someone asking for a reference std::shared_ptr, versus a value std::shared_ptr (it's a C++ quirk about conversions that would probably not be useful to fully discuss on the list). That means I can't construct something on the stack and then return it, because I would ultimately be returning a dangling reference when this conversion is done.
To reiterate what I said before, 2d confused some people because when a function return a "std::shared_ptr<some_type>(nullptr)" and then they retrieved a "std::shared_ptr" as an argument or something later, they expected a shared_ptr value to be made that was a nullptr. My advice to them was not to take the shared_ptr, just take a raw pointer, but sometimes APIs are fixed and some people do not want to write wrapper functions just to communicate through my library.
Hencewhy, I had a Second Idea, where I would NOT "lua_pushnil" but instead construct a shared_ptr that was filled with nothing (nullptr) and store that as a userdata in Lua. This made all the conversions work, at the cost of "my_userdata == nil" no longer evaluating to true (hence the reason I started this thread).
Does that make sense?