Re: [ Userdata ] __index aware of original lookup value

[JeanHeyd Meneide <jm3689@...>]

Dear Mr. Conner,

     Apologies for it taking a while for me to get back to you. Here is what I am trying to accomplish. Given a C++ type, such as:

        struct Person
        {
          std::string name;
          int age;
          bool hired;

          Person(std::string name, int age) : name(name), age(age), hired(false) {}

          void hire() { hired = true; }
          void fire() { hired = false; }
          bool isHired() const { return hired; }
          int getAge() const { return age; }
          std::string getName() const { return name; }
        };

(Note the change from what you wrote in that all fields are public). I am writing code that creates a class-like that is accessible in lua as so:

     local person = Person.new("Jay", 24)

     person:hire() -- call member functions through this syntax

     assert(person:isHired())
     assert(person.hired) -- also be able to get/set member variables through the obj.variable_name syntax

     person.age = 25 -- you can set variables this way too

I have gotten this code to work by using the "C Function for __index" appoach:

     - Make a userdata and bind a metatable to it

     - Make the Metatable with the necessary C functions bound to it, specifically
     - Overriding the `__index` metamethod with a C function

     - If the key matches a member function, ( e.g. obj:my_func() )
          - push the C function that is tied to the key and any closure information in upvalues

          - return this from the __index metamethod C function, let lua call that function [1]

     - If the key matches a member variable name, (e.g. obj.foo_var )

          - call the C function associated with getting/setting the variable directly from the C __index function

          - let that return the right value for the __index lookup or set the right value on __newindex

          - (NOTE: this sets/gets the actual instance's variables in memory)

This works as you and I would expect, because the "C Function for __index" gets called with the arguments (userdata, "my_func") or (userdata, "foo_var"). But [1]: the performance of this for member functions is not as fast as the method I am going to detail below, which I will call the "Table __index" approach:

     - Make a userdata and bind a Metatable to it

     - Make a plain table with the necessary C functions bound to it

     - Set the plain table to the Metatable's __index method

     - Call obj:my_func()

     - Lua VM finds obj:my_func() on the metatable's __index table

          - lua calls it without accessing a C function for lookup [2]

     - Access obj.foo_var

          - ???? - uh oh

Note that this second method, specifically at [2], results in MUCH less overhead (and is thusly much faster: refer to the benchmarks I sent earlier). However, this presents a distinct problem: it is impossible to bind the C++ userdata's member variables to the "print( obj.foo_var )" syntax I desire. The issue I was describing at the very beginning of this thread was my attempt to make the "Table __index" approach work for these member variables. My initial implementation to make that happen was a modification of the above steps, as follows (this approach does NOT work for member variables):

     - Make a userdata and bind a metatable to it

     - Make a plain table with the necessary C functions bound to it
     - Set the plain table to the Metatable's __index method

     [ Additions start here ]

     - Create another metatable table. Set it as the 1st plain table's metatable

     - Set this inner metatable's __index function to a C function that can look up member variables

          - Now, we have a chain of __index calls

     - Call obj:my_func()
     - Lua VM finds obj:my_func() on the metatable's __index table
          - lua calls it without accessing a C function for lookup

     - Access obj.foo_var

          - Lookup fails on on first __index table, checks it for an __index value, finds the C function

          - Calls the C function, but calls it with the first plain table and the key "foo_var"

          - [ Problem I was asking about in original post ] Cannot find a way to get to original "userdata", fail

As you can see, because we chain __index tables/functions, the arguments we get are not (userdata, "foo_var"), but (some table, "foo_var"). The "C Function for __index" works because the lookup fails with the userdata value (so I can always reference the userdata). The modified "Table __index" approach fails the lookup with (table_nobody_cares_about_that's_just_for_indexing_and_is_not_specific_to_userdata, "foo_var").

 I am wondering if there is a way around this?

At present, what I do when I generate the class binding is check if the binding uses member variables or has its own __index function specified by the user. If not, we use the fast "Table __index" method. If so, we use the "C Function for __index" method. I would like to know, however, if there's a better way to achieve what I'm doing to keep the performance up.

Sorry for the lengthy post, but does this make sense now?

On Wed, Feb 17, 2016 at 2:05 PM, Sean Conner <sean@conman.org> wrote:

It was thus said that the Great JeanHeyd Meneide once stated:
>     There is a significant speedup if you replace __index with a table
> instead of a function. If you look at this link here (
> http://satoren.github.io/lua_binding_benchmark/ ), there are some basic
> benchmarks of various frameworks and their call times for their
> implementations of C++ Member Functions ( I am the "sol2" cluster / row in
> the data ). You'll notice that we run fairly slow in terms of C++ member
> function call speed when compared to the fastest framework there. This
> puzzled me, as the process you describe for the function seemed like the
> only way to do it, so I looked into how other frameworks did it. After
> studying, I made a few changes and re-ran the benchmarks with the 2
> different styles of __index for accessing member functions when you do
> `myfoo:blah()`: http://i.imgur.com/kCi2HoL.png ( the associated commit for
> the 'sol2 - table __index' row you see there is here[1] ).

  I have to wonder what you are doing that makes the C++ calls more
expensive than C calls.  From what I can see, if you want speed, avoid C++.
I know that probably isn't an option but when call a C function is four
times faster than calling a C++ member function, something is wrong (unless
my understanding of C++ is in error and indeed, it is not as fast as C).

>      The process you described above is the old method we used (the bottom
> half of that CSV of data) and it is markedly slower. I do not know why it
> is markedly slower. But it is, and it's observable, and that's where my
> original question came from.

  If you don't know why it's slower, then perhaps an investigation is in
order.

> > "Could you please specify a sample of what you mean by this?  I just want
> > to make sure I understand what you are asking."
>
>     A theoretical way for me to solve my problem would be to have the
> __index metamethod be a function (and incur the function call overhead),
> but to also somehow set fields on the userdata. E.g.,

  No no no, not how you would like to solve it, what are you trying to
accomplish?  Is it to take a C class:

        class Person
        {
          std::string name;
          int age;
          bool hired;
        public:
          Person(std::string name, int age) : name(name), age(age), hired(false) {}
          void hire() { hired = true; }
          void fire() { hired = false; }
          bool isHired() const { return hired; }
          int getAge() const { return age; }
          std::string getName() const { return name; }
        };

and have Lua code automatically generated so one can call the various public
methods from Lua?  Not to automate this but make it easy to create functions
to do so?  Given the above, what would the Lua code look like?  The C++
code?

  -spc