per thread callback closures: was Lua workshop: call for backup presentations.

[Michael Surette <msurette@...>]

On Monday 28 August 2006 21:08, William Ahern wrote:
> On Mon, 2006-08-28 at 16:51 -0400, Michael Surette wrote:
> > On Tuesday 22 August 2006 15:48, Wim Couwenberg wrote:
> > > And one more thing: anyone with some suggestions for round table
> > > subjects?
> > >
> > > --
> > > Wim
> >
> > I've been following the thread "Mulitiple functions" with interest
> > because of my similar problem.  I have lua functions defining C callbacks
> > which need to be reentrant.  My C application calls each lua callback in
> > a separate lua thread which needs access to globals as defined in the
> > original script file. It works well, with the unfortunate side effect is
> > that all callback function variables must be declared local to avoid
> > cross-contamination between threads.
> >
> > I believe that a low-overhead native-to-lua way of "cloning" functions
> > between threads, giving each an environment of your choice would be an
> > ideal solution to this problem and a good topic for the round table.
>
> Not sure if you have additional restrictions, but I am doing exactly
> this in an event-based SMTP server. It's all non-blocking I/O for
> end-to-end, no threads except for heavily CPU bound tasks, so I really
> didn't want Lua spending too much time doing useless work. Granted, I
> was one of the people who brought up this topic last week because it was
> less than straightforward and still requires some constraints in
> practice.
>
> When the daemon starts up it creates a global lua_State and loads the
> Lua "firmware". Every SMTP connection creates a thread and installs an
> environment which links back the global environment via an __index
> metamethod. Also, a handful of context specific objects are instantiated
> at some point (E for event, C for client, M for MIME). Every SMTP event
> can be sent to a Lua function for processing. E.g.
>
> function onConnect(ptrname, inetaddr)
> 	client_inetaddr = inetaddr
>
> 	-- rbl.lookup will yield Lua during the DNS query
> 	-- and renter the daemon I/O event loop
> 	if (rbl.lookup('spamcop.net', inetaddr) == '127.0.0.1') then
> 		E:apply({
> 			['action'] = 'reject',
> 			['text']   = '450 rejected by spamcop',
> 		})
> 	end
> end
>
> function onEnvFrom(sender, ...)
> 	if client_inetaddr ~= C:inetaddr() then
> 		M:addheader('X-Lua: i am a liar and will be fired')
> 	else
> 		log.warn('we should actually be here')
> 		log.warn('even if rbl.lookup yielded and')
> 		log.warn('other onConnect events executed')
> 	end
> end

Your code looks very familiar ;-)

With more the tools that I use adding lua support all the time (now nmap is 
about to release it), I thought that I should learn it.  As a project I  
decided to do a sendmail milter interface for lua scripts.  Very similar to 
your code, although sendmail does rbl lookup natively, the idea seems very 
much the same.  I have a per-context table which I pass around as the first 
argument to each callback for inter-callback communication.  I'm attaching a 
small milter script as an example so you can see the similarity.  As I said, 
it works well, except for the callback variables having to be declared local.  
Having my per-context table as an environment for the threads rather than an 
argument to the functions would be ideal.

One of the design objectives was to have as much of the 'administrative 
details' as possible in the C program and have the lua script just do the 
setup and filtering logic.  The need for local declarations goes against 
this, but is much less so than say, requiring factory functions.

I've tried many variations of the change the envronment dance, none which 
would act as I wanted.  From what I've gathered, what you suggested in the 
previous thread is what I need,  ie breaking the lexical scoping of the 
callback functions.  Two weeks ago I had no idea what lexical scoping or a 
closure was, so I could easily be wrong.  What was your solution?

Mike

-- a lua milter which logs connection details and all headers to syslog

myname="testmilter"                      -- milter name as per sendmail

sock="/var/run/milter/testmilter"        -- milter socket as per sendmail

runas="milter"                           -- who are we after we drop privileges

milterflags=M_NONE                       -- what are we allowed to change

-- callback functions
function connect(msg,hostname,hostaddr)
    msg.log= {}
    msg.log[#msg.log+1]= string.format("[%i] connect from %s at %s",msg.msgid,hostname,hostaddr)
    return R_CONTINUE
end

function helo(msg,helohost)
    msg.log[#msg.log+1]= string.format("[%i] helo from %s ",msg.msgid,helohost)
    return R_CONTINUE
end

function envfrom(msg,whofrom)
    msg.verify= getsymval(msg.ctx,"{verify}")
    msg.log[#msg.log+1]= string.format("[%i] message verify: %s",msg.msgid,msg.verify)
    msg.auth= getsymval(msg.ctx,"{auth_authen}")
    msg.log[#msg.log+1]= string.format("[%i] message auth: %s",msg.msgid,msg.auth)
    msg.log[#msg.log+1]= string.format("[%i] message from %s",msg.msgid,whofrom)
    msg.whofrom= whofrom
    return R_CONTINUE
end

function envrcpt(msg,whoto)
    msg.log[#msg.log+1]= string.format("[%i] message to %s",msg.msgid,whoto)
    msg.whoto= whoto
    return R_CONTINUE
end

function header(msg,headerf,headerv)
    msg.log[#msg.log+1]= string.format("[%i] %s: %s",msg.msgid,headerf,headerv)
    return R_CONTINUE
end

function close(msg)
    syslog(LOG_INFO,msg.log)
    syslog(LOG_INFO,string.format("[%i] ========================================",msg.msgid))
    return R_CONTINUE
end