Re: Single-threading servers and select()

[Mike Pall <mikelu-0501@...>]

Hi,

duck wrote:
> In the real world, is this really an intractable problem? With very many
> connections, surely each select() will identify many connections with data
> available, thus amortising the cost of scanning the select array over many
> connections?

Your analysis is correct. At least as long as most connections are non-idle.
The replacement system calls (epoll(), kqueue()) have been designed with
environments in mind where you have many, many idle connections. And only
then it makes a difference.

Under real world use poll() is good enough, most of the time. And modern
kernels contain a few optimizations that make it less costly. Stay away
from select() of course, since it's limited to a fixed number of
descriptors (usually 1024).


On a related note, I can report that I recently had the dubious pleasure
to take a closer look at a system that was doing >10000 context switches
per second. Massive threading and forking going on. It was still very
responsive and it turned out that the context switches were the least of
its problems (it was disk related). The CPU was still mostly idle. ;-)

The work done tuning threading and context switches for Linux 2.6 plus
the work on NPTL in glibc has yielded impressive results.


Still, I like event driven systems (with coroutine sugar) a lot more.
Primarily because I've been into mutex hell with native threads and
into manual stack ripping hell with plain event driven systems.

There is an interesting paper at:

http://www.usenix.org/events/hotos03/tech/vonbehren.html
"Why Events Are A Bad Idea (for High-concurrency Servers)"

I think they are basically right and it reassured me that event driven
systems need to be complemented with good coroutine support. Lua is
perfect for this.


Jeff Koftinoff wrote:
> Read http://www.kegel.com/c10k.html
> 
> The problem is that select() requires the app to do a lot of work to 
> set up all the FD_SETs and scan them.

Yup. Unfortunately Dan hasn't updated his document for a while.
E.g. Linux 2.6 and NPTL are widespread nowadays.

There is a newer paper, that shows that epoll() has it's own share of
problems and that the advantage over poll() is less than one would
like to believe (unless you have tons of idle connections). Go to:

http://bbcr.uwaterloo.ca/~brecht/pubs.html

and read:

Louay Gammo, Tim Brecht, Amol Shukla, David Pariag, 
"Comparing and Evaluating epoll, select, and poll Event Mechanisms"
6th Annual Ottawa Linux Symposium, Ottawa, Canada, July, 2004.

Has anyone seen anywhere a followup from the kernel camp on this paper?

Bye,
     Mike