Re: on the cost of non-blocking LuaSockets

[William Ahern <william@...>]

On Thu, Dec 11, 2014 at 12:31:28PM -0500, Sean Conner wrote:
> It was thus said that the Great Valerio Schiavoni once stated:
> > 
> > Due to continuous 'timeout' errors returned by the receiving socket (which
> > is set to nonblocking, socket:settimeout(0)), the coroutine continuosly
> > yields: in this case, is the reception of the data somehow put 'on hold' ?
> 
>   You don't need to set the socket to non-blocking if you use select just
> for reading (and I just checked our codebase at work---I don't set any of
> the sockets to nonblocking).  All select() does is indicate when a file
> descriptor (in your case, a network socket) has received data (ready for
> reading), can write data without blocking (ready for writing [1]) or an
> error happened.

This is not true at all, and people who assume this are running code with
TOCTTOU bugs. Spurious wakeups can be very common. There are two common
scenarios.

1) Two threads reading from the same socket. Both get woken up, one drains
the socket, the other ends up stalled. This is easy to avoid, but proves the
point about the semantics of readiness notification. It applies equally to
writability as to readability.

2) Some systems such as Linux do delayed checksum verification. When a
packet comes in the kernel immediately signals input readiness on the
socket, postpones verification (including ACK if TCP), and sets a timer.
Unless the timer expires (rarely unless the user thread is CPU bound doing
something else), the packet is verified only when the thread tries to read
from the socket. If the verification fails, the packet is discarded and, if
TCP, the kernel requests retransmission. If the socket is in blocking mode,
you've just stalled your thread. Read won't return because it's still
waiting for some data to return.

The kicker is that packet corruption is rare unless and until your network
becomes heavily loaded. So it's precisely when you are relying on the
maximum performance and lowest latency of your code when this bug will
trigger. OTOH, with bulk transmissions you often won't notice the stalling
because eventually there'll be more data coming along. But the degraded
performance can be significant nonetheless; you just assume the bottleneck
is in the hardware, not a bug in your software.

Another poor assumption people make is thinking they don't need to set UDP
sockets to non-blocking when writing, or even bother with polling for write
readiness. Because UDP is lossy they assume that the kernel will simply drop
any packet that won't fit in the output queue, immediately returning control
to the writing thread. Not true, at least not on Linux. OTOH, Linux as a
default socket buffer size of 65536 which means, again, you only trigger
this bug under very heavy UDP load.