Linux TCP FASTOPEN in your sockets

Few years ago the concept of TCP_FASTOPEN (TFO) was introduced as a solution to improve performance on TCP connections reducing one roundtrip of the handshake process. The first operating system that implements TFO is Linux and have been demonstrated good improvements when used in a common network.

The implementation in the Linux Kernels have been made by parts, being Linux Kernel 3.6.1 the first one into implement the client side requirements and then Linux Kernel 3.7 who implements the server side socket behavior.

Client side

In a common TCP client flow, the following calls takes place:

/* create socket file descriptor */
fd = socket(domain, type, protocol);

/* connect to the target server/port */
connect(fd,...);

/* send some data */
send(fd, buf, size);

/* wait for some reply and read into a local buffer */ 
while ((bytes  = recv(fd, ...))) {
    ...
}

When using TCP_FASTOPEN the behavior its a little different. You not longer need to use connect(2), instead you use sendto(2) and it also gives you the opportunity to let the Kernel buffer some initial outgoing data. For short, the call sendto(2) its like an implicit connect(2) and send/write(2) same time:

/* create the socket */
fd = socket();

/* connect and send out some data */
sendto(fd, buffer, buf_len, MSG_FASTOPEN, ...);

/* write more data */
send(fd, buf, size);

/* wait for some reply and read into a local buffer */ 
while ((bytes  = recv(fd, ...))) {
    ...
}

Server side

A common (old-fashion) TCP server is created with the following calls:

/* create the socket */
fd = socket();

/* connect and send out some data */
bind(fd, addr, addrlen);

/* this socket will listen for incoming connections */
listen(fd, backlog);

Adding TCP_FASTOPEN support to the server side code is very easy, the required changes are minimum, you only need to set a new socket option between bind(2) and listen(2):

/* a hint value for the Kernel */
int qlen = 5;

/* create the socket */
fd = socket();

/* bind the address */
bind(fd, addr, addrlen);

/* change the socket options to TCO_FASTOPEN */
setsockopt(sockfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));

/* this socket will listen for incoming connections */
listen(fd, backlog);

Required macros

Even you are running the latest Linux Kernel 3.8, you will face some problems as in most of the cases the required macro values for  TCP_FASTOPEN and MSG_FASTOPEN will not be available  at compile time. As a workaround you can include the following code in one of your header files:

/* 
 * A generic protection in case you include this 
 * from multiple files 
 */
#ifndef _KERNEL_FASTOPEN
#define _KERNEL_FASTOPEN

/* conditional define for TCP_FASTOPEN */
#ifndef TCP_FASTOPEN
#define TCP_FASTOPEN   23
#endif

/* conditional define for MSG_FASTOPEN */
#ifndef MSG_FASTOPEN
#define MSG_FASTOPEN   0x20000000
#endif

#endif

Enabling TCP_FASTOPEN in your Kernel

By default the TCP_FASTOPEN feature is not enabled at runtime (unless you instructed that in the sysctl.conf file). Before to test this new feature make sure is enabled with the following command:

# echo 1 > /proc/sys/net/ipv4/tcp_fastopen

If you try to use a client with TCP_FASTOPEN enabled, its mandatory that the server have this same option set in the listener socket, otherwise the client will faill at the connection phase (due to protocol mismatch).

For a TCP_FASTOPEN server, it does not matter if the client uses the new protocol or not, it will work anyways. So if you develop a TCP server you can give it a try adding a simple system call to add this feature. If your project is open source, feel free to use the header macros example provided above.

Btw, of course Monkey Web Server have added this feature recently in our development repository

References