TCP Connection Setup (Phase 1)
The section of the TCP state transition diagram corresponding to connection setup is shown in Figure 4. A client can connect to a server only when the server is running and waiting to accept connections i.e. in the passive open or LISTEN state. To move to this state, the server application must bind to a chosen and well defined port by invoking the socket calls bind() followed by listen(). This moves the state of the TCP connection on the server to the desired LISTEN state. To setup the TCP connection, the client application must initiate a 3-way handshake, starting by creating a network socket and invoking the connect() socket call (which moves the state of this connection on the client machine from CLOSED to SYN_SENT). If the server machine receives the resulting SYN from the client (first part of the handshake), it moves the connection state from LISTEN to SYN_RECV and transmits a SYN|ACK to the client (the second part of the handshake). Again, if the client machine receives this SYN|ACK, it moves the connection to the ESTABLISHED state and transmits an ACK back to the server (the third part of the handshake). If the server machine receives this ACK, it also moves the connection state to ESTABLISHED, at which point the TCP connection enters Phase 2. (A TCP connection typically spends most of its time in this state.)
Figure 4: TCP state transition during connection setup.
In the preceding paragraph, we have assumed that the two machines are able to communicate successfully over the TCP connection. However, in the presence of abnormalities, the 3-way handshake may not successfully complete. For instance, a firewall between the client and the server may drop either the SYN or ACK packet from the client to the server, or the SYN|ACK response packet from the server to the client. Let us examine each of these situations.
Suppose the state of a TCP connection on the client machine is SYN_SENT i.e., the client application has sent the SYN message to the server and is awaiting the SYN|ACK response. First, suppose that the SYN message is dropped by the firewall and the client application is not informed of this packet drop (as is typical of a firewall). Since the server does not receive the SYN request, it will continue to maintain the TCP connection in the (stable) LISTEN state. In contrast, the client will maintain the TCP connection in the transient SYN_SENT state. Second, consider the situation where the server successfully receives the SYN message and responds with SYN|ACK (while moving the TCP connection to the SYN_RECV state), but this response is dropped by either the server-side or client-side firewall. In this situation, the TCP connection state is transient on both the client and server machines (SYN_SENT and SYN_RECV respectively). Third, suppose the client machine receives the SYN|ACK response but the ACK it sends the server (while moving the TCP connection to the stable ESTABLISHED state) is dropped. Here, the TCP connection on the server remains in the transient state SYN_RECV.
Each of these three scenarios can occur in an operational network. When the connection setup does not complete successfully, it will simply timeout and throw an error to the client application. Note that it is possible to distinguish between each of these cases by observing the TCP connection state on both the server and the client, but developers and network operators are generally unfamiliar with the fine-grained state of TCP connections, and they rarely know how to make inferences based on these observations that can lead to resolving the network/firewall issues.
We now describe hands-on exercises to develop this understanding. In addition to netstat, we will use two further tools: a network utility called netcat (nc)  that can be used to create both a TCP client and a TCP server (for the latter, run with the option –l), and a utility to implement firewalls functionality, called iptables . The first such scenario is shown in Figure 5 (client-side, where the command prompt is set as C_m) and Figure 6 (server-side, where the command prompt is set as S_m). For clarity, command windows are split into an upper and a lower panel, and each command is preceded by date to show the order in which the commands were issued. (Times are shown in IST on the client and in PST on the server. Two different time zones reflect real life situations, where two machines can be geographically apart and operate in different time zones.)
Figure 5: Client connection state with FW dropping SYN
The first two commands are in the lower panel of Figure 5 (issued at 17:03:18 IST and 17:03:37 IST). The first command uses iptables to implement a client-side firewall that is configured to drop all outgoing SYN packets with the destination port 5555. The second command uses netstat to observe the client-side state of the TCP connection on port 5555. Since no such connection has been established yet, only the output of the date command is observed. The third command is in the upper panel of Figure 5 (issued at 17:03:41 IST), which runs a client application initiating a TCP connection to port 5555 on the server with IP address 10.211.55.10. The fourth and fifth commands are two repetitions of the second command, shown in the lower panel of Figure 5 (issued 15 seconds apart at 17:03:45 IST and 17:04:00 IST). Since the connection is now established, netstat produces output showing that the connection has remained in the transient state SYN_SENT for 15 seconds. An experienced developer would immediately note that this is unexpected and would investigate the firewall configuration (ideally at both the client and server side) to diagnose the problem. The commands for the server side are shown in Figure 6. The second command in the upper panel of Figure 6 enables the server to listen for TCP connections on port 5555. As the subsequent netstat commands in the lower panel of Figure 6 show, the server remains in the LISTEN state since no SYN is received.
Figure 6: Server connection setup state
When the firewall is implemented on the server side and is configured to drop SYN|ACK responses from the server to the client, the TCP connection state maintained by both the client and the server (SYN_SENT and SYN_RECV respectively) are transient, as shown in Figure 7 (client-side) and Figure 8 (server-side). We note that when a server is maintaining the state of a TCP connection in the SYN_RECV state and it does not receive the third part of the 3-way handshake message (i.e., the ACK) within a few seconds, it may consider this as a SYN flooding attack and forcibly close the connection. (A detailed description of SYN flooding attacks and possible solutions are given in .) For this reason, one may not actually witness the SYN_RECV state while carrying out this exercise. See Exercise 2 for further details.
Figure 7: Client connection state unaware of Firewall
Figure 8: Server connection state with FW dropping SYN|ACK pkts