I have had a strange issue lately with Windows servers on VMware vSphere 5.1 hosts. Throughput of TCP connections between some virtual machines were very very slow, barely 10 mbit/s .
The behavior was easily reproducible : just start an iperf connection between a Windows Server 2008 and a Windows 2012 server, and you get 10 mbit/s .
Moving both virtual machines on the same ESXi host, or on different hosts, did not change anything.
Changing from E1000 to vmxnet3 or vice-versa did not make any difference.
Disabling some or all network offloading (Large Send Offload, TCP Checksum Offloading, etc.) did not make any difference either.
Out of despair, I tried to add the “-N” to my iperf command. That flag disables Nagle’s algorithm.
Instantly, the throughput went from 10 mbit/s to over 800 mbit/s .
Finally, after a bit of Googling, I ended up disabling Nagle’s algorithm and delayed TCP Acknowledgment in Windows registry. I will continue to monitor the behavior of the servers, but it fixed the throughput issues and latency is not affected.
You can do this by editing your registry HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces and, in each interface folder, add the two following 32bit D-WORD:
Name:TCPNoDelay Value:1
Name:TcpAckFrequency Value:1