Using Iperf to Diagnose Network Speed Issues

Many folks are unaware of the dramatic effect disk speeds have on there network speeds, since the bulk of your data comes from or goes to disk, and mass storage is the slowest link in the chain, it is the single most important factor in how fast you can download media home, download torrent payloads, or even upload to others.

The chain is:

DISK <==> Cache/Memory <==> Network

When tuning a system these are the three components that garner the most attention: how can I get things on to and off of my disk quickly; how best to optimize my cache; and then finally how can I insure my network is running at its peak.

In a Reddit post we looked at how best diagnose route and peering issues using the linux command mtr. But mtr looks at road conditions, what is likely slowing you down. It doesn't address how fast you can actually go. We can do that with Iperf, and significantly iperf removes your disk from the mix. MTR lays out the map, the roads you are taking, any potholes, traffic jams, bad interchanges, each turn and curve along the way. Iperf looks at how fast your car can go down that route, pure speed, in real numbers. MTR lays out the track, iperf runs the race.

This is a tutorial, it is recommended you follow along using your server and your home machine to run the commands in parallel.

Iperf, or its latest incarnation, iperf3, is a client/server tool. You run iperf on your server, then from a client connect and see how fast your server either receives or can send you packets. It is really quite simple.

First we need to install iperf3 on your server, as always if you have superuser this is simple, if not a bit harder, with sudo (everything here is debian or ubuntu):

sudo apt-get install iperf3

If you don't have root you'll need to manually install it, here is what you need to do:

mkdir -p ~/bin ~/lib wget --no-check-certificate -O ~/bin/iperf3 https://iperf.fr/download/ubuntu/iperf3_3.1.3 chmod +x bin/iperf3 export PATH=$PATH:~/bin wget --no-check-certificate -O ~/lib/libiperf.so.0 https://iperf.fr/download/ubuntu/libiperf.so.0_3.1.3 echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/lib" >>~/.bashrc source ~/.bashrc

Running things on the client is now easier than tying your shoes:

iperf -p 10666 -s

is the port that we will be using, the server will be listen for connections on. It is optional, but since the default port might be taken (5201), I've included how to set your own.

Now you need to install the client, a client for virtually any platform can be found over at iperf.fr, in particular https://iperf.fr/iperf-download.php

I'm going to assume you have windows, but it is largely the same for any non-Unix client, you download the zip file, unzip it into its own directory, then open a DOS or command window in that directory. With Windows, iperf3 uses the cygwin runtime in the form of  DLL(s).

Ok, with our server set up, and the client installed, a quick aside for those unanointed in parlance of networking. The two most common Internet protocols are TCP (Transmission Control Protocol) and UDP (User Datagram Protocol). TCP is like a phone call, a circuit is created between the caller and the person being called, they say something you'll hear it unless the whole line drops, it is reliable. UDP is more like sending a letter, you drop a packet on the wire with an address and best effort is made to get it delivered, it is inherently unreliable. We'll be using both in our tests. Thing to understand is UDP it is cheaper, and its unreliability can be useful.

Lets start with the simplest case:

iperf3 -p 10666 -c server.hostname.or.ip

This will SEND packets to the server for 10 seconds, and tell you how fast the packets went, the output will look like:

E:\mtrs\iperf-3.1.3-win64>iperf3 -p 10666 -c iperftest.chmuranet.com Connecting to host iperftest.chmuranet.com, port 10666 [  4] local 192.168.1.19 port 10358 connected to .... port 10666 [ ID] Interval           Transfer     Bandwidth [  4]   0.00-1.00   sec  1.00 MBytes  8.39 Mbits/sec [  4]   1.00-2.00   sec  2.25 MBytes  18.9 Mbits/sec [  4]   2.00-3.00   sec  2.25 MBytes  18.9 Mbits/sec [  4]   3.00-4.00   sec  2.25 MBytes  18.9 Mbits/sec [  4]   4.00-5.00   sec  2.25 MBytes  18.9 Mbits/sec [  4]   5.00-6.00   sec  2.25 MBytes  18.9 Mbits/sec [  4]   6.00-7.00   sec  2.25 MBytes  18.9 Mbits/sec [  4]   7.00-8.00   sec  2.25 MBytes  18.9 Mbits/sec [  4]   8.00-9.00   sec  2.25 MBytes  18.9 Mbits/sec [  4]   9.00-10.00  sec  2.25 MBytes  18.9 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bandwidth [  4]   0.00-10.00  sec  21.2 MBytes  17.8 Mbits/sec                  sender [  4]   0.00-10.00  sec  21.2 MBytes  17.8 Mbits/sec                  receiver

The last set of numbers  (sender / receiver) is our summary line, average speed. This is in megabits per second, so my upload speed is 2.23MB/s.

This though is just a test of how fast I can talk to my server, which isn't nearly as interesting as how fast my server can talk to me, to do that we need to tell iperf to receive. We add the  parameter

E:\mtrs\iperf-3.1.3-win64>iperf3 -p 10666 -c iperftest.chmuranet.com  -R

[  4]   0.00-1.00   sec  1.62 MBytes  13.6 Mbits/sec [  4]   1.00-2.00   sec  1.65 MBytes  13.8 Mbits/sec [  4]   2.00-3.00   sec  1.07 MBytes  8.99 Mbits/sec [  4]   3.00-4.00   sec   814 KBytes  6.67 Mbits/sec [  4]   4.00-5.00   sec   510 KBytes  4.18 Mbits/sec [  4]   5.00-6.00   sec   446 KBytes  3.66 Mbits/sec [  4]   6.00-7.00   sec   205 KBytes  1.68 Mbits/sec [  4]   7.00-8.00   sec  99.8 KBytes   818 Kbits/sec [  4]   8.00-9.00   sec   141 KBytes  1.16 Mbits/sec [  4]   9.00-10.00  sec   345 KBytes  2.83 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bandwidth       Retr [  4]   0.00-10.00  sec  8.83 MBytes  7.41 Mbits/sec   22             sender [  4]   0.00-10.00  sec  6.97 MBytes  5.85 Mbits/sec                  receiver

Now this is interesting, my speed down is hitting Sunday night traffic, and is significantly slower. See the number of retries (Retr)? There is some problem on the down route, you can also see the jitter (variations in latency or speed).

Now you should compare this to the speed your are getting on a single FTP connection using like Filezilla. Checking my connection, I'm getting .5MB/s that means the disk is costing me 25% of my download speed, not unusual. If the disk is re

To see how fast I can get things to, like segmented FTP, I'm going to need to use multiple simultaneous connections, lucky iperf3  can do that too. Lets do 10 segments, with the  parameter.

E:\mtrs\iperf-3.1.3-win64>iperf3 -p 10666 -c iperftest.chmuranet.com -R -P 10

[SUM]   0.00-10.00  sec  30.2 MBytes  25.3 Mbits/sec  239             sender [SUM]   0.00-10.00  sec  26.0 MBytes  21.8 Mbits/sec                  receiver

Just the summary lines, but you can see much faster. Lets try 30.

[SUM]   0.00-10.00  sec  77.0 MBytes  64.6 Mbits/sec  947             sender [SUM]   0.00-10.00  sec  64.4 MBytes  54.0 Mbits/sec                  receiver

That scales rather nicely, I am receiving 6.75MB/s, but look at my retransmits, that has scaled too. Server is sending at over 8MB/s.

At this point further escalation of connections, doesn't scale nearly as well.

OK, lets see what might be going on, we'll use UDP now (by adding the  parameter), with 10 connections, from me to my server, :

E:\mtrs\iperf-3.1.3-win64>iperf3 -p  10666 -c iperftest.chmuranet.com  -P 10 -u

[SUM]   0.00-10.00  sec  15.6 MBytes  13.1 Mbits/sec  2.989 ms  0/1998 (0%)

No packet loss, of the 1998 packets sent 0% were lost, at a speed of 13.1Mbits/s - pretty good.

Now going the other way  (-R):

E:\mtrs\iperf-3.1.3-win64>iperf3 -p  10666 -c iperftest.chmuranet.com  -P 10 -u -R

[SUM]   0.00-10.00  sec  12.7 MBytes  10.6 Mbits/sec  1.192 ms  9/1620 (0.56%)

Fewer packets sent and an overall loss of .56% of the packets wandered off. I'm willing to bet if I did an mtr, I'd see some bad jitter, and probably some packet loss.

Yep, checking mtr, we see where the problem is:

4. ae-21-52.car1.Chicago1.Level3.net            1.9%    340   95.5  95.6  95.4  97.0   23

1.9% packet loss in Chicago, those ruffians.

So you can see, iperf can be an effective tool, primarily for determining the hit you are taking because of your hard disk, but also in analyzing route dynamics, what is costing you against your ideal. I've covered most of the interesting parameters, the only other flag I use is  as in , so that iperf will run for longer (30 secs vs 10 sec default), which can give you a better picture. But with all linux commands, you should take a look at the manpage yourself: https://www.mankier.com/1/iperf3

''Disclaimer: I fiddled with the numbers to make this more interesting, I also ran a laptop against a Chmuranet 10G server, a DC based CPU-laden 10G server can send packets much faster then my poor laptop can handle, this is part of the packet loss you saw in my examples. Consider this when looking at your own server.''