Why do all the speed test websites differ, which to use?

Inconsistent speed testsThe most common speed test people use to check their web connection throughput is Ookla’s Speedtest.net service. While it’s pretty good at measuring the potential throughput available between the user and its ISP’s internal network, the actual throughput the user receives from the Internet can be much lower, especially when it comes to congested networks such as cellular and wireless connections.

Ookla’s Speedtest has quite a sophisticated way of measuring the throughput. Going by its methodology, it makes multiple connections to its test server and divides the test up into 20 time slices. Once the test completes, it discards the slowest 30% and fastest 10% time slice measurements and averages the rest. Due to the unpredictability of the Flash running the plug-in, this helps avoid unexpected spikes and dips affecting the test result.

In theory, over a consistent fixed broadband connection with no packet loss, its test results will match up to any real world download or upload with a fast enough host. However, when it comes to congested connections with fluctuating throughput or packet loss, its test results can be far greater than what’s achievable with actual real world web traffic.

Don’t skew the test!

Regardless of what speed test you choose, a speed test must only be carried out when nothing else is using the connection.  For example, background traffic such as a streaming set top box or a phone backing up pictures will skew the speed test, similarly if the speed test is carried over Wi-Fi or HomePlugs (which carries data over the building’s electrical wiring).  Ideally, the device carrying out the speed test should be connected directly to the router with a network cable and every other web connected device disconnected or switched off.

Speed test comparison, which to use

Download / streaming performance: TestMy.net (default mode), M-Lab (also in Google search) and Leaseweb’s file test (see below) – Most web large downloads and streaming services operate over a single connection to the server, so it makes sense to measure the throughput available over a single connection. Personally this is my preferred test when comparing ISPs even though it may not show what the connection is capable of with multiple simultaneous connections. If you use TestMy, make sure the chosen server (top-right of the page) is nearby, such as ‘London, UK’ for tests within Ireland.

As most browsers show the download speed in real-time, Leaseweb’s file test is pretty useful for checking how stable the speed is, especially if speed tests are giving inconsistent results one test after the next.  To see the speed in Mbps and a traffic graph, just bring up the Windows Task Manager, go into the Performance tab and click the Ethernet or Wi-Fi network icon showing activity.

The Internet speed test Google provides is run by M-Lab:

Google M-Lab speed test

Sustained capacity measurement (downlink):Fast.com and TestMy.net (multithread mode) – Netflix’s test Fast.com makes multiple connections to its content server to saturate the link, basically to mimic how Netflix itself connects to stream high bandwidth content such as 4K video. It would not make sense for Netflix to ignore throughput dips in its test since these dips will have an impact on streaming video performance, so its test result should be pretty accurate to what can be achieved with a multi-part download manager. TestMy in multithread mode works similar by downloading multiple parts simultaneously to work out the average throughput for the lot.  This also simulates the way a webpage with multiple images loads as web browsers download multiple images simultaneously.

Peak throughput between user & ISP: Speedtest.net – While I don’t find its test results particularly useful for comparing ISPs, it does a pretty good job at measuring what the link is capable of delivering between the user and ISP, such as when making adjustments to ISP connection or antenna for a wireless provider. It uses the nearest server, makes multiple connections to max out the connection and ignores dips (e.g. brief throughput drops, packet loss, etc.) that would lead to a lower average result.  As Ookla hosts servers at ISPs, some tests may run without any data traffic reaching the Internet, such as a Vodafone customer testing with a Vodafone server.

Real time capacity measurement (downlink) to adjust an antenna: Run multiple simultaneous downloads from Leaseweb’s file test site. With the exception of adjusting the computer’s Wi-Fi antenna, the PC should be connected directly to the router with a wired connection and this should be carried out off-peak such as early on a weekend morning. Bring up the Windows Task Manager and go into the ‘Performance’ tab. Then go into Ethernet or Wi-Fi on the left depending on which shows activity or how your computer is connected.

Start 4 downloads of the Leaseweb’s 1GB file test, which should be adequate to saturate the download capacity of most Internet connections. With the Task Manager network graph in view, adjust the antenna to achieve the highest sustained graph.

Just beware that this technique is very bandwidth intensive on high speed connections, so can heavily run up a metered Internet connection.  As a rough guide, a connection that sustains 45Mbps will consume roughly 1GB every 3 minutes.

Stopwatch test

One accurate way to measure speed is to do an actual file download from a fast web server where the congestion is unlikely to be at the server end. Leaseweb’s test file site offers a variety of test files. Just note that as its nearest mirror may be overseas, the speed may be affected by the performance of the international link the traffic is carried over. Generally, this is only an issue with very fast ISPs such as over 100Mbps and those with poor peering.

Go to Leaseweb’s download test page and have your stopwatch ready. Choose an appropriate file size according to your connection – 10MB for up to about 5Mbps, 100MB for up to about 50Mbps and 1GB for faster connections. Start the stopwatch the moment link is clicked and choose somewhere handy to save the dummy file if requested. Note that the download is running even before choosing where to download, hence why the timing should start the moment the link is clicked. Stop the stopwatch the moment the download completes.

Divide the number of seconds the download took into the file size to get the speed in MB/s. Multiply the figure by 8 to convert into Mbps. For example, let’s say the 100MB file took 25 seconds to download, then 100 / 25 = 4MB/s. To convert into Mbps, 4 x 8 = 32Mbps. For test results under 1Mbps, multiply the figure by 1024 to convert into Kbps. For example, if the result was 0.5Mbps, then 0.5 x 1024 = 512Kbps.

If the download stalls for a few seconds at the start or end, try the download in another web browser or temporarily disable your antivirus software. Some anti-virus products analyse each file download as it starts or just before it completes and this delay will potentially result in a lower than expected speed test result.

How congestion & fluctuation affect tests

Ideally it should be possible to get the full throughput with a single connection to the server such as when downloading a large ISO image or uploading a high definition video file. Fibre and DSL based connections are generally capable of delivering this, especially with fast nearby hosts. For example, a large download from Heanet should have no problem maxing out one’s Internet connection in Ireland.

In reality, shared Internet connections such as 3G/4G (LTE) and wireless slow down the more users that are online using the same cell tower or mast. This can also happen if the network connecting the masts is saturated, especially since many relay their traffic through other masts with point-to-point microwave links. So even if the local mast has plenty of spare capacity, the mast it relays its traffic through may be severely congested.

Linear vs Multi-threaded connectios

Generally, as congestion builds up, the throughput on individual connections starts to suffer. So let’s say a user subscribed to a 30Mbps fixed wireless link gets 512KB/s (=4Mbps) maximum no matter what they try downloading, the congestion at some point along the route between the user and the ISP has resulted in connections getting an average of 4Mbps. In this case, the user could try using a download accelerator that splits the download in 4 parts so it can make 4 connections to download them simultaneously. In theory, this can potentially bring the throughput up to about 2MB/s (=16Mbps).

While running simultaneous connections may speed up file transfers, some connections cannot be sped up like this. Most streaming services such as YouTube make a single connection to the content server to stream the video. So even if a user can get 16Mbps with a 4 simultaneous connections, the video streaming service would still be limited to 4Mbps over a single connection, insufficient for many high definition streaming services. A similar issue applies with file transfers that can be only made over a single connection, such as many large file sending services, e.g. WeTransfer.

This is where we run into the first problem with Ookla’s Speedtest – It always runs its test over multiple connections to the test server, usually 4 to 8 connections. So let’s say that person getting only 4Mbps per connection runs a test on speedtest.net, they will most likely get around 16Mbps and probably be confused why YouTube is struggling to play in 1080p despite what appears like ample bandwidth.

Fluctuating bandwidth

Wireless networks generally suffer from bandwidth fluctuation as a result of weather conditions, interference and even objects such as trees. The bandwidth can also fluctuate simply a result of intermittent traffic by other users on the same mast.

For example, let’s say a download fluctuates between 2Mbps and 16Mbps and averages to about 6Mbps over an extended period of time. Depending on the speed test methodology, one speed test can return a completely different figure to another, especially those that ignore throughput dips and spikes.

For example, as Ookla’s Test discards 30% of the slowest measurements during its test, it is possible for the connection throughput to dip way down for up to 30% of the test duration without affecting the test result. A good real world example would be running Ookla’s Speedtest in a moving vehicle where trees and buildings are intermittently obscuring the signal. With a real download or stream, there would be no way to achieve the test result speed as the test result excluded the traffic dips during its test.

Port prioritisation / congestion

Web traffic is carried across a range of protocols (mainly TCP and UDP) and port numbers 1 to 65535. Although ISPs are supposed to treat traffic equally, the actual throughput can vary across port numbers such as if an ISP decides to prioritise or throttle certain types of traffic or if it has content filtering in place on certain port numbers that is struggling to keep up with the flow of traffic.

The most common ports used are 21 for FTP, 53 for DNS, 80 for HTTP, 443 for HTTPS and 25/110 for e-mail. With most web traffic including streaming video being carried over ports 80 and 443, it would make sense to run speed tests using either port to get a realistic test result. However, some test services choose seldom used port numbers, with 8080 being the port number Ookla uses.

This makes it fairly straight forward for Internet Providers to manipulate speed test traffic that uses uncommon port numbers. For example, if an ISP prioritises traffic over port 8080, Ookla Speedtest can potentially deliver substantially higher test results than what can be achieved with real world traffic that is delivered over common port numbers such as 80 and 443.

One fairly reliable way of checking for port prioritisation is to compare the download speed reported by fast.com and speedtest.net. As both tests run their threads multi-threaded, they should deliver similar results. If speedtest.net delivers substantially higher test results (e.g. 50% or more higher), this is a fairly good indication of traffic manipulation as fast.com runs its test over port 443, the same port most secure websites and streaming services use such as YouTube and Netflix.

TestMy runs its tests over the standard http port 80, the port number that most websites and file downloads run over.  So a fairly good indication if an ISP manipulating ordinary web traffic is to compare testmy.net in multithread mode against fast.com.  If fast.com delivers substantially lower test results, especially 4Mbps or lower when testmy.net is delivering much higher figures, this is a fairly good indication that the ISP is throttling port 443 to limit the video resolution of streaming services.

Streaming bitrates

Streaming services such as YouTube and Netflix require a minimum sustained download speed to reliably stream video.  In fact, a slow connection with a consistent speed is preferable over a fast connection with erratic throughput. This is especially important for live streaming such as IPTV, which cannot build-up a large buffer to counteract speed dips.

If the speed drops during playback, the time it takes for the player to connect to the next lower resolution will potentially result in the infamous buffering wheel appearing.  Some streaming services may even maintain the lower resolution for the remainder of the playback even if more bandwidth becomes available.

High packet loss and even brief intermittent drop-outs can render most streaming services unusable or at least make them uncomfortable to use.

The following is a rough guide what sustained connection speeds are required for each step up in resolution:

  • <1Mbps – 360p video (VHS quality)
  • 1-2Mbps – 480p video (Standard Definition, broadcast quality)
  • 3-4Mbps – 720p video (High Definition, about HD broadcast quality)
  • 5-8Mbps – 1080p video (Full HD, HD broadcast quality)
  • 10-25Mbps – 4K video (Ultra HD broadcast quality)

Throughput vs data usage

On metered connections such as many 4G based ISPs, a high throughput can lead to rapid use of the data plan limit, especially with streaming services that will automatically play in 1080p or potentially even 4K if available.

The following gives a rough idea of how much data can be downloaded or uploaded for a fully saturated connection under typical conditions:

  • 40Kbps – 0.29MB/minute – 17.5MB/hour (Dial-up typical, GPRS)
  • 200Kbps – 1.46MB/minute – 88MB/hour (Edge)
  • 1Mbps – 7.5MB/minute – 450MB/hour
  • 2Mbps – 15MB/minute – 900MB/hour (Long DSL line)
  • 5Mbps – 37.5MB/minute – 2.2GB/hour (DSL typical)
  • 8Mbps – 60MB/minute – 3.5GB/hour
  • 10Mbps – 75MB/minute – 4.4GB/hour (3G HSPA)
  • 15Mbps – 113MB/minute – 6.6GB/hour (ADSL2 typical)
  • 20Mbps – 150MB/minute – 8.8GB/hour (3G HSPA+)
  • 25Mbps – 188MB/minute – 11GB/hour
  • 30Mbps – 225MB/minute – 13.2GB/hour
  • 50Mbps – 375MB/minute – 22GB/hour (LTE rural)
  • 75Mbps – 563MB/minute – 33GB/hour (LTE urban)
  • 100Mbps – 750MB/minute – 44GB/hour (LTE+)
  • 150Mbps – 1.1GB/minute – 66GB/hour (150Mbps FTTH)
  • 300Mbps – 2.2GB/minute – 132GB/hour (Cable / 300Mbps FTTH)
  • 1000Mbps – 7.3GB/minute – 438GB/hour (1Gbps FTTH)

7 thoughts on “Why do all the speed test websites differ, which to use?”

  1. So if Fast.com is using a multi-thread connection for its speed test does that mean that Netflix is doing the same thing when we watch a video from it? It’s not the same with YouTube’s linear connection?

    1. From a quick check, I see that YouTube can stream over multiple threads as shown below with the Windows Resource Meter (YouTube playing in Firefox). Previously when I checked, it would only stream over a single connection.

      YouTube multi-threaded connection

      In Chrome, YouTube connects using QUIC, which is UDP based. This can speed up the connection further as UDP packets can be sent independent of each other, effectively acting like a large number of simultaneous threads. I’m not sure how Netflix connects as I don’t have a Netflix subscription. But based on its Fast.com service being multi-threaded, I reckon their streaming client makes multiple connections, particularly for bandwidth intensive HD or UHD content.

      1. Thanks, Sean.
        I opened a tab in Firefox but I can see a lot of TCP Connections for firefox.exe. How did you know in that screenshot that the remote address IPs are what YouTube is using for?

        1. You can ping the host name shown in the “Stats for nerds” screen, followed by ‘.googlevideo.com’. It should match the remote address in the TCP connection list:

          Pinging a YouTube host name

          1. Thanks for the tip!
            If this is the case then what is the real advantage of using a single connection speed test? What are the things we can learn from it?

          2. The main benefit is to find out what the connection is capable of with a single thread, much like finding out how well a PC CPU performs with a single thread. It can reveal if there is significant network congestion along the route such as with the ISP’s peering. Another benefit is to find out how quickly one can download without having to install multi-threaded download software or where this is not possible (e.g. waiting for Android to download a large system update). For example, most web browsers including Edge, Firefox, Chrome, etc. download each file over a single connection. Smart TVs and set-top boxes with older software will still be limited to streaming over a single connection also.

            If a single connection delivers much faster to an Irish host than a host in let’s say the UK, then the ISP either uses a congested route to the UK or is throttling such traffic. If most of the single connection tests deliver poor performance compared to multithreaded ISPs, then this indicates a congested route between the user and the ISP’s network. Examples of this can be seen with many wireless and 3G/4G ISPs during peak time.

            It would be worth asking this on the TestMy forum as they specialise in single connection testing.

Leave a Reply

Your email address will not be published. Required fields are marked *