Why do all the speed test websites differ, which to use?

Inconsistent speed testsThe most common speed test people use to check their web connection throughput is Ookla’s Speedtest.net service. While it’s pretty good at measuring the potential throughput available between the user and its ISP’s internal network, the actual throughput the user receives from the Internet can be much lower, especially when it comes to congested networks such as cellular and wireless connections.

Ookla’s Speedtest has quite a sophisticated way of measuring the throughput. Going by its methodology, it makes multiple connections to its test server and divides the test up into 20 time slices. Once the test completes, it discards the slowest 30% and fastest 10% time slice measurements and averages the rest. Due to the unpredictability of the Flash running the plug-in, this helps avoid unexpected spikes and dips affecting the test result.

In theory, over a consistent fixed broadband connection with no packet loss, its test results will match up to any real world download or upload with a fast enough host. However, when it comes to congested connections with fluctuating throughput or packet loss, its test results can be far greater than what’s achievable with actual real world web traffic.

Don’t skew the test!

Regardless of what speed test you choose, a speed test must only be carried out when nothing else is using the connection.  For example, background traffic such as a streaming set top box or a phone backing up pictures will skew the speed test, similarly if the speed test is carried over Wi-Fi or HomePlugs (which carries data over the building’s electrical wiring).  Ideally, the device carrying out the speed test should be connected directly to the router with a network cable and every other web connected device disconnected or switched off.

Speed test comparison, which to use

This section is split up into multiple test types – Single thread tests to test streaming performance, multithread tests to test the link capacity and and a peak capacity test to measure the current ‘Up to’ speed.

Download / streaming performance

TestMy.net (default mode), M-Lab (also in Google search) and Leaseweb’s file test (see below) – Most web large downloads and streaming services operate over a single connection to the server, so it makes sense to measure the throughput available over a single connection. Personally this is my preferred test when comparing ISPs even though it may not show what the connection is capable of with multiple simultaneous connections. If you use TestMy, make sure the chosen server (top-right of the page) is nearby, such as ‘London, UK’ for tests within Ireland.

As most browsers show the download speed in real-time, Leaseweb’s file test is pretty useful for checking how stable the speed is, especially if speed tests are giving inconsistent results one test after the next.  To see the speed in Mbps and a traffic graph, just bring up the Windows Task Manager, go into the Performance tab and click the Ethernet or Wi-Fi network icon showing activity.

The Internet speed test Google provides is run by M-Lab:

Google M-Lab speed test

Other single thread tests:

SpeedOf.Me – Block based testing, test with progressively larger block sizes to measure the average speed over a minimum period, similar to TestMy. Shows a visual graph of the individual test blocks.

Comparitech – Tests with a single TCP SSL connection throughout the download test, again in the upload direction. For each user that runs a speed test, they donate $0.01 and split the raised funds between Computers 4 Africa and the Electronic Frontier Foundation (EFF).

Sustained capacity measurement

Netflix’s test Fast.com makes multiple connections to its content server to saturate the link, basically to mimic how Netflix itself connects to stream high bandwidth content such as 4K video. Going by its test methodology, it ignores the initial ramp-up period when the test starts.  It then measures the average throughput over the rest of the test, including any dips or fluctuations.

Fast extended testIt would not make sense for Netflix to ignore throughput dips in its test since these dips will have an impact on streaming video performance, so its test result should be accurate to what can be achieved with a multi-part download manager.  NetFlix recently updated Fast with an upload and latency test.  It shows latency when idle and during the test.  Click the ‘Show more info’ button after the test completes to see the additional test detail as shown on the right.  Click the Settings to specify the maximum test time and the number of threads.

TestMy in multithread mode works similar by downloading multiple parts simultaneously to work out the average throughput for the lot.  This also simulates the way a webpage with multiple images loads as web browsers download multiple images simultaneously.  Like it’s linear test, TestMy measures the throughput over the entire block size.  If the test completes within 7 seconds, it automatically retests with a larger block sizes.  It also has the capability to test against multiple servers simultaneously during a test to simulate a multithread download against multiple servers, such the way BitTorrent downloads.

DSLReports is also a multithread test, designed to test a wide variety of connection types.  It first starts by asking what type of Internet connection is being tested, e.g. Cable, DSL, etc.  It then measures the latency against multiple servers and runs the test with multiple TCP connections in each direction.  In addition to the upload, download and ping results, it measures Bufferbloat, which is the latency in each direction during the tests. For example, a high Bufferbloat would be a good indicator of bad VoIP quality and online game rubberbanding during a multithread download or upload.

Peak throughput between user & ISP

Speedtest.net – While I don’t find its test results particularly useful for comparing ISPs, it does a pretty good job at measuring what the link is capable of delivering between the user and ISP.  An example would be when making adjustments to ISP connection or antenna for a wireless provider to measure the current ‘Up to’ speed. It uses the nearest server, makes multiple connections to max out the connection and ignores dips (e.g. brief throughput drops, packet loss, etc.) that would lead to a lower average result.  As Ookla hosts servers at ISPs, some tests may run without any data traffic reaching the Internet, such as a Vodafone customer testing with a Vodafone server.

Real time downlink capacity to adjust an antenna

Run multiple simultaneous downloads from Leaseweb’s file test site. With the exception of adjusting the computer’s Wi-Fi antenna (see below), the PC should be connected directly to the router with a wired connection and this should be carried out off-peak such as early on a weekend morning. Bring up the Windows Task Manager and go into the ‘Performance’ tab. Then go into Ethernet or Wi-Fi on the left depending on which shows activity or how your computer is connected.

Start 4 downloads of the Leaseweb’s 1GB file test, which should be adequate to saturate the download capacity of most Internet connections. With the Task Manager network graph in view, adjust the antenna to achieve the highest sustained graph.

Just beware that this technique is very bandwidth intensive on high speed connections, so can heavily run up a metered Internet connection.  As a rough guide, a connection that sustains 45Mbps will consume roughly 1GB every 3 minutes.

Measuring and optimising Wi-Fi speed

For adjusting optimally positioning a router or PC’s Wi-Fi antenna, I recommend using a utility such as NetStress between a pair of computers.  Connect one computer directly to the router with an Ethernet cable.  The utility will measure the throughput between the two computers.  It’s then just a matter of tweaking the Wi-Fi antennas to maximise the throughput readings.  Try also unplugging equipment to check whether any device is affecting the Wi-Fi throughput, especially nearby power supplies.

Stopwatch test

One accurate way to measure speed is to do an actual file download from a fast web server where the congestion is unlikely to be at the server end. Leaseweb’s test file site offers a variety of test files. Just note that as its nearest mirror may be overseas, the speed may be affected by the performance of the international link the traffic is carried over. Generally, this is only an issue with very fast ISPs such as over 100Mbps and those with poor peering.

Go to Leaseweb’s download test page and have your stopwatch ready. Choose an appropriate file size according to your connection – 10MB for up to about 5Mbps, 100MB for up to about 50Mbps and 1GB for faster connections. Start the stopwatch the moment link is clicked and choose somewhere handy to save the dummy file if requested. Note that the download is running even before choosing where to download, hence why the timing should start the moment the link is clicked. Stop the stopwatch the moment the download completes.

Divide the number of seconds the download took into the file size to get the speed in MB/s. Multiply the figure by 8 to convert into Mbps. For example, let’s say the 100MB file took 25 seconds to download, then 100 / 25 = 4MB/s. To convert into Mbps, 4 x 8 = 32Mbps. For test results under 1Mbps, multiply the figure by 1024 to convert into Kbps. For example, if the result was 0.5Mbps, then 0.5 x 1024 = 512Kbps.

If the download stalls for a few seconds at the start or end, try the download in another web browser or temporarily disable your antivirus software. Some anti-virus products analyse each file download as it starts or just before it completes and this delay will potentially result in a lower than expected speed test result.

How congestion & fluctuation affect tests

Ideally it should be possible to get the full throughput with a single connection to the server such as when downloading a large ISO image or uploading a high definition video file. Fibre and DSL based connections are generally capable of delivering this, especially with fast nearby hosts. For example, a large download from Heanet should have no problem maxing out one’s Internet connection in Ireland.

In reality, shared Internet connections such as 3G/4G (LTE) and wireless slow down the more users that are online using the same cell tower or mast. This can also happen if the network connecting the masts is saturated, especially since many relay their traffic through other masts with point-to-point microwave links. So even if the local mast has plenty of spare capacity, the mast it relays its traffic through may be severely congested.

Linear vs Multi-threaded connections

Generally, as congestion builds up, the throughput on individual connections starts to suffer. So let’s say a user subscribed to a 30Mbps fixed wireless link gets 512KB/s (=4Mbps) maximum no matter what they try downloading, the congestion at some point along the route between the user and the ISP has resulted in connections getting an average of 4Mbps. In this case, the user could try using a download accelerator that splits the download in 4 parts so it can make 4 connections to download them simultaneously. In theory, this can potentially bring the throughput up to about 2MB/s (=16Mbps).

While running simultaneous connections may speed up file transfers, some connections cannot be sped up like this. Most streaming services such as YouTube make a single connection to the content server to stream the video. So even if a user can get 16Mbps with a 4 simultaneous connections, the video streaming service would still be limited to 4Mbps over a single connection, insufficient for many high definition streaming services. A similar issue applies with file transfers that can be only made over a single connection, such as many large file sending services, e.g. WeTransfer.

This is where we run into the first problem with Ookla’s Speedtest – It always runs its test over multiple connections to the test server, usually 4 to 8 connections. So let’s say that person getting only 4Mbps per connection runs a test on speedtest.net, they will most likely get around 16Mbps and probably be confused why YouTube is struggling to play in 1080p despite what appears like ample bandwidth.

Fluctuating bandwidth

Wireless networks generally suffer from bandwidth fluctuation as a result of weather conditions, interference and even objects such as trees. The bandwidth can also fluctuate simply a result of intermittent traffic by other users on the same mast.

For example, let’s say a download fluctuates between 2Mbps and 16Mbps and averages to about 6Mbps over an extended period of time. Depending on the speed test methodology, one speed test can return a completely different figure to another, especially those that ignore throughput dips and spikes.

For example, as Ookla’s Test discards 30% of the slowest measurements during its test, it is possible for the connection throughput to dip way down for up to 30% of the test duration without affecting the test result. A good real world example would be running Ookla’s Speedtest in a moving vehicle where trees and buildings are intermittently obscuring the signal. With a real download or stream, there would be no way to achieve the test result speed as the test result excluded the traffic dips during its test.

Port prioritisation / congestion

Web traffic is carried across a range of protocols (mainly TCP and UDP) and port numbers 1 to 65535. Although ISPs are supposed to treat traffic equally, the actual throughput can vary across port numbers such as if an ISP decides to prioritise or throttle certain types of traffic or if it has content filtering in place on certain port numbers that is struggling to keep up with the flow of traffic.

The most common ports used are 21 for FTP, 53 for DNS, 80 for HTTP, 443 for HTTPS and 25/110 for e-mail. With most web traffic including streaming video being carried over ports 80 and 443, it would make sense to run speed tests using either port to get a realistic test result. However, some test services choose seldom used port numbers, with 8080 being the port number Ookla uses.

This makes it fairly straight forward for Internet Providers to manipulate speed test traffic that uses uncommon port numbers. For example, if an ISP prioritises traffic over port 8080, Ookla Speedtest can potentially deliver substantially higher test results than what can be achieved with real world traffic that is delivered over common port numbers such as 80 and 443.

One fairly reliable way of checking for port prioritisation is to compare the download speed reported by fast.com and speedtest.net. As both tests run their threads multi-threaded, they should deliver similar results. If speedtest.net delivers substantially higher test results (e.g. 50% or more higher), this is a fairly good indication of traffic manipulation as fast.com runs its test over port 443, the same port most secure websites and streaming services use such as YouTube and Netflix.

TestMy runs its tests over the standard http port 80, the port number that most websites and file downloads run over.  So a fairly good indication if an ISP manipulating ordinary web traffic is to compare testmy.net in multithread mode against fast.com.  If fast.com delivers substantially lower test results, especially 4Mbps or lower when testmy.net is delivering much higher figures, this is a fairly good indication that the ISP is throttling port 443 to limit the video resolution of streaming services.

Streaming bitrates

Streaming services such as YouTube and Netflix require a minimum sustained download speed to reliably stream video.  In fact, a slow connection with a consistent speed is preferable over a fast connection with erratic throughput. This is especially important for live streaming such as IPTV, which cannot build-up a large buffer to counteract speed dips.

If the speed drops during playback, the time it takes for the player to connect to the next lower resolution will potentially result in the infamous buffering wheel appearing.  Some streaming services may even maintain the lower resolution for the remainder of the playback even if more bandwidth becomes available.

High packet loss and even brief intermittent drop-outs can render most streaming services unusable or at least make them uncomfortable to use.

The following is a rough guide what sustained connection speeds are required for each step up in resolution:

  • <1Mbps – 360p video (VHS quality)
  • 1-2Mbps – 480p video (Standard Definition, broadcast quality)
  • 3-4Mbps – 720p video (High Definition, about HD broadcast quality)
  • 5-8Mbps – 1080p video (Full HD, HD broadcast quality)
  • 10-25Mbps – 4K video (Ultra HD broadcast quality)

Throughput vs data usage

On metered connections such as many 4G based ISPs, a high throughput can lead to rapid use of the data plan limit, especially with streaming services that will automatically play in 1080p or potentially even 4K if available.

The following gives a rough idea of how much data can be downloaded or uploaded for a fully saturated connection under typical conditions:

  • 40Kbps – 0.29MB/minute – 17.5MB/hour (Dial-up typical, GPRS)
  • 200Kbps – 1.46MB/minute – 88MB/hour (Edge)
  • 1Mbps – 7.5MB/minute – 450MB/hour
  • 2Mbps – 15MB/minute – 900MB/hour (Long DSL line)
  • 5Mbps – 37.5MB/minute – 2.2GB/hour (DSL typical)
  • 8Mbps – 60MB/minute – 3.5GB/hour
  • 10Mbps – 75MB/minute – 4.4GB/hour (3G HSPA)
  • 15Mbps – 113MB/minute – 6.6GB/hour (ADSL2 typical)
  • 20Mbps – 150MB/minute – 8.8GB/hour (3G HSPA+)
  • 25Mbps – 188MB/minute – 11GB/hour
  • 30Mbps – 225MB/minute – 13.2GB/hour
  • 50Mbps – 375MB/minute – 22GB/hour (LTE rural)
  • 75Mbps – 563MB/minute – 33GB/hour (LTE urban)
  • 100Mbps – 750MB/minute – 44GB/hour (LTE+)
  • 150Mbps – 1.1GB/minute – 66GB/hour (150Mbps FTTH)
  • 300Mbps – 2.2GB/minute – 132GB/hour (Cable / 300Mbps FTTH)
  • 1000Mbps – 7.3GB/minute – 438GB/hour (1Gbps FTTH)

7 thoughts on “Why do all the speed test websites differ, which to use?”

  1. So if Fast.com is using a multi-thread connection for its speed test does that mean that Netflix is doing the same thing when we watch a video from it? It’s not the same with YouTube’s linear connection?

    1. From a quick check, I see that YouTube can stream over multiple threads as shown below with the Windows Resource Meter (YouTube playing in Firefox). Previously when I checked, it would only stream over a single connection.

      YouTube multi-threaded connection

      In Chrome, YouTube connects using QUIC, which is UDP based. This can speed up the connection further as UDP packets can be sent independent of each other, effectively acting like a large number of simultaneous threads. I’m not sure how Netflix connects as I don’t have a Netflix subscription. But based on its Fast.com service being multi-threaded, I reckon their streaming client makes multiple connections, particularly for bandwidth intensive HD or UHD content.

      1. Thanks, Sean.
        I opened a tab in Firefox but I can see a lot of TCP Connections for firefox.exe. How did you know in that screenshot that the remote address IPs are what YouTube is using for?

        1. You can ping the host name shown in the “Stats for nerds” screen, followed by ‘.googlevideo.com’. It should match the remote address in the TCP connection list:

          Pinging a YouTube host name

          1. Thanks for the tip!
            If this is the case then what is the real advantage of using a single connection speed test? What are the things we can learn from it?

          2. The main benefit is to find out what the connection is capable of with a single thread, much like finding out how well a PC CPU performs with a single thread. It can reveal if there is significant network congestion along the route such as with the ISP’s peering. Another benefit is to find out how quickly one can download without having to install multi-threaded download software or where this is not possible (e.g. waiting for Android to download a large system update). For example, most web browsers including Edge, Firefox, Chrome, etc. download each file over a single connection. Smart TVs and set-top boxes with older software will still be limited to streaming over a single connection also.

            If a single connection delivers much faster to an Irish host than a host in let’s say the UK, then the ISP either uses a congested route to the UK or is throttling such traffic. If most of the single connection tests deliver poor performance compared to multithreaded ISPs, then this indicates a congested route between the user and the ISP’s network. Examples of this can be seen with many wireless and 3G/4G ISPs during peak time.

            It would be worth asking this on the TestMy forum as they specialise in single connection testing.

Leave a Reply

Your e-mail address will not be published. Required fields are marked *