Why are TestMyNet download tests so much slower than other sites?

[jimerman]

I regularly use 3 sites to test my network speed:  Ookla (flash-based), SpeedOf.me (HTML5-based), and TestMy.net.  However, the first 2 seem to report similar results, while TestMy.Net reports very different (and slower) results.

 

Ookla I can rationalize, as they have local servers that might give lower latency.  SpeedOf.Me correlates to Ookla.  I am regularly getting 52 - 60 Mbps download, and 10-12 Mbps upload.

 

However, when I run TestMy - I see 10 Mbps down, and 8 Mbps up.  WAAAY off - this is the auto test.  When I run the Download only test, I got 39 Mbps down, which is still way off.  Does anyone know why?

 

Before I switched ISP, TestMy seemed to be similar results to the other 2, and I really do like the automatic test every hour so I can collect stats across time.

[CA3LE]

I don't know how the other guys do it.  When I test with speedof.me I get 197 Mbps... yeah right.  That would be nice if that was reality.

 

 

I was watching the activity monitor (polling at 1sec) while the test was running.  Which isn't perfect but can give you a good general idea.

 

 

The test is reading 197 Mbps live in the browser when in reality it's running around 180 Mbps.

 

TestMy.net results are calculated from start to finish taking the entire test into account.

 

 

Peaked at 189 Mbps but the overall result was knocked down to 173 Mbps when you consider how quickly (or slowly) the test initiated and any fluctuations during.

 

Take this result for instance.

 

 

Sure, you achieved 21 Mbps but it ramped up really slow and then... ...died out in the middle.  So, would you call that 21 Mbps?  I wouldn't.

 

The numbers here might not correlate with other tests because other tests may only be considering the best portions of your result.

 

First thing I'd do if I were you.  Update OSX and all of your software.  Then, reset your modem and router... leave them unplugged for at least 10 seconds.  Then, if you're on wifi try to connect with ethernet.  If that's not an option then test close to the router.

 

Sometimes my laptop (new macbook pro) will throw low numbers, I then test on one of my wired iMacs and I'm rockin' at full speed.  Sometimes in that scenario just turning wifi off and back on will result in the speed falling back in line with what I'd expect to see.  Other times I have to reset the router.

 

Run a few more multithread tests.  I see that you were able to get up to 57 Mbps in multithread.  That may be a clue.  Run a few multithread tests, then disable multithread and run a few normal download tests.  After turning multithread on and off navigate to the download test page and manually select your test size.  A longer test will tell us more so select 100 MB.  Select the same size each time manually to keep the variables consistent.  You can also manually select the size on the auto test.

 

Let me know when you're done and I'll take a look at your results again.

 

[jimerman]

That is a lot to think about, thanks for that fantastic reply.  As a "lay" person with regards to net speed, it seemed to me like a simple "what's my speed" question, and I did not realize the complexities (and misleading methodologies in some cases) involved.  I'll also try running Performance Monitor and Activity Monitor on the systems I run speed tests from, but since I have 6 people relying on the connection, I don't think I will attempt the disruptive complete disconnect/reconnect.  As far as OS X, I do have all machines set to update to the latest (the one I primarily test from is 10.11), so I don't think it's the OS.

 

That brings up the question of what I should hold my ISP accountable to?  If they say they provide me 50 Mbps average speed, and it tests at 30, they will go to their Ookla test and voila, with multithreading and convenient ignoring of outlier packet statistics, they will say it is 50.  So then do I get into an argument with the tech, whose place it isn't to set policy, and who has no power over it whatsoever?  I guess it's a moot point, but it also reminds me of a rant years ago that the storage industry snuck in a new standard of memory measurement, and that is what they use to advertise disk size, instead of what we were taught in computer class.  Were you aware that the terms "Megabyte" and "Gigabyte" mean, precisely, 1 million bytes and 1 trillion bytes, and not (which I thought) 1,073,741,824 bytes and 1,099,511,627,776 bytes (1024 multiplied out)?  Those numbers I thought, are called Mebibytes and Gibibytes - a highly technical and phonetically confusing obfuscation.  So it looks like this similar obfuscation is going on here.  Oh, and by the way - when they say Mbps, do they mean Megabits or Mebibits????

[jimerman]

Another thought here, up until recently my ISP was AT&T with 18 Mbps, and the 3 speed test web sites I used showed pretty much similar speeds when testing.  Now that I have Comcast with 50 Mbsp, maybe that is why there is more disparity with the speed tests - because it is a higher bandwidth (and slightly different line technology).  I am testing around 40 Mbps with TestMy.net single-threaded test, while multi-threaded is showing almost 60 Mbps.  However, don't web browsers and download managers download multi-threaded anyhow?

[oooonsk]

you guys do know that you need to select a test size of 50 mbs or larger right? I know with my comcast, when you do a small sample size, it shows like 120 mbps, but use a test size of 50 mb. or more and I get about 33.....This is comcast is illegally throttling their internet to allow small downloads to go faster. I see a lot of people posting their speeds, but I bet they did not use the correct sample size

[Sean]

Generally what I check as a real-world test is start a download of a popular Linux ISO such as Ubuntu and pick the nearest fastest mirror they offer, such as in this list for Ubuntu.  Generally with a decent connection, the download will max out the connection and in turn reveals well the ISP sustains the throughput throughout the download.  Just multiply the reported download speed by 8 to convert to Mbps.

 

As my DSL connection is running slow this evening, I can make a demonstration...

 

First, TestMy (Linear) vs Speedtest.net:

 

 

Let's try that Linux download:

 

 

116KB/s = 928Kbps (0.91Mbps).  Now let's try FTP with my webhost - This generally maxes out the connection, e.g. 8.6MB/s on my friend's VDSL link:

 

 

103.5KB/s = 828Kbps (0.81Mbps).  YouTube's "Stats for nerds" is another useful test and is as realistic as it gets for measuring streaming video throughout as it's obviously measuring the speed of what's being played:

 

 

That's 1.24Mbps.  The only way I can get near Ookla's result is with multi-threaded, which is a bit extreme just trying to max out my 4.3Mbps DSL connection!  This is typical in the evening and one of the reasons I got a second Internet connection from a fixed wireless provider.

 

So is it 3.8Mbps or 4.22Mbps?  Well, let's repeat that Linux download with a multi-threaded download manager...

 

I used wxDownload and run it in 5 threads.  As wxDownload does not show the average speed, I took two readings - one at the 2 minute mark and another at the 3 minute mark, so I could work out the average speed over the 60 second duration based on what it downloaded:

 

 

The result is 3.61Mbps.  In fact, as Ookla ran its test, the download figure fluctuated about and it seemed to take the 4.22Mbps peak reading as its measured result.

 

While in this instance the Ookla reading overshot by 16%, I've seen it give results 2 to 3 times of what I can achieve even with multi-threaded downloads, particularly with 3G and 4G (LTE) connections. 

[jimerman]

you guys do know that you need to select a test size of 50 mbs or larger right? I know with my comcast, when you do a small sample size, it shows like 120 mbps, but use a test size of 50 mb. or more and I get about 33.....This is comcast is illegally throttling their internet to allow small downloads to go faster. I see a lot of people posting their speeds, but I bet they did not use the correct sample size

I had been choosing 100 Mbps on TestMyNet.  The other 2 sites don't let me choose sample size.  In contrast to what you say above, when I test 6 MB sample size, I get a slower speed (30 Mbps) than when I test 50 MB and 100 MB, where I get 57 Mbps and 60 Mbps respectively.  And that makes sense intuitively, as a transfer of large data is typically faster than lots of smaller data, simply because of the overhead in establishing and de-establishing communications for the data transfer.

[jimerman]

@editorsean, all great points.  My suggestion was that with a slower connection, the disparity is smaller, and conversely bigger with a faster one.  Also one small correction, you stated to multiply by 8 to figure Mbps - actually you should have said multiply by 8 to figure MBps, but Mbps is typically how line speed is measured, while MBps is how throughput is measured by application software (like web browsers), but truly it is confusing as all hell because take for example Filezilla, it measures in a completely different base of measurement - KiBps (Kibibytes per second), which is the computer science base-2 multiple number.  I don't know why we don't just settle on a single standard of measurement - like the rest of the world has settled on the Metric system, I think the Base-10 is a good way to do it.  (And for heaven's sake, why don't operating systems report disk space in the base-10 instead of base-2?  Drive capacity is sold on Base-10, so you buy a 1 TB drive, but only get 0.7 TiB - which is the same, but confusing?)

 

So, in the interest of cutting through all this confusion, and determining A ) what is the best method to use for testing my Internet speed, and B ) what is the best site to use for testing my Internet speed, here's what I am thinking.

 

Testing a known data provider (like Canonical) is great.  However, we need to provide complete consistency.  Putting a response server as a PoP on a somewhat local backbone does not provide a "real-world" example of, say, downloading a distro ISO.  However, it does give us a "best case scenario" of completely consistent data.  As well, multi-streaming also is realistically how software transfers data, and so it seems tests should also be multi-streamed.  But the ad-hoc discarding of data, that to me is unethical and misleading (seemingly deliberate, and thus the "unethical" label).  So I am glad that TestMyNet does not do this.

 

Testing to a backbone PoP server does provide useful data, eliminating such issues as a "public/commercial" server, and all the connections between it and its backbone.  So the "test" as I see it, is testing "what's my best-case (aka multi-streamed) throughput between me and the backbone" - in order to answer a question like "why is it slow right now downloading the latest OS X from Apple, when usually it goes faster?  Is it some line problem on my end?"

 

For example, last October, I began having intermittent downloads of 2-5 Mbps on an 18 Mbps DSL line.  Obviously something wrong - but where?  Since it was widespread (across sites), it was assumed to be between my computers and the backbone, and not out to the remote endpoints to which I was connecting.  Worse was, when AT&T would come out, the line would show nominal, and speed was also nominal (around 18-24 Mbps), with no reported errors on the system.  With a series of visits and equipment replacement, we finally identified the cause as a faulty connection at the main street junction - and accurate speed tests were an invaluable tool in determining when issues occurred.

@editorsean, I'm a bit curious on the math at the end of your post?  You show the first download, it says size is 681.5 MB, Completed 50.5 MB.  What exactly does this mean, how much was transferred?  I assume 50.5 MB?  So we multiply by 8 to get Mb, and divide by 120 seconds (2m 0s), I get a throughput of 3.37 Mbps.  Likewise, 3.449 Mbps on the second one.  And like I said, maxing out a connection to test its throughput is exactly what I want when I go to a speed test web site - otherwise, if I wanted a real world test, I would just go to an actual site I wanted to download from, and just download.  I want to know, what's the fastest my line can go right now, and is it what my ISP promises?

[Sean]

@editorsean, I'm a bit curious on the math at the end of your post?  You show the first download, it says size is 681.5 MB, Completed 50.5 MB.  What exactly does this mean, how much was transferred?  I assume 50.5 MB?  So we multiply by 8 to get Mb, and divide by 120 seconds (2m 0s), I get a throughput of 3.37 Mbps.  Likewise, 3.449 Mbps on the second one. 

When wxDownload starts its download, it makes a single connection first, probably to check whether the server supports multi-threaded connections on a single file.  Then it establishes the second connection, third and so on until it has all the threads running (in this case 5).  This is why I didn't measure from the start as it takes several seconds until it has all its connections established. 

 

I would have measured between the first and second minute, but happened to be looking away at the time.  So I took the readings between the second minute (50.5MB) and third minute (77.6MB), which gives a 27.1MB difference.  Divided by 60 gives 0.452MB/s and multipled by 8 gives 3.61Mbps. 

[jimerman]

OK, thanks.

[Sean]

Yesterday I couldn't figure out why YouTube consistently showed higher bitrates than other single-threaded transfers and it turns out that YouTube is actually multi-threaded.  When I played a video in 1080p mode and ran TCPView, it showed two active connections to the same server with their received bytes adding up as the video played.  It also seems to vary between IE11 and Firefox.  In IE11, it established three connections, then fell to a single connection for the rest of the video.  So probably a HTML5 video feature.  Interesting...