Don't yell at me - Speedtest.net comparison

[Mike Hammett]

I read the guide explaining why speedtest.net reports higher speeds than testmy.net.

 

http://www.speedtest.net/my-result/3962390815

 

vs

 

https://testmy.net/db/OPnq5Eg(multithread)

 

That's over 3x the difference.

 

That seems to be more than the existing explanations can justify.

[j7n]

I observed high CPU usage when running the multi threaded test at TestMy, which turns it into browser performance test instead. A portion of the time the test just sits there processing something and not receiving any data. It's unusable for me in Opera, which has below average javascript engine.

Best multithreaded test: a well seeding torrent, or a few LeaseWeb bins loaded into a download manager and restarted all at once.

[Mike Hammett]

*nods* Just looking for something that's accurate for the general public to use.

 

I'll try next week comparing my Core2Quad with my i7.

[mudmanc4]

I read the guide explaining why speedtest.net reports higher speeds than testmy.net.

 

http://www.speedtest.net/my-result/3962390815

 

vs

 

https://testmy.net/db/OPnq5Eg(multithread)

 

That's over 3x the difference.

 

That seems to be more than the existing explanations can justify.

The multi threaded test is setup to push the connection and browser to the limits, which rely on hardware performance themselves, and to supply the test participant with a real world hard result.

This test function is however not setup or meant to print results that make the user believe anything outside of very specific qualifications concerning data TX/RX paths, yet fluent or not so fluent transmissions. 

[Mike Hammett]

I can fill the GigE port with iPerf, so I'm sure the hardware is capable.

 

The rest of what you said doesn't really make sense. What I read it as was, "This test just throws some numbers out there that aren't really relevant to anything." However, I assume that's not what you meant.

[nanobot]

The difference of the two you say?

Speedtest.net uses a pure flash-based approach, and drops the top and bottom 5% of your tested speed. Why? Because the first instant of testing is typically slower due to the requirement to establish a connection. Then there's often a sudden burst which is typically irrelevant to your speed. Speedtest.net is more useful for a total-throughput test. It's often handy for gamers to test what the typical speed of their connection is. (After all, they are using UDP mostly which is connectionless, so there is no establishment of a connection.)

Testmy.net uses a pure HTTP based approach, and does not remove any data. Why? Because almost all of the typical users network traffic is over HTTP, which requires the establishment of a connection. (Downloads, webpages, style-sheets, images, etc.) Testmy.net is much more relevant to testing your total connection throughput. Basically giving you a true establishment of how websites should download.

Thanks,

EBrown

[mudmanc4]

The multi threaded test is setup to push the connection and browser to the limits….

 

 

I can fill the GigE port with iPerf, so I'm sure the hardware is capable.

 

The rest of what you said doesn't really make sense. What I read it as was, "This test just throws some numbers out there that aren't really relevant to anything." However, I assume that's not what you meant.

 

I hope the purified comment helps you understand. 

 

The multi test here is pulling from more than one resource at a time. From real world locations. Real data, in the attempt to show what a true connection stream appears as. 

 

testing more than one connection , more thn one stream, more than one thread. 

 

Look into how to activate or use firefox multi threading. 

[CA3LE]

Your result at speedtest.net is only 4ms from the server.  TestMy.net is routing you across the Internet to locations where real websites are hosted.  Not on the edge of your providers network.  Of course your faster if you don't really go out to the Internet.  It's like downloading from within your own network.  That's NOT your Internet speed.  We should call that your ultimately ideal situation speed. 

 

The multithread test is designed to be harder.  Just now I ran a normal download test and CPU maxed at 100% (one of my cores).  I ran a multithread test and hit 275% (nearly 3 cores).  But I've found that it's not the same case for Mercury tests (which are also multithread).  A Mercury test vs netflix.com I saw a peak of 110%.  So you can multithread with Mercury and it's not quite as CPU intensive. Always keep in mind that TestMy.net isn't just an Internet test, it's also a test of the client computer and in the case of Mercury it's also a test of the server. I give you many different ways to test because your Internet can be delivered in many different ways.

 

You don't win popularity contests by telling people they have food in their teeth.  But they usually appreciate it in the end.  Trust the results, they have to be affected by something... results are only low here for good reason.   Having said that... you're pulling >100 Mbps consistently, that's pretty sweet.  How much do you pay and what do you expect for that price? .. what are they quoting your speed at?

[Mike Hammett]

Your result at speedtest.net is only 4ms from the server.  TestMy.net is routing you across the Internet to locations where real websites are hosted.  Not on the edge of your providers network.  Of course your faster if you don't really go out to the Internet.  It's like downloading from within your own network.  That's NOT your Internet speed.  We should call that your ultimately ideal situation speed. 

 

The multithread test is designed to be harder.  Just now I ran a normal download test and CPU maxed at 100% (one of my cores).  I ran a multithread test and hit 275% (nearly 3 cores).  But I've found that it's not the same case for Mercury tests (which are also multithread).  A Mercury test vs netflix.com I saw a peak of 110%.  So you can multithread with Mercury and it's not quite as CPU intensive. Always keep in mind that TestMy.net isn't just an Internet test, it's also a test of the client computer and in the case of Mercury it's also a test of the server. I give you many different ways to test because your Internet can be delivered in many different ways.

 

You don't win popularity contests by telling people they have food in their teeth.  But they usually appreciate it in the end.  Trust the results, they have to be affected by something... results are only low here for good reason.   Having said that... you're pulling >100 Mbps consistently, that's pretty sweet.  How much do you pay and what do you expect for that price? .. what are they quoting your speed at?

None of these are on the same network. They are all out on the public Internet. This is from my desk at my client. It's an enterprise-grade connection. Cogent\Zayo SLA backed service. I'm not privy to what they pay, but being SLA backed, if they don't get their full gigabit 24/7 (obviously there's other things using the circuit either at my client or the servers), they're due a credit or can escape the contract.

 

http://www.speedtest.net/my-result/3967940640

http://www.speedtest.net/my-result/3967943227

http://www.speedtest.net/my-result/3967946468

http://www.speedtest.net/my-result/3967947756

http://www.speedtest.net/my-result/3967949382

http://www.speedtest.net/my-result/3967951961

http://www.speedtest.net/my-result/3967953595

http://www.speedtest.net/my-result/3967966426

http://www.speedtest.net/my-result/3967968134

 

Nine different servers on nine different networks that are all less than 10 ms and are not on-net. I could keep going to additional servers, but I don't feel that's necessary to debunk the idea that I'm running it to an internal, on-net server. Some of them are even a three hour drive away.

 

I'll try the Mercury tests and I'll try my laptop which has a lot more CPU later this week. Maybe I can get them to get me a bigger machine. Even though it has 16 GB of RAM, it's getting long in the tooth.

 

I'm not here to trash one methodology or to promote another. I'm just trying to find the test that's the most accurate the most of the time. I do agree on the single thread vs. multithread.

 

Seeing the latency during the test would be nice. I wish there was something like Cisco's IP-SLA that one could use for these. While the test was occurring, there's realtime loss, latency and jitter measurements. Great for finding connections that falter under load.

 

 

I'm actually involved in a few projects to increase throughput and decrease latency in many locations on the Internet.

[jrtcjrjr]

From the Speedtest.net site:

Download Speed

1. Your computer downloads small binary files from the web server to the client, and we measure that download to estimate the connection speed.
2. Based off this result, we choose how much data to download for the real test. Our goal is to pick the right amount of data that you can download in 10 seconds, ensuring we get enough for an accurate result, but not take too long.
3. We prevent caches from throwing off results by appending random strings to each download.
4. Once we start downloading, we use up to four HTTP threads to saturate your connection and get an accurate measurement.
5. Throughput samples are received at up to 30 times per second.
6. These samples are then aggregated into 20 slices (each being 5% of the samples).
7. The fastest 10% and slowest 30% of the slices are then discarded.
8. The remaining slices are averaged together to determine the final result.

Why do we discard certain results? We want to ensure we're giving you the most accurate assessment of your connection's maximum sustained throughput. Here's how we do that:

• Outlying 10%: Since we're measuring data transported over HTTP (via Flash), speed can be affected by a few things, such as potential protocol overhead; buffering due to the many layers between our application and the raw data transfer; or throughput bursting due primarily to CPU usage. To account for these variables, we initially drop the top 10% and bottom 10% of our slices as outliers.
• Test Ramp-Up Period: We keep the default test length short to improve user experience, but in doing so, the ramp-up period can take up a significant portion of the beginning of the test. In consideration of that, we also drop another 20% of the bottom result slices.

-----

Speedtest is being much more direct in explaining how the results are calculated - as compared to only a few months ago. Good for them.

 

My observations from comparing Speedtest.net and testmy.net are first, the results cannot be rationally compared. Testmy.net results are calculated using the complete test while speedtest.net results are form a 60% slice of the total test. 

 

I also see a difference in how Speedtest is (seemingly) implemented by various ISPs. Speedtest.net tests are observed to take from 3 to 8 seconds depending on which ISP hosted test I use. As has been already noted, there is typically a surge in speed during the first few seconds of download. A Speedtest result that is obtained after only a few seconds provides me with a result which averages 150% of the throughput I pay for. That speed in not sustained, however. Any test, testmy.net or Speedtest.net that takes 7 seconds or more shows the result of speed settling back to or below my paid for throughput. 

 

My opinion is that various uses of the Internet impact how we view speed. Using testmy.net I can see the chart illustrating the ramp-up of connection - the subsequent speed burst - the typically sustained throughput - and finally, the tailing off as the download is completed. The early burst in speed is certainly appreciated when simply browsing or checking email. The snappiness provides a warm and fuzzy feeling. When I am streaming video for an extended period then the sustained throughput at the speed I pay for is important.

 

It is no wonder that most ISPs use Speedtest.net as their official throughput test. It usually makes them look good. 

[Mike Hammett]

Not all and I'd say probably most ISPs don't implement bursting.

 

I do agree that a longer speedtest.net test would be helpful.

 

 

Many ISPs discount the credibility of speedtests. I, for one, encourage them to learn what they reveal. Yes, some of the hosts may be crappy, however, the poor tests that are not the fault of a crappy host indicate piss poor performance either at that ISP or beyond that ISP. They say they aren't responsible for what's beyond them. I say they are. They need to get their upstreams to fix the issue or switch upstreams.

[shark91962]

I set up a Twitter account ([email protected]) to prove to AT&T that I was not "getting what I paid for". At first, I took an average of TMN and speedtest.net (always 5+ Mbps ↑). When I enabled multithreading, my speed doubled on TMN to approx 35 Mbps and speedtest.net strangely "capped" at 20. ???