CA3LE

Why Do My Results Differ From Speedtest.net / Ookla Speed Tests?

61 posts in this topic

10/4, I'll do that Post , maybe we can Make It Faster! Thanks, Mud.. :)

There are a few people here that know DSL much better then myself , I hope they chime in for you on your thread.

Share this post


Link to post
Share on other sites

Very good discussion here, thanks to all. Further, thanks CA3LE for fixing the "Details" button so that the copy to clipboard works correctly once again; kudos to you for all your hard work! :) I've been missing for a while, had a HD drive crash on Saturday during Memorial Day weekend. Finally back up & running as of this past Friday afternoon. Glad I'd done a recent full drive backup just 2 days prior to the crash! Replaced the dead drive w/a solid state, what a difference in boot & application load times. OK, my 2 cents for a while, thanks again for providing us this great (and most accurate on the net to my knowledge) speed test website. Very best regards, jerry aka "The Old Marine in Texas". :)

iceb and CA3LE like this

Share this post


Link to post
Share on other sites

Vaniety speed tests . ? and Tdawnaz's does my bum look big in these jeans ( never IMHO )

I test here for the reasons stated , maybe the results don't flater my ego , but over 3000 miles

are correct indeed ,

Share this post


Link to post
Share on other sites

I'm with Centurylink, I pay for 1.5/256 service, my downstream rate is 1760Kbps and I always get 1.9 Mbps or better on this website. Speedtest.net is usually right around 1.5 or slightly less. Its hard to believe I receive more than I am able to get..

jb847 likes this

Share this post


Link to post
Share on other sites

CA3LE. And you should also mention that most of the OOKLA based test sites' speed tests won't run on an iOS device because the tests require Flash (the memory and battery hog technology that's not allowed on iOS devices). So kudos for you having iOS compatible testing. And although Speedtest.org now supports iOS/Safari, the results are way off. They even have a disclaimer:

"Warning

For best testing experience and accuracy, Broadband Speed Checker requires at least version 8 of Flash. Please update your client by clicking here."

And when you try the support Contact Us on Speedtest.org, you get:

"Not Found. The requested URL /contact/ was not found on this server."

Plus you can't choose a server like you can on TestMy.

There is an OOKLA Speedtest app for iOS that allows you to pick servers, and the results trend kind of like the results I get here on TestMy, but the UI and the results displays and overall functionality of the app is orders of magnitude less than this site.

For iOS device users, nobody else that I've found is even in the same universe as TestMy.

Edited by Pgoodwin1
CA3LE, jb847 and TriRan like this

Share this post


Link to post
Share on other sites

When running an older system, or one that is overloaded with software, it's also important that Flash consumes CPU power to draw the fancy animations, which might become the limiting factor. Sure, the semitransparent 'speedometer' might give a better impression to a client, but for precise results factors other than the network connection should have as little impact as possible.

Also, the Ookla test isn't very clear about how many connections it creates, whether it uses TCP or UDP. Being multi-threaded, the test fails to show the effect of adjusted TCP windows (with TCPOptimizer). Until recently I didn't think that Optimizer did anything at all. But when looking at a single connection, the difference is clearly there.

I am using the Opera Browser for everything, including these speed tests I did earlier at TestMy. Sadly it's now discontinued. It can be configured to show the averaged speed of every regular non-flash, non-ajax upload and download on the status bar (such as when uploading attachments to webmail or a file host), allowing to observe the state of the Internet without doing an explicit speed test.

Another good method of measuring speeds is to use an FTP server/client. They often have a settings for the socket buffer to test different values without rebooting, as would be necessary when usign TCP Optimizer.

Finely tuned TCP works very well. The recent swithover to UDP for file transfers is a scam.

1911824057.png

Share this post


Link to post
Share on other sites

I understand now that there is a big difference in what I will get with testmy.net and speedtest.net.

 

I did a test every 5 minutes for 1 hour and my average download speed was a pitiful 12.8 Mbps and I am doing it on a laptop that is connected to the router with an ethernet cable  so it takes wifi out of the equation

 

It also looks to me like my connection is very unstable. I have a link to my test here: http://testmy.net/quickstats/jmk

 

Cox cable is supposed to be giving me up to 50 Mbps so is this based on using speedtest.net (Which is what I have always used) or should I be getting close to 50 Mbps relying on testmy.net?

Share this post


Link to post
Share on other sites

I might have already replied to this topic a while ago, so if I did, forgive me...I just didn't check whether I had or not.

 

I'd just like to give a quick summary of my view of TMN vs.  "the others." I am a networking guy by profession as well as by degree and certifications, so it is natural for me to be the "curious cat" about everything networking and to try to "fool the system" (e.g. find bugs that cause erroneous results) as well as attempt to prove or disprove the validity of someone's claim (in this case, the accuracy of TMN).

 

Many (probably most) people do not realize that there have been TCP (and other transports) benchmarking for just about as long as the transport itself has been around.  Some of the most powerful are command line tools found (typically) in Linux systems that offer extreme flexibility in testing (e.g. packet sizes, compression algorithms, hardware offload for things like checksums and VLAN or QOS tagging, certain kernel path bypass mechanisms, window scaling heuristics, and literally dozens of other adjustable parameters to test the maximum Tx/Rx speed between two or more computers).  That said, I've used pretty much all of them at one point or another and have done very meticulous comparisons to the results on TMN.  The margin of error is astonishingly low (generally less than 5%).  Compared to Ookla's Flash based test, this is a factor of 10 difference in accuracy because the average from that site that I have found is around 50% (with a huge standard deviation).  One day, I will perform the tests again and post the results in a forum here..I didn't save everything last time and want to make my post "legitimate" by including all methods and screenshots utilized.  I'll try to get around to it sometime soon.

 

That was the first thing to get out of the way.  Secondly, as explained in other posts, Flash is a VERY expensive technology (expensive meaning processor and memory intensive) and adds quite a bit of latency as well due to the complex paths the data flow must go through.  Is it appealing to the eye?  Absolutely.  Would I choose the most graphically appealing test if my goal is to get the most accurate results?  Absolutely not.  See paragraph above...command line is about as ugly as you can get - but also as accurate as you can get (using the correct tools).

 

I'll preface point three by saying that I don't know enough about the internals of either TMN or Ookla to make a totally accurate claim about their inner workings but I may through my obversations, I can assure you that Ookla has major problems with regard to the results.  One of the best ways to test this on your own is to use a program called Wireshark and start a capture of your network packets (make sure to choose the correct network card!!) during a TMN test and an Ookla test.  The first thing to notice is the amount of data that is transferred during a test.  I cannot figure out, for the life of me, the algorithm with which Ookla determines how much actual data to transfer.  By "data," I mean, for example, how many megabytes are transferred to your box during the test.  Second thing I noticed was a lot of "noise" in the packets that seemed to be upstream communication to the Ookla host server from my computer during a download test (NOT ACKS, so please don't call me out saying it was ACKS).  There is some type of communication to the server going on - which OBVIOUSLY interferes with an accurate download score if a Download test is also simultaneously transmitting information other than standard TCP Acknowledgments, replies, etc.  That doesn't occur on TMN.  There is simply an ACK and SYN as expected during a raw transfer.  The TMN server determines whether or not more data is needed to determine an ACCURATE result based on how quickly you down- or uploaded the information (7 seconds down and 5 seconds up for a specific amount of data transfer).  Assuming a download, at first you will receive the smallest continuous piece of data and if it took less than 7 seconds to transfer, TMN will push the next size to you.  This process repeats until:

 

 

  1. The seven seconds expire and you have not received the entire download chunk - or -
  2. You reach the maximum size (200MB) and complete the download in less than seven seconds.

This way, during the download, the only cost incurred on your PC is that of the Operating System's networking routines and the CPU usage for offloaded tasks (for instance, checksum offload).  BTW, this occurs during ANY network communication and there is no way around it.

 

Another thing to consider if you are receiving results that are inconsistent amongst testing sites is the location of the hosting server.  If you go to speedtest and live in Atlanta, Speedtest will choose the location closest to you with the least latency (and, in this case, would be in Atlanta).  The further a byte has to travel, the more latency introduced and (generally) more hops must be taken to reach the destination.  All of which introduces decreases in speed with increase in hops and latency.  So, if you're testing on Speedtest in Atlanta on an Atlanta based server and then hop over to TMN and use a Dallas server, it is only natural to expect that the transfer speed will (again, typically) be slower and vice-versa.  So, a more accurate way to compare the sites would be to choose a Dallas location on speedtest, take the test and then test via Dallas on TMN.  Or, you can just trust me..TMN is better :thumbsup:

 

The last point I'll make in this post is that with TMN, the data transfer occurs via standard HTTP, which is how the vast majority of your real world downloads and browsing occurs.  One exception is on a secure site that uses SSL and is preceded with "https://" - that normally occurs on port 443 instead of 80 as in HTTP and incurs a heavy performance penalty for the encryption and decryption of the data after is is received.  There are tons of other protocols such as FTP, SCP, SSH, CIFS, SMB, NFS, etc. but, like I said, 99% of the typical user's internet browsing occurs on HTTP.  I don't know exactly how the data is transmitted and received on Ookla based sites but I do not believe it is HTTP - I think it is an embedded part of the Flash wrapper.

 

So, to close this post that I meant to be short and to the point and went way overboard, my opinion and experience is that the most accurate measurement of your bandwidth is going to be found on TMN

 

I apologize for the rambling.  I hope at least someone finds this helpful!  Take Care....more to come (in the future, sometime!)

 

--SIETEC--

 

ImEcl4b.OB2LEUl.png

Edited by sietec
KAGarver, djpenn3 and CA3LE like this

Share this post


Link to post
Share on other sites

Thanks for clarifying speeddtest results. I came hear to see why I am having such losy down load and upload speeds. 

Before I contact the tech idiots at cox..I though it was a good idea I check out results from another location and service.

 

After rebooting the modem and my router It seems things are a little better. I hate paying for something I will never get. I also hate being lied too.

 

Thanks for the info this helps my understanding of what I am testing.

 

I'm new and glad there are others with the same mind set!

 

dremi 

Share this post


Link to post
Share on other sites

I have Suddenlink and pay for 30\2 MBps.  I am hopeful that the following provides a vivid representation of just one (of the several) reasons for wildly different test sppeds.

 

I just finished testing my conection with OOKLA.  I am in Lake Havasu City AZ and (based on ping? 34ms) OOKLA chose a server in Mohave, California (roughly 180 miles west as the crow flies).  Multiple tests (3) and the average I tested at was 27.86 MBPS. 

 

It took about 30 seconds to change the host server to Austin, Texas.  Same ping (34ms) but the 3 test average from the Austin server was 10.16MBps.

 

Before you start blaming the messenger here, it might be an eye opener for you to take a look at a little of your own testing with some of the variables you will see in this forum repeatedly (and I do mean REPEATEDLY). 

 

From the way I see it, it is entirely up to the user what they want.... unbiased information or perhaps being able to brag about how fast some of the speed tests out there can blow smoke up their.. :tongue8:.. well you know what I mean.

 

 

Thanks for letting me put in my few bits.

Share this post


Link to post
Share on other sites

I have Suddenlink and pay for 30\2 MBps.  I am hopeful that the following provides a vivid representation of just one (of the several) reasons for wildly different test sppeds.

 

It took about 30 seconds to change the host server to Austin, Texas.  Same ping (34ms) but the 3 test average from the Austin server was 10.16MBps.

 

Have you tried TCP Optimizer?  I think it may help in your situation.  You're running "Windows NT 6.0" so that's vista or server 2008.  TCP optimization is especially helpful (necessary even) for older version of windows.  When I ran windows prior to windows 8.1 the first thing I did with a fresh windows install was optimize TCP and MTU.  Without this optimization your normal downloads will suffer as well as your classic linear download test results.  Results from other speed tests will often show much faster speed... they're misleading you.  Other tests want to give you the best chance possible to score high... TMN on the other hand wants to help you ACTUALLY score as high as possible.  The truth hurts sometimes, it helps all the time.

 

You may have an older computer with other limitations or issues so it may or may not help.  It can't hurt to try.

 

Remember... TestMy.net is a full, unbiased picture.  Results can be harsh, but it's the truth.  For people that can't pull decent speed here I say...

 

o9wCJOes.png

As long as the provider and equipment are delivering I'm able to max out every connection I touch.  So should you.  I'm not doing anything special here... great provider, great networking, great computer...  doesn't matter if I'm in OS X, Linux or win 8.1 -- Firefox, Safari, Chrome, IE -- all the same, I run all default settings and consistently test this fast. 

 

If you can raise your score on TMN you're Internet (and computer as a whole) will be noticeably faster.   period.

 

I wish everyone around the world could experience a minimum of my connection speed.  The world will change greatly when 100 Mbps is considered as slow as dial-up is today.

Share this post


Link to post
Share on other sites

I did the tpcoptimizer.. before it i was getting 9.1 to 10.. on here.. after it was closer to what i am paying for 74.6. I tested againks my cable companies Ookla which gave 74.2 direct .. befoire the 9.1 here 2 minutes difference .. Optimizer improved 100% my speed .. thanks

LadyQ

 

http://testmy.net/compID/625470500507

 

post-2-0-99226600-1424484696_thumb.png

Edited by CA3LE
added screenshot and stats link
CA3LE likes this

Share this post


Link to post
Share on other sites

Based on my experience using the Ookla Speedtest mobile app, the Android app can't be trusted!

 

Have a look at the following two tests run on my mobile (OnePlus One):

 

1280401151.pngeMDz8l0.zIDPWh3i.png

 

The following is the router's DSL status page which I ran both tests with:

 

post-248268-0-71146700-1431534844.png

 

Notice the overinflated uplink result in the Speedtest app result vs. my DSL line speed. :huh:

 

On my fixed wireless ISP connection, it underrates the downlink:

 

1280408407.pngWgX1B3Q.4CzJUen6.png

 

As I don't have access to a status page showing what my Wireless ISP is limited at, I ran an FTP test right after the following tests completed:

 

post-248268-0-50435900-1431535542.png

 

3.4MB = 27.2Mbps, so it's obvious which result is more accurate. :wink:

 

These are the results form a YouTube video I created showing these tests in action.  Usually the Speedtest.net app reports about 50% above my actual uplink for any connection, but in this case it was out by over 3 times the link speed!  Plus catching it in a screen recording made was even more exciting as I think I only ever seen it go that high once before. :cool:

 

sourcejedi and CA3LE like this

Share this post


Link to post
Share on other sites

Based on my experience using the Ookla Speedtest mobile app, the Android app can't be trusted!

 

Have a look...

 

Love it!  Thanks for the post!

Share this post


Link to post
Share on other sites

on the mercury test, using the tumbler site, i reached my service I pay for, 250 Mbps (30 MBps)  maybe the server is just closeby or something.

Share this post


Link to post
Share on other sites

Two more things to add:

Testing very slow connections (especially uplink):

Based on my experience, Ookla's browser based Speedtest can't seem to handle connections below about 0.1Mbps, at least on the uplink.  My fixed wireless ISP's connection has had an issue for over a month now where the uplink struggles to perform better than 0.1Mbps, then intermittently spikes to about 9Mbps as shown below:

TestMy_erratic_uplink_results.thumb.png.

While Ookla's browser based Speedtest seems to measure the link during the brief periods the uplink is running quick, I have yet to see it record a single speed test with a result below 0.1Mbps.  When the uplink is running slow, this is what happens when I run the Ookla's Speedtest:

Speedtest_net_jammed_uplink.thumb.png.e1

That's as far as it gets, even when I check an hour later - It stalls like trying to drive a manual gear stick car below 5MPH without pressing the clutch. :huh:

I'm not sure if this ISP has access to the test results since it hosts an Ookla server, but if it does, they'll obviously be skewed as the only test results it would have logged from my end were the ones the test successfully completed, i.e. during the brief periods my uplink worked.  Then again, this ISP also uses a carrier grade NAT, so if it did have access to the logs with IP addresses, it probably couldn't map them with each logged IP address being shared amongst many customers.

Single threaded tests:

Another problem I noticed with Ookla's app is that it does not provide the ability to perform a single threaded speed test.  While multi-threaded tests are great at determining the maximum bandwidth available on an ISP's connection, it does not show what one will get when streaming video or even downloading a large individual file.  This potentially also explains why some users who get fast results on Speedtest have problems with video on demand services repetitively buffering or showing a grainy low resolution stream.

While on a business trip in Germany, I stayed at a Leonardo hotel which had plenty of advertising about having free Wi-Fi.  Going by Ookla's Speedtest, it looked reasonably good with a 2Mb download test result, yet when I tried watching YouTube, it would only play in 240p with a grainy picture even though I know 2Mbps is plenty to play videos at 480p and some even at 720p. 

When I tested with TestMy, sure enough it reported my speed as about 500kbps consistently, which explains the YouTube playback problem.  When I started a download in my web browser, it came in at a steady 60KB/s.  However, when I started running simultaneous downloads, each single one continued to run at about 60KB/s.  Sure enough, when I ran TestMy in multi-threaded mode, I got over 2Mbps as the test result.

I saw a flyer on the table mentioning about high speed Wi-Fi, so was curious to see what it said after my strange Wi-Fi experience.  It mentioned that the complementary Wi-Fi is suitable for general web browsing and e-mail only and in order to stream video, download large files and so on, I would need to purchase premium Wi-Fi access for €8 per hour. :o

So what I reckon they've done is throttle individual connections to 500kbps, which is just fast enough to comfortably browse the web as web browsers make multi-threaded connections to download all the web page elements, pictures, etc.  However, as video streaming services only establish a single connection to run the stream, 500kbps is not enough for most streaming services as many require at least 1Mbps to stream at all, yet Ookla's Speedtest will falsely give the impression that the connection is well capable of streaming video when throttling occurs on individual connections.

Some mobile/cellular and fixed wireless services also seem to throttle individual connections, which effectively reduces network load as video streaming services generally stream at the maximum resolution possible over a single connection.  For example, a 4G provider could throttle individual connections at 4Mbps would prevent most streaming services being able to play full HD, yet if one uses the popular Speedtest app which performs a multi-threaded test, it will report a vastly higher test result.

CA3LE likes this

Share this post


Link to post
Share on other sites

I had an interesting call with my work colleague in the UK who got in Pulse8's Fibreoptic 38Mbps Broadband package, which runs over VDSL. 

 

I asked him what speed he's getting and sent him the TestMy combo speed test link.  He came back saying he's getting 4.8Mbps down and 7.5Mbps up.  So I called his VoIP phone and asked him to accept a remote desktop connection so I could try some further tests.

 

I then tried several download speed tests with both the UK and German servers.  It seemed like each time I ran a test, it gradually got slower and slower, hitting 3.8Mbps after 3 x 50MB tests with both the UK and German servers.  So I then checked what Speedtest.net gives and it gave 16Mbps on the download and 8Mbps on the upload. 

 

He was quite surprised and wondered why there such a drastic difference.  As I know Speedtest uses multiple threads, I decided to do a 50MB multi-threaded test on TestMy using the UK server.  As soon as the test started, my remote desktop picture went to a crawl where it started painting chunks of pixels at a time and my colleague could only hear the odd word I spoke.

 

About 2 minutes later, the test completed along with my remote desktop screen and VoIP quality returning back to normal.  The test result showed 3.5Mbps. 

 

My work colleague told me that Pulse8 said to use their website's speed test to check if there is any speed issue.  So I decided to check that one.  Like Speedtest.net, it also reported about 16Mbps down and 8Mbps up and sure enough it showed 'OOKLA' at the top-right corner, which clearly is the same engine as Ookla's Speedtest.net site. 

 

Unlike TestMy's test, Ookla's tests did not cause my remote desktop screen or VoIP call to break up.  So it's pretty clear his ISP (or Tiscali it operates over) has traffic shaping to give plenty of bandwidth to Ookla's servers, but only 3.5Mbps to 4.8Mbps overall capacity to the Internet, at least around 4pm to 5pm when I was testing his connection.

CA3LE likes this

Share this post


Link to post
Share on other sites
7 minutes ago, Sean said:

As soon as the test started, my remote desktop picture went to a crawl where it started painting chunks of pixels at a time and my colleague could only hear the odd word I spoke.

 

Unlike TestMy's test, Ookla's tests did not cause my remote desktop screen or VoIP call to break up.

 

Proof during the TMN test that you were topping out the connection. 

 

Funny the ookla tests read higher yet didn't cause your voip and rd streams to die out.  The only way in my mind that happens is if there's funny business or it's just plain BS.

Sean likes this

Share this post


Link to post
Share on other sites

The one and only reason Ookla is even remotely available on the net today, is the simple fact first of all, they have the ISP insiders market flooded.

 

Who else knowing, would harbor a flash base script within their internal network, unless the ends outweighed the means, or likelihood of an infiltration. If your confident enough to allow this thing to live within your public network, than obviously you've got said network protected. Fair enough right? That type of ISP network protection would likely come at the cost of the consumer. If for no other reason than isolation, which is the complete antithesis of the meaning to testing throughput across networks, that at the very least represents what the intentions are for the masses taking the test.

 

Secondly the ISP know well the entire idea of a test being run on any high level network, is useless and nothing more than a 'feel good' item.

 

These tests of Ookla, should be completely isolated on a per ISP level, if they must be used at all. Of course they are useful for tech to determine if there is or not an issue between modem/ head end (or wherever these flash tests are living). No?

Pgoodwin1 likes this

Share this post


Link to post
Share on other sites

There is actually a browser-independent tool that tests the Internet speed using Ookla's servers. It's a Python script. (https://github.com/sivel/speedtest-cli)

 

I think there might be some difference in results since testing this way doesn't involve plugins like Flash or Java, nor web page rendering. I am not sure how much this affects the results, though. I wonder if TMN offers any API that allows scripts like this to be written. Scripts also allow testing rented servers remotely using SSH (where it's impossible to test via a browser), and I'd really like to see TMN offering this.

Share this post


Link to post
Share on other sites

I just created an account for saying this... TestMy results seem to be wrong. In a download speed test, it took so long to download a 42 MB test file... and it showed me a speed of 1.9 MBPS. I have a 5 MBPS connection. And ookla speed test is spot on.

To verify this I downloaded something else (and it was of size 42 MB too) and I got correct speeds there, and it took much shorter time to download.

 

What does this prove?

Share this post


Link to post
Share on other sites
3 hours ago, TestMySeemsWrong said:

I just created an account for saying this... TestMy results seem to be wrong. In a download speed test, it took so long to download a 42 MB test file... and it showed me a speed of 1.9 MBPS. I have a 5 MBPS connection. And ookla speed test is spot on.

To verify this I downloaded something else (and it was of size 42 MB too) and I got correct speeds there, and it took much shorter time to download.

 

What does this prove?

What makes you think the ookla speed test is giving you the truth? Just because your ISP says you get "up to 5Mbps", that doesn't mean your throughput from the Internet is at that plan max.

 

go read all the tabs here. It will help you understand that many ookla based speed tests were picked by ISPs just because they were created to show a high speed

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now