Jump to content

KAGarver

Members
  • Content Count

    3
  • Joined

  • Last visited

  • Speed Test

    My Results

Reputation Activity

  1. Like
    KAGarver reacted to sietec in Why Do My Results Differ From Speedtest.net / Ookla Speed Tests?   
    I might have already replied to this topic a while ago, so if I did, forgive me...I just didn't check whether I had or not.
     
    I'd just like to give a quick summary of my view of TMN vs.  "the others." I am a networking guy by profession as well as by degree and certifications, so it is natural for me to be the "curious cat" about everything networking and to try to "fool the system" (e.g. find bugs that cause erroneous results) as well as attempt to prove or disprove the validity of someone's claim (in this case, the accuracy of TMN).
     
    Many (probably most) people do not realize that there have been TCP (and other transports) benchmarking for just about as long as the transport itself has been around.  Some of the most powerful are command line tools found (typically) in Linux systems that offer extreme flexibility in testing (e.g. packet sizes, compression algorithms, hardware offload for things like checksums and VLAN or QOS tagging, certain kernel path bypass mechanisms, window scaling heuristics, and literally dozens of other adjustable parameters to test the maximum Tx/Rx speed between two or more computers).  That said, I've used pretty much all of them at one point or another and have done very meticulous comparisons to the results on TMN.  The margin of error is astonishingly low (generally less than 5%).  Compared to Ookla's Flash based test, this is a factor of 10 difference in accuracy because the average from that site that I have found is around 50% (with a huge standard deviation).  One day, I will perform the tests again and post the results in a forum here..I didn't save everything last time and want to make my post "legitimate" by including all methods and screenshots utilized.  I'll try to get around to it sometime soon.
     
    That was the first thing to get out of the way.  Secondly, as explained in other posts, Flash is a VERY expensive technology (expensive meaning processor and memory intensive) and adds quite a bit of latency as well due to the complex paths the data flow must go through.  Is it appealing to the eye?  Absolutely.  Would I choose the most graphically appealing test if my goal is to get the most accurate results?  Absolutely not.  See paragraph above...command line is about as ugly as you can get - but also as accurate as you can get (using the correct tools).
     
    I'll preface point three by saying that I don't know enough about the internals of either TMN or Ookla to make a totally accurate claim about their inner workings but I may through my obversations, I can assure you that Ookla has major problems with regard to the results.  One of the best ways to test this on your own is to use a program called Wireshark and start a capture of your network packets (make sure to choose the correct network card!!) during a TMN test and an Ookla test.  The first thing to notice is the amount of data that is transferred during a test.  I cannot figure out, for the life of me, the algorithm with which Ookla determines how much actual data to transfer.  By "data," I mean, for example, how many megabytes are transferred to your box during the test.  Second thing I noticed was a lot of "noise" in the packets that seemed to be upstream communication to the Ookla host server from my computer during a download test (NOT ACKS, so please don't call me out saying it was ACKS).  There is some type of communication to the server going on - which OBVIOUSLY interferes with an accurate download score if a Download test is also simultaneously transmitting information other than standard TCP Acknowledgments, replies, etc.  That doesn't occur on TMN.  There is simply an ACK and SYN as expected during a raw transfer.  The TMN server determines whether or not more data is needed to determine an ACCURATE result based on how quickly you down- or uploaded the information (7 seconds down and 5 seconds up for a specific amount of data transfer).  Assuming a download, at first you will receive the smallest continuous piece of data and if it took less than 7 seconds to transfer, TMN will push the next size to you.  This process repeats until:
     
     
    The seven seconds expire and you have not received the entire download chunk - or - You reach the maximum size (200MB) and complete the download in less than seven seconds.
    This way, during the download, the only cost incurred on your PC is that of the Operating System's networking routines and the CPU usage for offloaded tasks (for instance, checksum offload).  BTW, this occurs during ANY network communication and there is no way around it.
     
    Another thing to consider if you are receiving results that are inconsistent amongst testing sites is the location of the hosting server.  If you go to speedtest and live in Atlanta, Speedtest will choose the location closest to you with the least latency (and, in this case, would be in Atlanta).  The further a byte has to travel, the more latency introduced and (generally) more hops must be taken to reach the destination.  All of which introduces decreases in speed with increase in hops and latency.  So, if you're testing on Speedtest in Atlanta on an Atlanta based server and then hop over to TMN and use a Dallas server, it is only natural to expect that the transfer speed will (again, typically) be slower and vice-versa.  So, a more accurate way to compare the sites would be to choose a Dallas location on speedtest, take the test and then test via Dallas on TMN.  Or, you can just trust me..TMN is better
     
    The last point I'll make in this post is that with TMN, the data transfer occurs via standard HTTP, which is how the vast majority of your real world downloads and browsing occurs.  One exception is on a secure site that uses SSL and is preceded with "https://" - that normally occurs on port 443 instead of 80 as in HTTP and incurs a heavy performance penalty for the encryption and decryption of the data after is is received.  There are tons of other protocols such as FTP, SCP, SSH, CIFS, SMB, NFS, etc. but, like I said, 99% of the typical user's internet browsing occurs on HTTP.  I don't know exactly how the data is transmitted and received on Ookla based sites but I do not believe it is HTTP - I think it is an embedded part of the Flash wrapper.
     
    So, to close this post that I meant to be short and to the point and went way overboard, my opinion and experience is that the most accurate measurement of your bandwidth is going to be found on TMN. 
     
    I apologize for the rambling.  I hope at least someone finds this helpful!  Take Care....more to come (in the future, sometime!)
     
    --SIETEC--
     

×
×
  • Create New...