Verizon and TM.net results hugely different

[haseric]

We are using Verizon's business service and the difference between their multi-threaded, OOKLA tool's results and TM.net's is outrageous. I'm trying to get my head around why they would be so different. I even used TM.net's multithreaded test just to try and get apples-apples somehow. (The single-threaded TM.net results were 2.9 MB/sec!!).

 

Multithreaded

Verizon: 80-94 mb/sec

TM.net: 14.1 mb/sec

 

Download of file is at about 86-100 KB (!) per sec.

 

What gives here? I have to believe that if Verizon were just out-right lying that someone would have smoked that out of them somehow by now, what with all the transparency floating round the Interwebs these days.

 

Is there any technical explanation for this massive disparity in results?

 

Does their test somehow only test the speed from their central office to the Verizon router? Or would it also include our switch that is downstream from their router?

 

For example, hypothetically, let's say our swtich is bad.  if TM.net is testing the whole line from the download location all the way to my PC, the results would include the faulty switch.

On the other hand, if the Verizon test only tests from their central office (or even from the download site) to their router but doesn't go any further, it might explain the whacky disparity.

 

I'm just spitballing here. Could really use someone with some expertise in this kind of networking to enlighten me.

 

[CA3LE]

There are many reasons why you may see higher results with your ISPs speed test, in this case Verizon's.  Some of which you touched on.

 

First, it's within Verizon's network so you're not really being routed out to the Internet.  TestMy.net is outside of all ISP networks, hosted in the same data centers as millions of other websites.  My servers and the networks that connect them to the Internet are >.  Purpose built for the task, very over powered.  The main server is only running at 10% capacity, even under high stress it rarely goes above 15%.  I replace servers before they regularly run at 20%... my point is that I keep plenty of resources on tap.

 

When testing at TestMy.net it's more likely than not that you'll have a dedicated core for the duration of your test.  No matter how many people are testing resources don't collide.

 

Location can be a factor but the technology behind the test first must be sound.  And... the other one you're using isn't.  It's a well known fact that ookla speed tests don't return all the information.  Portions of the result are clipped right out, they say that it's to offset high CPU and blah blah blah during the test.  Whatever the reason, the algorithm usually makes results look better... TestMy.net calls it how it is, making no adjustments.

 

Sometimes something as simple as TCP tuning can make all the difference.  Looks like you're on Windows 7, TCP Optimizer is a free and easy tool for adjusting that in Windows.  MTU or rwin may be set wrong and that would slow you down.  When that happens it often doesn't show on other speed tests but trust me it affects your browsing and transfer speed.

 

Before you get into that let's run some different tests.  Take your average speed in Mbps and double it, we'll just say 25.  So for the next tests we'll manually select 25 MB from the download test.  Manual selection will give a better comparison between results... less variables are a good thing.  (For our purpose here we ONLY need the download test, save time and skip the upload test.)  Try the mirrors...

 

 

...select the mirrors one at a time then retest.  Testing across multiple routes can help you discover if the problem is beyond your control.  If you get similar results to all US servers the problem is most likely closer to home... possibly within your home network or client machine.  If you get slow results to only a few servers you may have a bottleneck or other issue along the route.

 

Now, see the server with the absolute best results, if this is Dallas look to your second best.  Run a multithread test focusing on that server.

 

Let's remove more variables now that we have a baseline.  Take your router out of the equation, hook directly into your computer.  Once online disable multithread and test again, the same as before.  Improvement will show if the router is to blame.  Check cables and connections too.  Cables can pass signal and still be bad enough to cripple your connection speed so keep that in mind.

 

After you run those tests let me know and I'll look at your results again.  Just make sure you're signed in so all of your results are saved by username.

[haseric]

Thanks for the excellent, detailed response CA3LE. 

 

I'll get to work on these tests.

 

The one thing I don't know that I can do is taking the router out of the equation. (I'm assuming you're talking about the switch we have here?)

 

Here's how we are set up, roughly:

 

Verizon termination point ---------> Cisco Security appliance --------> Linksys 48 port switch -----------> PCs  (via ethernet)

 

The rest of it I think I can do though.

[TriRan]

Yeah usually on fios the router and modem are one in the same

[CA3LE]

check the speed before the switch, if available use a different cat-5 cable than the one that's connecting the Cisco Security appliance to the switch.  Once you confirm that the speeds are the same with or without the switch then you can look deeper.  It sucks when you spend a bunch of time troubleshooting only to find out that the cable leading to the switch or the switch itself is bad.  Keep in mind that CAT-5 cables can appear good when in fact they're passing degraded signal.  For instance, I have a 100' cat-5 patch.  Last time I used it the link speed was 100baseT instead of the normal 1000baseT that it was usually capable of.  Maybe a kink, a weird twist or something ruined it... it happens.  And I've seen it happen with cables that were never moved or unplugged so it's not only cables that are put under abuse like my patch cable.

[mudmanc4]

I'll take a stab in the dark and say there could be some filtering / QOS within the policing policy of the security appliance. Or more likely this is a class maps issue.

 

When you take the ISP's test, chances are the streams are coming from the internal network, consisting even of various nodes within their network,  not a very good gauge for real word throughput. Where testing the same way at testmy.net , you are pulling an aggregate from any variable you set before the test.

 

The simultaneous packet/per flow if set determines this factor. A good setting to test might be "session packet 45"

 

Depending on the model, you may also want to look at 'service-policy police' within the class maps. And see if there are a pile of dropped packets, before making many changes to the config.

[haseric]

Hi again,

 

Okay so I ran the tests using each mirror site as a default and selecting 25MB as the download file.

 

Here are the results:

• Central US >> Dallas, TX USA
  • 1.2 Mbps
• East Coast US >> Washington, D.C.
  • 4.9 Mbps
• North West Coast US >> Seattle, WA USA
  • 2.3 Mbps
• South West Coast US >> San Jose, CA USA
  • 1.7 Mbps
• Northern Europe >> Amsterdam, NL
  • 2.3 Mbps
• South Asia >> Singapore, SG
  • 1.5 Mbps

Based on CA3LE's original reply, this would seem to suggest the problem is "closer to home".  I'm going to see if I can jack the FIOS Cat 5 right into a notebook I can bring in and then will try again with a new Cat 5 to see if that produces new and improved results.

 

It's interesting that a global multithread test about two hours prior to the above testing today showed ~16 MB/sec results.

 

Any other input would be welcome.

I'll take a stab in the dark and say there could be some filtering / QOS within the policing policy of the security appliance. Or more likely this is a class maps issue.

 

When you take the ISP's test, chances are the streams are coming from the internal network, consisting even of various nodes within their network,  not a very good gauge for real word throughput. Where testing the same way at testmy.net , you are pulling an aggregate from any variable you set before the test.

 

The simultaneous packet/per flow if set determines this factor. A good setting to test might be "session packet 45"

 

Depending on the model, you may also want to look at 'service-policy police' within the class maps. And see if there are a pile of dropped packets, before making many changes to the config.

 

Thanks mudmanc4!

 

Unfortunately, the stuff I marked in red bounced directly off my forehead and landed on the floor.

 

I.e., there are terms in there with which this surface-geek-turned-marketing-guy is not familiar.

 

But I'll send it over to our CTO and see if it rings any bells for him (he holds the IT manager job "from above" if you will) 

[haseric]

Looks like we are getting a new switch early next week. Will see if that solves the problem. Will post an update once it's all installed.