jimharle

So this morning I did the same four-threaded download from Germany, and got slightly better results this time, with the average speeds of each being 307k, 318k, 318k, and 277k, which collectively is around 9Mbps. I should point out that the ISP did "upgrade" us to 30Mbps as part of the initial troubleshooting, and left it there. I did that same download from home (Comcast) and got a bit over 20Mbps collectively. So as far as the CentOS ISO from the UofU mirror site you suggested, which is one of the "known good" endpoints for the ISP we called out to them (latency is just under 7ms), a single download (one instance) averaged 3333k (almost 27Mbps). When I did four at a time, collectively it was a scoach higher (around 28Mbps), with individual instance speed fluctuations between 278k and 1336k. The average speeds of each was 871k, 862k, 1195k, and 1186k. So if "everything else" performed like that, or even reasonably close to that, we wouldn't be complaining. It's obvious that the fiber connection itself is fine with the 30Mbps, yet the ISP continues to act clueless thinking the other problems are related to the router or something we're doing.

So I ran that test again today, with no other significant traffic on the line. There were fluctuations on each download thread, ranging from 50k to as high as 400k, but usually in the 100s somewhere. They did not seem to move in unison, but I was eyeballing it. I let the four threads run their course, and the average dl speeds of the four threads were 160k, 162k, 182k, and 181k. What do the fluctuations indicate? Incidentally, they sent a tech out this morning, who replaced their Cisco router (this is the 2nd time they've done that). I'll give you one guess on whether that made any difference. Edit: I guess I gave it away in the previous paragraph, but it was a rhetorical question anyway. Edit #2: You did imply what the fluctuations meant in your previous post. Would you expect "stability" when being intentionally shaped? I know my home Comcast downloads fluctuate very little. It's sad when $600/month to download 700MB files in over an hour is supposed to be acceptable.

Here is the e-mail we received today from our account manager: We received your complaint from the BBB. I believe they are going to try switching out equipment before they make the determination on whether you will be released from your contract. Someone should be in touch shortly to schedule that. Perhaps I'm reading in too much, but this comes across as "we're annoyed" to me. But honestly, what do they expect?

Here's a screenshot of that Curl test (ftp.hosteurope.de is about 155ms away):

Thank you mudman, I will do that specific test and report back. From the previous tests we've done, when we get multiple threads going like that, they do all progress at "about" the same rate (although still quite volatile). We can almost "max out" our [now 30Mbps] connection, but we have to get *many* threads going, to do that. We've explained to the ISP that would be great and all if we were doing bittorrent downloads all of the time, but like most businesses, that's not how we roll. On a side note, have you looked at Aria2? It is a great multi-threaded TCP download tool. You did bring up a good point, in that you wouldn't necessarily want one user in a "business connection" maxing out the pipe and making it slow for everyone else, which is what gave me pause in the beginning, thinking they were using a bandwidth arbitrator to do something like that. However, I would think that if such a device were in play, the transfers would be "steady" at a restricted rate, rather than erratically jumping all over the place. It would also make sense that such a device would be "smart," and only invoke rate-limiting on individual connections when the entire pipe was reaching a certain utilization threshold. My BBB complaint was "approved" this morning and submitted to them, and they have until July 31st to respond. We also heard from our legal that we are "okay" to stop approving invoices. Last but not least, I searched my LinkedIn connections for someone at the ISP, and came across a fellow who shares a connection with me, and he is a senior network engineer, having been with them for a little over a year, and previously with another ISP for something like 12 years. I reached out to him to have a look at our issue as well, and he did respond and say he's looking into it, and he asked for followup information which I provided. What I'm curious of, is if the real technical people there have their hands tied by poor business decisions, such as oversubscribed peering links or something, and can't really do anything about it quickly. I just can't imagine they'd still be in business if they were playing games like that, but who knows.

We haven't heard anything yet from the ISP after sending the comparison matrix. We are asking our legal team if it is okay to stop approving their invoices, so that if the ISP sues us for breach of contract, they were aware it was coming. Additionally, I filed a complaint with the BBB this morning. One of the questions on that was "would you be willing to speak to the media?" to which I answered yes.

We gathered some download and traceroute data, and submitted it to the ISP's engineering group late yesterday...attached is the main page showing the comparisons.

Thanks for your feedback! I agree with you on the “no contracts” premise; I’m not sure if that is an option, at least here, for “business-class” Internet, even with Comcast. I’m still trying to figure out what “business-class” even means; I believe it has to do with transfer caps, in that there are none, whereas on a “home” connection you might have to pay extra after a monthly threshold has been met. I’ve had Comcast at home for many years, and they used to say the limit was 250GB/month, although they haven’t enforced that for a long while now, whilst they “test” upcharging in other markets. At our office, we’re easily transferring over 1TB a month in backup data, mostly over the weekends. But I’m getting off-topic…the ISP we are using for the office is regional vs. one of the “big boys.” I don’t really have a problem with the “excessive” rate they are charging us, as we’re supporting our local economy, but I do expect to receive what we’re paying for, and we’re just not. I think that if we can’t get them to acknowledge our problems amicably and reasonably, we’re just going to stop paying the bill and see if they push it, and involve the lawyers if we have to... We work for a large company, but our office is the only location in Utah, and we can’t bluff them into thinking we’ve moved (one of their offices is in a suburb a few miles away…they know where we live ). Our company might even consider paying the $10K and just be done with it, but it’s become a “they need to do what is right” ethical thing amongst us, more than the money. Perhaps a complaint with the BBB is another course of action…especially if we can uncover that their other customers are experiencing the same problems. I have asked them (via e-mail) about whether a bandwidth arbitration device was in play, but didn’t get a response; I need to bring that up again during a voice call. I’m sure they’ll deny they are doing any shaping. As for the computers in our office, we probably have around a dozen devices counting desktops/laptops/servers. But all of those are irrelevant to the problem, as all of the formal testing we’ve done has bypassed all of that. The ISP terminates on a Cisco router they provide, and it’s an Ethernet handoff to us, which connects to our Cisco router. The ISP had nothing to do with our internal networking; we’re actually all computer networking professionals to varying degrees…it’s what we do for a living. When we test “for real,” we connect a computer (via Ethernet) to their handoff cable and go to town. But there’s no point in doing that every time now, as we’ve established the problem has nothing to do with CPE. Lastly, we did find something “new” in the last few days, in that ISO downloads from a couple of “localish” URLs are actually almost normal for us in terms of single-file HTTP downloads. Those URLs are http://mirrors.xmission.com/slackware-iso/slackware-14.1-iso/ and http://slackware.cs.utah.edu/pub/slackware/slackware-14.1-iso/. When we download files from either of those, the transfer rate is 20Mbps, and if we download a couple of files simultaneously, we do approach 30Mbps. XMission is an SLC-based ISP, and our ISP seems to have a peering relationship with them (based on a traceroute), and it is close (around 5ms latency for us). The route to the University of Utah network goes through two different providers beyond our ISP, but it is also close (around 9ms). I find this curious, and believe it points to routing/peering issues (with other sites) as being the source of our problems. I sent this info to the ISP before leaving yesterday; I’m curious to hear what they have to say about that, but again that’ll probably happen during a voice call. I’ll certainly keep this thread updated as I can with any pertinent info. Cheers, Jim

Are you saying that your testmy.net results are different based on time of day?

Additionally, are there any devices between the FIOS box and your computer(s) such as another router? For a long time, I used an appliance firewall which sat between the cable modem and the wireless router, as it had a nice filtering function which kept our kids from happening upon the "bad stuff." The firewall had 100Mbps Ethernet physically, but after upgrading our Internet service to 50Mbps, I found that appliance couldn't reliabliy handle traffic faster than around 30Mbps. I removed it from the mix, and things have been fine since. Regardless, routers (including the FIOS device) are computers too, and sometimes need to be rebooted to make them happy again, especially if you're commonly using applications that open many remote connections for long periods of time (like bittorrent).

I have no idea what "normal" is for cable modems, but here are the details from mine (I'm not having any problems...at home that is) Model Name: SB6121 Vendor Name: Motorola Firmware Name: SB_KOMODO-1.0.6.10-SCM00-NOSH Boot Version: PSPU-Boot(25CLK) 1.0.12.18m3 Hardware Version: 5.0 Firmware Build Time: Oct 29 2012 18:07:13

We are paying for a 20Mbps symmetrical “business-class” Internet connection with a local ISP. We have been with the ISP for about a year. In December of last year, we moved our small (six person) office from one SLC suburb to another. Prior to the move, our “last mile” was serviced by a metro fiber provider called Utopia. When we moved, Utopia was not available at the new location, so our ISP switched us to a fiber line provided by XO Communications. “Everything else” allegedly stayed the same – our public IP range, termination point on the ISP’s equipment, etc. Our problem is, ever since the move, our single-threaded downloads using TCP protocols like HTTP and FTP have been unusually slow. The speed of such downloads can vary wildly depending on time of day, but can also vary quite a bit during a download. Generally speaking, the speed of a given download ranges from roughly 100KB/s to 800KB/s (800Kbps to 6.4Mbps). Most of our downloads fall somewhere within that range, meaning that “at best” we’re usually only getting about 1/3 of line speed for a single download. We have also confirmed these results by connecting laptops (configured with one of our static public IPs) directly to the ISP’s router, bypassing our equipment, to rule out any “contention” issues. As far as “what” we are downloading – the source hasn’t mattered, whether it be ISO files from Microsoft or VMware (both use the Akamai CDN), files from a public FTP server in a Los Angeles data center (with a 1Gbps+ Internet connection) we control, files from Linux distribution mirrors, download tests from different locations with testmy.net…everything we try usually falls into the range I specified above. Now, there have been times when Akamai downloads exceed that general range (in the 12Mbps realm), but that happens rarely (and usually outside of business hours), and not “steadily” (the throughput jumps around between 8 and 16Mbps sporadically). When we come in to test at zero-dark-thirty, is usually when we see those “fast” speeds. During business hours, the speed is usually within the range I specified above. And during a long ISO download from Akamai, I can pause the download from my laptop, switch over to a Comcast WiFi hotspot in the area, resume the download, and it will progress much faster (averaging 24Mbps). If I pause again, and switch back to our wired ISP connection and resume, the slowness returns. Doing a traceroute to the Akamai source address from both ISPs shows a similar hop count and latency, with the Comcast one being about 10ms lower. I don’t think 10ms should equate to a 3x to 6x speed difference. One last data point to mention, is that when performing multi-threaded downloads, we can, and do, approach the line speed, although “maxing it out” is still challenging. We use some backup software which performs multiple HTTP transfers from a remote location, and when that software is working on a directory with many files in it, is when we see the most aggregate bandwidth being used. Each individual file’s throughput falls within that same range I specified. We have to be transferring around 10 files simultaneously to start approaching the line speed. So what has our ISP done to address this, in our five months of complaining about it? The first thing they did, was to send out a tech with a laptop, who hooked it up to their router and went straight to speedtest.net, and when it “tested fine” at the full 20Mbps, he told me “it’s working at 20Mbps, I don’t know what to tell you.” So of course I had him go to testmy.net, which painted a completely different picture. He scratched his head with that, and told me he’d report it to his engineering group, but nothing developed from it. They have sent XO technicians out twice to test the fiber transport, sent their own technicians out to do transfer tests, increased our bandwidth to 30Mbps, temporarily increased it to 100Mbps, temporarily switched our public IP range with another, and tested the complete circuit with Fluke devices on both ends. None of these actions have made a difference, other than the higher bandwidth enabling us to collectively use more of it, when transferring many files simultaneously…single downloads remain sucky. I have tried to use the analogy with our account manager (who is non-technical), that we need faster lanes, not a wider freeway. I have also sent them a video illustrating my own test results, where I was switching between them and Comcast for that ISO download. Beyond our account manager acting like she cares, the ISP’s engineering group is continuing to toe the line of “things test fine to our infrastructure, we can’t control the rest of the Internet, we don’t know what to tell you.” It’s maddening. So now you’re probably thinking, why have we put up with this for so long, and not simply dropped them for another ISP? Yes, we would love to do that, as we could get a 100Mbps Comcast business connection for around $220/month. However, a year ago, we signed a three-year contract with this ISP (which I personally was not involved with), paying close to $600/month for a 20Mbps connection. I have read the contract, and the only provisions in there which talk about SLAs/remedies have to do with complete outages, and network latency…there is no language which talks about actual throughput or performance problems. If we were to cancel the contract “without cause,” we would potentially be on the hook for 75% of the remaining contract, so around $10K. I appreciate the time anyone has dedicated to read my long story here, and would like to hear feedback on what YOU would do in this situation, and what ISPs in general can or cannot (generally) get away with. Our contract does specify that the “product” is a 20Mbps connection, but how does one prove/define that, if the ISP’s only responsibility is performance to their own infrastructure? Using that logic, all of their external peering connections could be at dialup speeds, and they could get away with it. Thanks for reading, Jim