What are an ISP's responsibilities, anyway?

[jimharle]

We are paying for a 20Mbps symmetrical “business-class” Internet connection with a local ISP.  We have been with the ISP for about a year.  In December of last year, we moved our small (six person) office from one SLC suburb to another.  Prior to the move, our “last mile” was serviced by a metro fiber provider called Utopia.  When we moved, Utopia was not available at the new location, so our ISP switched us to a fiber line provided by XO Communications.  “Everything else” allegedly stayed the same – our public IP range, termination point on the ISP’s equipment, etc.

 

Our problem is, ever since the move, our single-threaded downloads using TCP protocols like HTTP and FTP have been unusually slow.  The speed of such downloads can vary wildly depending on time of day, but can also vary quite a bit during a download.  Generally speaking, the speed of a given download ranges from roughly 100KB/s to 800KB/s (800Kbps to 6.4Mbps).  Most of our downloads fall somewhere within that range, meaning that “at best” we’re usually only getting about 1/3 of line speed for a single download.  We have also confirmed these results by connecting laptops (configured with one of our static public IPs) directly to the ISP’s router, bypassing our equipment, to rule out any “contention” issues.  As far as “what” we are downloading – the source hasn’t mattered, whether it be ISO files from Microsoft or VMware (both use the Akamai CDN), files from a public FTP server in a Los Angeles data center (with a 1Gbps+ Internet connection) we control, files from Linux distribution mirrors, download tests from different locations with testmy.net…everything we try usually falls into the range I specified above.

 

Now, there have been times when Akamai downloads exceed that general range (in the 12Mbps realm), but that happens rarely (and usually outside of business hours), and not “steadily” (the throughput jumps around between 8 and 16Mbps sporadically).  When we come in to test at zero-dark-thirty, is usually when we see those “fast” speeds.  During business hours, the speed is usually within the range I specified above.  And during a long ISO download from Akamai, I can pause the download from my laptop, switch over to a Comcast WiFi hotspot in the area, resume the download, and it will progress much faster (averaging 24Mbps).  If I pause again, and switch back to our wired ISP connection and resume, the slowness returns.  Doing a traceroute to the Akamai source address from both ISPs shows a similar hop count and latency, with the Comcast one being about 10ms lower.  I don’t think 10ms should equate to a 3x to 6x speed difference.

 

One last data point to mention, is that when performing multi-threaded downloads, we can, and do, approach the line speed, although “maxing it out” is still challenging.  We use some backup software which performs multiple HTTP transfers from a remote location, and when that software is working on a directory with many files in it, is when we see the most aggregate bandwidth being used.  Each individual file’s throughput falls within that same range I specified.  We have to be transferring around 10 files simultaneously to start approaching the line speed.

 

So what has our ISP done to address this, in our five months of complaining about it?  The first thing they did, was to send out a tech with a laptop, who hooked it up to their router and went straight to speedtest.net, and when it “tested fine” at the full 20Mbps, he told me “it’s working at 20Mbps, I don’t know what to tell you.”  So of course I had him go to testmy.net, which painted a completely different picture.  He scratched his head with that, and told me he’d report it to his engineering group, but nothing developed from it.  They have sent XO technicians out twice to test the fiber transport, sent their own technicians out to do transfer tests, increased our bandwidth to 30Mbps, temporarily increased it to 100Mbps, temporarily switched our public IP range with another, and tested the complete circuit with Fluke devices on both ends.  None of these actions have made a difference, other than the higher bandwidth enabling us to collectively use more of it, when transferring many files simultaneously…single downloads remain sucky.  I have tried to use the analogy with our account manager (who is non-technical), that we need faster lanes, not a wider freeway.  I have also sent them a video illustrating my own test results, where I was switching between them and Comcast for that ISO download.  Beyond our account manager acting like she cares, the ISP’s engineering group is continuing to toe the line of “things test fine to our infrastructure, we can’t control the rest of the Internet, we don’t know what to tell you.”  It’s maddening.

 

So now you’re probably thinking, why have we put up with this for so long, and not simply dropped them for another ISP?  Yes, we would love to do that, as we could get a 100Mbps Comcast business connection for around $220/month.  However, a year ago, we signed a three-year contract with this ISP (which I personally was not involved with), paying close to $600/month for a 20Mbps connection.  I have read the contract, and the only provisions in there which talk about SLAs/remedies have to do with complete outages, and network latency…there is no language which talks about actual throughput or performance problems.  If we were to cancel the contract “without cause,” we would potentially be on the hook for 75% of the remaining contract, so around $10K.

 

I appreciate the time anyone has dedicated to read my long story here, and would like to hear feedback on what YOU would do in this situation, and what ISPs in general can or cannot (generally) get away with.  Our contract does specify that the “product” is a 20Mbps connection, but how does one prove/define that, if the ISP’s only responsibility is performance to their own infrastructure? Using that logic, all of their external peering connections could be at dialup speeds, and they could get away with it.

 

Thanks for reading,

 

Jim

[Pgoodwin1]

I got a similar response from my ISP: “things test fine to our infrastructure, we can’t control the rest of the Internet, we don’t know what to tell you.” when I was having similar issues. The very name Internet Service Provider means that they're selling more than just the connection. They are supposed to be managing how their customers are being routed. I don't know how they do that (contracts with other ISPs?) but they will do something if enough customers complain.

"Miraculously" one day, they upgraded their service and things got 2-3x better, until just recently when things have begun to slow down again.

You might try to find out who their other customers are and get a read on what their performance has been. If theirs too has been slow, get them to call in an issue. If theirs has been fast, ask the ISP what the difference is between you and them.

Here's what Testmy's comments are on Speedtest.net

https://testmy.net/ipb/topic/28902-why-do-my-results-differ-from-speedtestnet-ookla-speed-tests/

[CA3LE]

...

So what has our ISP done to address this, in our five months of complaining about it?  The first thing they did, was to send out a tech with a laptop, who hooked it up to their router and went straight to speedtest.net, and when it “tested fine” at the full 20Mbps, he told me “it’s working at 20Mbps, I don’t know what to tell you.”  So of course I had him go to testmy.net, which painted a completely different picture.  He scratched his head with that, and told me he’d report it to his engineering group, but nothing developed from it.  They have sent XO technicians out twice to test the fiber transport, sent their own technicians out to do transfer tests, increased our bandwidth to 30Mbps, temporarily increased it to 100Mbps, temporarily switched our public IP range with another, and tested the complete circuit with Fluke devices on both ends.  None of these actions have made a difference, other than the higher bandwidth enabling us to collectively use more of it, when transferring many files simultaneously…single downloads remain sucky.  I have tried to use the analogy with our account manager (who is non-technical), that we need faster lanes, not a wider freeway.  I have also sent them a video illustrating my own test results, where I was switching between them and Comcast for that ISO download.  Beyond our account manager acting like she cares, the ISP’s engineering group is continuing to toe the line of “things test fine to our infrastructure, we can’t control the rest of the Internet, we don’t know what to tell you.”  It’s maddening.

 

So now you’re probably thinking, why have we put up with this for so long, and not simply dropped them for another ISP?  Yes, we would love to do that, as we could get a 100Mbps Comcast business connection for around $220/month.  However, a year ago, we signed a three-year contract with this ISP (which I personally was not involved with), paying close to $600/month for a 20Mbps connection.  I have read the contract, and the only provisions in there which talk about SLAs/remedies have to do with complete outages, and network latency…there is no language which talks about actual throughput or performance problems.  If we were to cancel the contract “without cause,” we would potentially be on the hook for 75% of the remaining contract, so around $10K.

...

 

Sorry it took me a while to respond, you wrote a lot and I was too busy over the weekend to properly read it.

 

So, you're in a sticky situation.  It has to be frustrating being locked into a contract knowing how green the grass is ... just inches away.  I don't like seeing anyone in Internet contracts... they're bad for the consumer.  In my experience dealing with people and providers around the world... the ISPs that force contracts are generally poor.  In my opinion they don't try as hard to make you happy.  Your provider needs to be afraid that you'll leave... they need to have competition breathing down their neck to make them better.  Same is true for any industry.

 

If they aren't going to deliver and you've tried reasoning with them do what I've done... trick them.  If you have business offices elsewhere, a forwarding address or some type of physical address you can get mail at (OUTSIDE of their service area), tell them that you're moving there.  I did this with Qwest (qWORST) years ago.  I gave them my mother-in-law's address, "Oh really, you don't have service there, oh that's too bad... guess you'll have to let me out of that contract then so I can move on with my life."  -- no joke... no questions asked, they just forwarded the final bill and I was done.   ... eff them.       Then again, I wasn't on the hook for $10k... but if you have an office elsewhere they can't really say anything.  If they can't provide service there they have to let you out. (unless something is explicitly written in the contract otherwise)

 

... a lot easier than trying to argue about the performance and whether or not they're technically holding up their end of the deal.  From what you said, in my opinion they aren't... especially for what you're paying them.  But are they ever going to admit it, probably not... most just point fingers.  Sounds to me like they may even be shaping your bandwidth... Comcast will treat you better.

 

How many computers do you have in that six person office?  Including servers etc.  Did your ISP do all of the networking?

[jimharle]

Thanks for your feedback!

I agree with you on the “no contracts” premise; I’m not sure if that is an option, at least here, for “business-class” Internet, even with Comcast.  I’m still trying to figure out what “business-class” even means; I believe it has to do with transfer caps, in that there are none, whereas on a “home” connection you might have to pay extra after a monthly threshold has been met.  I’ve had Comcast at home for many years, and they used to say the limit was 250GB/month, although they haven’t enforced that for a long while now, whilst they “test” upcharging in other markets.  At our office, we’re easily transferring over 1TB a month in backup data, mostly over the weekends.  But I’m getting off-topic…the ISP we are using for the office is regional vs. one of the “big boys.”  I don’t really have a problem with the “excessive” rate they are charging us, as we’re supporting our local economy, but I do expect to receive what we’re paying for, and we’re just not.  I think that if we can’t get them to acknowledge our problems amicably and reasonably, we’re just going to stop paying the bill and see if they push it, and involve the lawyers if we have to...  We work for a large company, but our office is the only location in Utah, and we can’t bluff them into thinking we’ve moved (one of their offices is in a suburb a few miles away…they know where we live ).  Our company might even consider paying the $10K and just be done with it, but it’s become a “they need to do what is right” ethical thing amongst us, more than the money.  Perhaps a complaint with the BBB is another course of action…especially if we can uncover that their other customers are experiencing the same problems.

I have asked them (via e-mail) about whether a bandwidth arbitration device was in play, but didn’t get a response; I need to bring that up again during a voice call.  I’m sure they’ll deny they are doing any shaping.

As for the computers in our office, we probably have around a dozen devices counting desktops/laptops/servers.  But all of those are irrelevant to the problem, as all of the formal testing we’ve done has bypassed all of that.  The ISP terminates on a Cisco router they provide, and it’s an Ethernet handoff to us, which connects to our Cisco router.  The ISP had nothing to do with our internal networking; we’re actually all computer networking professionals to varying degrees…it’s what we do for a living.  When we test “for real,” we connect a computer (via Ethernet) to their handoff cable and go to town.  But there’s no point in doing that every time now, as we’ve established the problem has nothing to do with CPE.

Lastly, we did find something “new” in the last few days, in that ISO downloads from a couple of “localish” URLs are actually almost normal for us in terms of single-file HTTP downloads.  Those URLs are http://mirrors.xmission.com/slackware-iso/slackware-14.1-iso/ and http://slackware.cs.utah.edu/pub/slackware/slackware-14.1-iso/.  When we download files from either of those, the transfer rate is 20Mbps, and if we download a couple of files simultaneously, we do approach 30Mbps.  XMission is an SLC-based ISP, and our ISP seems to have a peering relationship with them (based on a traceroute), and it is close (around 5ms latency for us).  The route to the University of Utah network goes through two different providers beyond our ISP, but it is also close (around 9ms).  I find this curious, and believe it points to routing/peering issues (with other sites) as being the source of our problems.  I sent this info to the ISP before leaving yesterday; I’m curious to hear what they have to say about that, but again that’ll probably happen during a voice call.

I’ll certainly keep this thread updated as I can with any pertinent info.

 

Cheers,

 

Jim
 

[jimharle]

We gathered some download and traceroute data, and submitted it to the ISP's engineering group late yesterday...attached is the main page showing the comparisons.

 

 

[TriRan]

Nice I'm interested to see what they say

[jimharle]

We haven't heard anything yet from the ISP after sending the comparison matrix. We are asking our legal team if it is okay to stop approving their invoices, so that if the ISP sues us for breach of contract, they were aware it was coming.  Additionally, I filed a complaint with the BBB this morning.  One of the questions on that was "would you be willing to speak to the media?" to which I answered yes.

[TriRan]

I'm really surprised they haven't even responded it's not like you guys are a small contract

[mudmanc4]

We haven't heard anything yet from the ISP after sending the comparison matrix. We are asking our legal team if it is okay to stop approving their invoices, so that if the ISP sues us for breach of contract, they were aware it was coming.  Additionally, I filed a complaint with the BBB this morning.  One of the questions on that was "would you be willing to speak to the media?" to which I answered yes.

You in my opinion are doing everything you can in the attempt to solve the issue, since the ISP is not able to provide a (any) solution thus far, your legal team will certainly gain headway. 

 

I'm more interested in the fact that any business line will be costly, since as you correctly state, "we need faster lanes, not a wider freeway" - which is actually the antithesis of the general business line. As a business line would be the difference between a fat pipe Vs. a skinny pipe. Where this is apparently not performing anywhere near a true business line. You would expect very stable throughput,  at the cost of speed. 

 

It would appear they have placed you on a residential line. In which they do not have the allocated bandwidth to contract, hence they have themselves already breached the contract, if this is the case. From what you say I believe this to be the truth. 

 

I know you have already tested in many ways, and in one I may have missed, therefore I might open at least four terminal windows, type the following into each before hitting return --

curl -C - -O http://ftp.hosteurope.de/mirror/ftp.suse.com/pub/suse/discontinued/i386/9.3/iso/SUSE-9.3-Prof-i386-CD1.iso

Watch the speed of each download, they should all be nearly the same. This is a 686MB download. Of course you do not have to complete any download.

 

The location I selected specifically not to max out your connection but put the toll on the line outside of your local network, and test volatility.  Which should be well below what your 20Mb can handle. 

 

Below is a snapshot during the download (one stream) for comparison

[jimharle]

Thank you mudman, I will do that specific test and report back.  From the previous tests we've done, when we get multiple threads going like that, they do all progress at "about" the same rate (although still quite volatile).  We can almost "max out" our [now 30Mbps] connection, but we have to get *many* threads going, to do that.  We've explained to the ISP that would be great and all if we were doing bittorrent downloads all of the time, but like most businesses, that's not how we roll.

 

On a side note, have you looked at Aria2?  It is a great multi-threaded TCP download tool.

 

You did bring up a good point, in that you wouldn't necessarily want one user in a "business connection" maxing out the pipe and making it slow for everyone else, which is what gave me pause in the beginning, thinking they were using a bandwidth arbitrator to do something like that.  However, I would think that if such a device were in play, the transfers would be "steady" at a restricted rate, rather than erratically jumping all over the place.  It would also make sense that such a device would be "smart," and only invoke rate-limiting on individual connections when the entire pipe was reaching a certain utilization threshold.

 

My BBB complaint was "approved" this morning and submitted to them, and they have until July 31st to respond.

 

We also heard from our legal that we are "okay" to stop approving invoices.

 

Last but not least, I searched my LinkedIn connections for someone at the ISP, and came across a fellow who shares a connection with me, and he is a senior network engineer, having been with them for a little over a year, and previously with another ISP for something like 12 years.  I reached out to him to have a look at our issue as well, and he did respond and say he's looking into it, and he asked for followup information which I provided.  What I'm curious of, is if the real technical people there have their hands tied by poor business decisions, such as oversubscribed peering links or something, and can't really do anything about it quickly.  I just can't imagine they'd still be in business if they were playing games like that, but who knows.

[jimharle]

Here's a screenshot of that Curl test (ftp.hosteurope.de is about 155ms away):

 

[jimharle]

Here is the e-mail we received today from our account manager:

 

We received your complaint from the BBB.

I believe they are going to try switching out equipment before they make the determination on whether you will be released from your contract.

Someone should be in touch shortly to schedule that.
 

Perhaps I'm reading in too much, but this comes across as "we're annoyed" to me. But honestly, what do they expect?

[mudmanc4]

Did you see the fluctuations during the downloads or did they stay rather stable, or at least move in unison, either way a bit too slow for a fiber business line claiming 20Mbps, you should be able to reach the same relative speeds my connection can. 

 

The response from them is still using subterfuge, interesting canned response. 

 

The company went private in 2011, after trying on the OTC and a debacle with Icahn ect. 

[jimharle]

Did you see the fluctuations during the downloads or did they stay rather stable, or at least move in unison, either way a bit too slow for a fiber business line claiming 20Mbps, you should be able to reach the same relative speeds my connection can. 

 

So I ran that test again today, with no other significant traffic on the line.  There were fluctuations on each download thread, ranging from 50k to as high as 400k, but usually in the 100s somewhere.  They did not seem to move in unison, but I was eyeballing it.  I let the four threads run their course, and the average dl speeds of the four threads were 160k, 162k, 182k, and 181k.  What do the fluctuations indicate?

 

Incidentally, they sent a tech out this morning, who replaced their Cisco router (this is the 2nd time they've done that).  I'll give you one guess on whether that made any difference.  Edit: I guess I gave it away in the previous paragraph, but it was a rhetorical question anyway. 

 

Edit #2: You did imply what the fluctuations meant in your previous post.  Would you expect "stability" when being intentionally shaped?  I know my home Comcast downloads fluctuate very little.

 

It's sad when $600/month to download 700MB files in over an hour is supposed to be acceptable.

[mudmanc4]

"50k to as high as 400k" could be interpreted in different ways. If you had no prior influence of a problematic connection , this could indicate nothing more than the un water cable traffic spiked during the transmission. It's difficult to pin down rounding the planet. 

 

Now you have me tagged as much more interested none the less. 

 

Are you able to try running the same tests from CentOS-6.5-x86_64-LiveDVD.iso located in SaltLake City Utah ?

[jimharle]

"50k to as high as 400k" could be interpreted in different ways. If you had no prior influence of a problematic connection , this could indicate nothing more than the un water cable traffic spiked during the transmission. It's difficult to pin down rounding the planet. 

 

Now you have me tagged as much more interested none the less. 

 

Are you able to try running the same tests from CentOS-6.5-x86_64-LiveDVD.iso located in SaltLake City Utah ?

 

So this morning I did the same four-threaded download from Germany, and got slightly better results this time, with the average speeds of each being 307k, 318k, 318k, and 277k, which collectively is around 9Mbps.  I should point out that the ISP did "upgrade" us to 30Mbps as part of the initial troubleshooting, and left it there.  I did that same download from home (Comcast) and got a bit over 20Mbps collectively.

 

So as far as the CentOS ISO from the UofU mirror site you suggested, which is one of the "known good" endpoints for the ISP we called out to them (latency is just under 7ms), a single download (one instance) averaged 3333k (almost 27Mbps).  When I did four at a time, collectively it was a scoach higher (around 28Mbps), with individual instance speed fluctuations between 278k and 1336k.  The average speeds of each was 871k, 862k, 1195k, and 1186k.  So if "everything else" performed like that, or even reasonably close to that, we wouldn't be complaining.  It's obvious that the fiber connection itself is fine with the 30Mbps, yet the ISP continues to act clueless thinking the other problems are related to the router or something we're doing.

[jimharle]

Technicolor BBB Complaint.pdf

 

So the ISP responded to my BBB complaint, and I am attaching their response here.

 

I forwarded this to our legal, and he asked if this was an acceptable resolution, to which I responded:

 

"It seems wrong to me.  The fiber provider (XO Communications) already had a presence in our building before we moved in, and all that needed to be provisioned was a “port” on their equipment…no new infrastructure/fiber needed to be laid, and I believe Veracity confirmed this in an e-mail.  I get that they must have a term agreement with XO so that they would get favorable pricing (which is then marked up and passed on to us as part of the monthly bill), but why should we be harmed because of a failure of Veracity to deliver services?

 

The way I see it, both Technicolor and XO are being harmed by Veracity, so if we pay their ETL, Veracity suffers nothing for providing crappy service.

 

The XO fiber link is like an eight-lane bridge that traverses a large body of water, and at the other end of that bridge, are small, congested roads leading to other places.  Veracity is saying “there’s nothing wrong with the bridge,” which is true, but it is wrong to not take responsibility for the small roads on the other side.  The traffic going across the big bridge can only go as fast as those small roads will allow."

 

For me, this little battle is all about principle, and I no longer care what monetary resolution is established between our companies. What I do care about, is the likelihood that many other (if not all) of this ISP's customers are also receiving substandard service, and I want justice if that's the case.

 

I'm still thinking about what my next move should be.

[jimharle]

Here's the e-mail response I sent this morning:

 

Dear Mr. Erb,

 

I received your response to the BBB complaint I filed against your company, and have attached it here for reference.  I trust you realize that is a public record.

 

So to confirm that I’m understanding this correctly:  Veracity Networks is unable, or unwilling, to provide us Internet service with speeds reasonably close to what we’re paying for.  Veracity Networks will allow us to cancel the contract for Internet service, provided we pay the ETL for the “last mile” fiber provider (XO Communications).  Even though no new fiber had to be laid to the building from which we are leasing office space, Veracity is paying XO a monthly fee for riding that fiber transport, which is then passed on to us as part of the monthly bill.  Presumably, Veracity made a commitment with XO on how long that term would be, and cancelling early will trigger the ETL.  You wish to collect that ETL from us, and excuse yourselves from any responsibility or liability.

 

There is one word which comes to my mind in reaction to this:  Stunning.

 

Both Technicolor and Veracity agree that the problems we are experiencing are unrelated to the underlying XO circuit.  Therefore, the problem must be originating from within Veracity’s network infrastructure, in terms of routing or peering with your upstream providers.  However, Veracity continually fails to acknowledge this, conveniently hiding behind the notion that you can’t guarantee the performance of the “greater Internet.”  While it is true you can’t control the performance of the “greater Internet,” or specific endpoints on it, you can, and do, control the size/type/quality of the connections to your upstream providers.  From our perspective, these are woefully insufficient, with the two known exceptions being your connections to XMission and the University of Utah.

 

It is clear that your technical staff is either incapable of determining root cause and correcting the problem, or unable to because of business decisions over which they have no control.  The way I see it, there are two possibilities:

• Veracity maintains [mostly] poor/oversubscribed peering connections with their upstream providers, meaning that all of Veracity’s customers are experiencing the same problems we are, though may not be aware they are getting ripped off.
• There is some idiosyncrasy with the way our circuit terminates within Veracity’s network infrastructure, be that routing, a physical port or device, whatever.  This would mean that the problem is isolated to our connection, and not impacting other customers.

I really want to believe that it’s the second one, but I feel that if it were that, then all that would need to happen would be, you’d move our termination point to a different device or location (the same as one of your “working” customers), and we’d be off to the races.  But that hasn’t happened, so it begs the question, is it really the first one?  If it were that, it may not be “illegal” to maintain inadequate peering connections, but it would be [at the very least] highly unethical.  Which is it?

 

Asking us to pay the ETL is insulting, and I’m struggling with what it would take for you to face reality here.  Do we need to approach the local news media?  Do I need to publish my “story” on social networks so that all of my LinkedIn contacts can read it and potentially distribute it?  What needs to happen for you guys to do the right thing?

 

Some of my frustrated colleagues in the office want us to just stop paying the bill, let you sue us, and go with Comcast…which would be faster and cheaper.  But personally, I think we should be willing to finish out the contract we originally committed to.  While it is expensive, if it were performing as advertised, it would be meeting our needs, and we would be supporting the local economy.  But the problem is, you are not holding up your end, it is impacting our business, and that is just plain wrong.  What if the water pressure at your house was only averaging 25% of normal all of the time?  That would make you pretty ticked, wouldn’t it?

 

Disclaimer: I realize that this isn’t a “perfect” analogy, as one can’t choose between different suppliers of water; but please stay with me, the general points are the same.

 

Me:        I need to report a problem with the water supply to our house.  The water pressure is very low, and has been that way since we moved in.  We and our kids have to get up several hours earlier than normal, so that we have time to take the resultant long showers, so we’re not late for church.  We had a plumber come in to take a look, and after measuring the flow, he said we’re only receiving an average of 25% of the water we should be.  He said we should be able to receive twenty gallons per minute, but we’re only averaging five.

 

City:       Okay, that sounds odd.  Are you sure you don’t have a leak somewhere?  Perhaps a toilet is malfunctioning and running all the time?

 

Me:        No, we’ve checked for things like that, and everything appears normal.  We’ve also analyzed the water bills from the city, and they verify that our water consumption has been very low.  Incidentally, our bill is ten times higher than what our friends pay in different neighborhoods.  Why is that?

 

City:       Well, your neighborhood requires us to install special main pipes, which connect your house to our water system.  Those pipes are very high quality, made out of a titanium alloy, and capable of handling over a thousand gallons per minute, although we have that limited to twenty, as that’s what your currently paying for.  However, unlike your friends, you are allowed to consume as much water as you want every month, for no additional charge.  Much of what you are paying for is that special pipe.

 

Me:        Okay, that sounds odd.  But whatever, please send someone out to troubleshoot, as we’re only getting five instead of twenty.

 

City:       Okay, we’ll send someone out.

 

Days/weeks pass, as the city sporadically sends technicians out to troubleshoot.  Each time they do, the technician has to disconnect the house from the water supply, which is disruptive and inconvenient.

 

City:       We haven’t been able to find a problem, so we changed the setting on your titanium pipe to allow thirty gallons per minute.  Try it now.

 

Me:        That hasn’t made a difference; we’re still only averaging five gallons per minute.  A different plumber came to the house and verified that.  Will you send someone out again, please?

 

City:       We’ll check some things and get back to you.

 

More days/weeks pass.

 

City:       Good news!  We tested your titanium pipe, and it is functioning normally – we can push thirty gallons of water per minute through it just fine, when we hook up our water truck to it.

 

Me:        That’s great, but we’re still only getting five.  Since the titanium pipe is not the problem, perhaps the problem is with your water supply not being good enough to handle the neighborhood’s capacity needs?

 

City:       Well, we can’t guarantee that your house, or anyone else’s, will always receive the maximum allotment.  What if twenty people in the neighborhood are taking showers at the same time?  We don’t have control over that.

 

Me:        Yes, I understand that, but shouldn’t your water system be built with enough capacity, so that your customers can receive reasonable amounts of water and pressure most of the time?

 

City:       What?

 

Me:        What?

 

City:       We turned your dial up to one hundred gallons per minute.  Try it now.

 

Me:        Nope, we’re still only getting five.

 

City:       We don’t know what to tell you.  Your titanium pipe is capable of thirty, and we’ve proven that.

 

Me:        Yes, but you can only deliver five, which is not acceptable.  We’d like to switch to a different water provider.

 

City:       No problem, we’re here to help.  Just pay us seven thousand dollars, and you can be on your way.

 

Me:        Seven thousand dollars?!  For what?

 

City:       For the titanium pipe.  Someone has to pay for that.  That stuff is expensive!  Tony (our Titanium supplier) is going to make us pay for it, whether you use it or not.  So you need to pay for that.

 

Me:        But you’re only able to deliver five gallons of water anyway, not near the twenty we’ve been paying for (for months), and even further from the thirty and one hundred you set it to…what would happen if we needed and paid for one thousand?  Why should we have to pay for your ineptitude?

 

City:       You don’t seem to understand how water delivery works.  Pay us our money and move along.

 

Thanks for reading.

 

-Jim

[jimharle]

I received a reply this morning...looks like I'll be meeting with them tomorrow, which is encouraging.

 

Jim,

 

Would you and your team be willing to meet with our CTO, COO and myself to dig deeper into this matter.  Veracity does not want unhappy or disenfranchised customers.  We’d like to meet on Tuesday at our offices in Draper, would this be amiable to you.  If so, can you suggest a couple of time options for meeting?

 

Marshall E. Erb

[jimharle]

So we met with Mr. Erb and other folks from his company on Tuesday afternoon.

 

To make a long story short, they seem to be determined to solve our problem, and are “just as frustrated as we are.”  We talked at length about tests that they and we have performed.  I believe we generated some mutual respect, and a determination to fix the problem.  It does seem that we are an isolated case for them, and that their service doesn’t suck for all of their customers.

We are scheduled for a conference call on Friday, during which time we’ll be disconnecting our router and doing some detailed troubleshooting.  They also requested some packet captures, which I provided today.

I also installed an Ethernet switch between our respective routers, and plugged a vswitch from our ESXi hypervisor into it, so that I could give them a couple of VMs on the outside they could test remotely with.  So there’s a Linux box and a Windows box configured with veracity user accounts.  I told them they could do “whatever they want” with them.

The BBB response to their response is due Monday, so I was going to hold off on that until then.

Cheers!

Jim

[TriRan]

Well that's a bit of good news amongst all that bad news at least it's been and being escalated

[jimharle]

We had our troubleshooting session today with Chris (VP of Engineering for Veracity)...here is the post-mortem summary e-mail I sent:

 

All,

 

After extended testing with Chris today, he determined that the problem must be due to packet loss within the XO network.  This packet loss is inconsequential with low-latency connections, but becomes more evident over more distant connections.  This jives with what we’ve experienced with those XMission/UofU sites which were relatively fast, as the latency to those is low.  Chris did his own packet captures, and the ones that I did also confirm the theory:

 

Comcast-to-Technicolor LADC:

     Client-side suspected TCP retransmissions: 56

     Server-side suspected TCP retransmissions: 0

 

Veracity-to-Technicolor LADC:

     Client-side suspected TCP retransmissions: 6,854

     Server-side suspected TCP retransmissions: 4,022

 

After Chris left, I was concerned about the Ethernet run within our building walls, going from the XO device in the equipment room, to a wall jack in our suite.  So we ran our own long Ethernet cable along the floor, and connected the Veracity router directly to the XO device.  This exercise did not change the symptoms, so we can reasonably eliminate the cable run as a problem.

 

Chris was going to push back with XO – and Chris, please correct anything I may have misstated.

 

Thanks!

 

Jim

[jimharle]

Here’s the response I filed with the BBB yesterday.

 

CASE ID: 22168058
On August 11, 2014, you provided the following information:

(The consumer indicated he/she DID NOT accept the response from the business.)

After Mr. Erb's response was posted to the BBB site, I e-mailed him and other Veracity personnel, indicating that their request to have Technicolor pay the XO ETL was not fair nor equitable. Mr. Erb then invited us to their Draper office for a meeting, which we did accept and attend on August 5th. In that meeting, we shared our mutual frustrations, and agreed on efforts to work together to address the problem to resolution.

In the days that followed, we gathered additional technical information, and then on Friday the 8th, participated in a technical "deep dive" with their high-level engineering staff, which took place over several hours. At the conclusion of that technical session, both parties had gathered sufficient technical evidence indicating that the root of the problem is persistent packet loss/congestion within XO's network, which sits between our building and Veracity's infrastructure. Consequently, Veracity's engineering staff is pushing back on XO to get the problem corrected. Neither Technicolor nor Veracity have direct control over XO's network, although Veracity contracts with XO to provide "last mile" service for our connection, since no Veracity fiber is present in our building.

We are hopeful that Veracity is successful in getting XO to correct the problem, and feel it would be reasonable to give them until August 31st to get the problem corrected. Technicolor has provided Veracity remote access to virtual machines within our building, so that they can perform any necessary testing 24/7 without relying on our involvement, although we are happy to assist with any troubleshooting should it be necessary.

We maintain that the relationship with XO rests with Veracity, and that it is their responsibility to drive them to resolution. If resolution can be obtained by August 31st, we feel it would be reasonable to finish out the contract. If not, we feel that it would be reasonable for Veracity to release us from the contract, with no ETL or other penalties.

Sincerely,

Jim Harle

[jimharle]

So this saga essentially came to an end today; our account manager called me and said they are releasing us from the contract, with no penalties. They were unable to get XO to fix the last mile problem, and do not have other alternatives for us.  They are sending that to us in writing for the lawyers, and we're in the process of provisioning Comcast Business service. There will surely be more bureaucracy with the latter, as it needs to go through our corporate sourcing, but at least we can see a light at the end of the tunnel now.