jimharle

FWIW there is one major problem with an on-net test, whether it's Ookla-based or not. An on-net endpoint is going to be low-latency to the customer (~ 7ms or so), and *if* there are packet loss problems within the last mile, those will be masked by a low-latency test because the TCP retransmissions will "recover" so quickly. When performing a low-latency test, it needs to be done in parallel with a packet capture to ensure there is no significant packet loss. See this thread for more: https://testmy.net/ipb/topic/32071-upload-speed-issue-comcast/

So if I were in your shoes, I would want to demonstrate to Comcast where the packet loss is occurring. Firstly I would identify several static endpoints to test with, and ones where they are not part of a pool of possibilities (in other words, when uploading to YouTube, I'm not sure that you're hitting the same endpoint(s) every time you do an upload - I would include endpoints that don't change such as FTP servers). I would then gather the packet captures for the multiple endpoints (during the problem window), to show that the packet loss is not specific to one endpoint (which they could argue might be part of the problem). Next I would take my laptop (always testing with the same computer rules out issues on it) to my in-laws' house (who live in a different suburb but still have Comcast) and do the same packet captures from there. Assuming no problems from there, I would submit the information, along with traceroutes, to Comcast to show that the problem is always at your house, and doesn't follow your laptop or other locations you test from. When we were narrowing down the problem in our office, I had the advantage of being able to run packet captures on the remote end (an FTP server) at the same time as the client side, to show there were not [excessive] packet loss issues with different client locations, and that our office was demonstrably different than another endpoint using the same ISP where the traceroute differed in only the last couple of hops (the "last mile" of our connection). Does this make sense?

Yes, and a lot of retransmissions. Is this during the problem time?

Eric, have you tried a pingpath test, both when the problem is occurring and when it is not? Another way to validate that packet loss is occurring, is to run a Wireshark packet capture during both times (filtering for a specific endpoint to remove noise), and using the Analyze --> Expert Info feature of Wireshark to look at the number of retransmissions. We have known packet loss at our office, and we have compared the same transfer from the same endpoint from a "known good" ISP (which, ironically, is Comcast for this), and here are the results: Office: Comcast: See the difference?

I installed Windows 10 in a VM to see what the hubub was about. The decision to include both a Start Screen and a Start Menu (but not both enabled at the same time) is what they should have done in Windows 8. Had they done that, the latter would have been a lot more successful with desktop users. I don't get the versioning either, but IMO this should be called Windows 8.2 at this point, as I'm not seeing major differences, at least not yet. I'm also not seeing "Tesla vs. Prius" differences from Windows 7.

Perhaps, but the default view of the FileZilla client is to show the average throughput for each stream - I always set it to show the "momentary" speed instead of the average speed (look at the "Interface" section under Settings). Additionally, it's possible that packet loss due to a congested route could be "symmetrical." All I'm really trying to say is that it's not a forgone conclusion that Comcast is shaping. Another approach to take, is to compare results from different ISPs to the same endpoints, during the troubles on the one. Isn't there also a "pingpath" test that Damon offers?

Just to throw it out there, this could be the result of packet loss somewhere along the path vs. throttling on Comcast's part. That's one way to explain the limited single threads but still "normal" in the aggregate.

Do you have evidence of this, or is any of it speculation? I liken the Microsoft Account with the Apple ID in OS X. Both make life much more convenient when you use multiple computers. I'm not saying there's nothing shady going on, I'm just skeptical about the level of nefariousness.

Seems to be working better now

When you have a torrent downloading "at full speed," look at the details of a given torrent in your client, to see what the per-peer speeds are. I'd bet that each peer's speed to you is also relatively slow...something to check.

I'd love to bask in the slowness of the test, but I still can't get it to work, even when clicking links in the provided list. For example, when I click on https://testmy.net/mX/C0yz1 (googleusercontent.com), I get a blank page when using Firefox on Windows/Mac, a blank page using Safari on Mac, and an HTTP 500 error from IE11 on Windows.

I changed the test file to a 300KB one, but I'm still having problems . Am I doing something wrong?

Does it work with https URLs? I'm trying https://cinema.technicolor.com/kdmrepo/test.jpg and it looks like it takes it, but when I click "start test" nothing meaningful happens.

@creativestuff - I'm sure someone will correct me if I'm wrong, but I believe the Auto Test feature was designed to gather performance results over time, not to diagnose resilliancy problems. How are you connected to your gateway device - wired or wireless? If I suspected that my ISP's equipment was intermittently failing, I would perform a traceroute to something (say, 8.8.8.8 ) when it was working, from that determine what public address is the first "hop" after my own equipment, and then monitor that public address from a different ISP's connection, or perhaps from a friend/family member in a different location who shares my ISP, but is not having problems.

Here’s the response I filed with the BBB yesterday. CASE ID: 22168058 On August 11, 2014, you provided the following information: (The consumer indicated he/she DID NOT accept the response from the business.) After Mr. Erb's response was posted to the BBB site, I e-mailed him and other Veracity personnel, indicating that their request to have Technicolor pay the XO ETL was not fair nor equitable. Mr. Erb then invited us to their Draper office for a meeting, which we did accept and attend on August 5th. In that meeting, we shared our mutual frustrations, and agreed on efforts to work together to address the problem to resolution. In the days that followed, we gathered additional technical information, and then on Friday the 8th, participated in a technical "deep dive" with their high-level engineering staff, which took place over several hours. At the conclusion of that technical session, both parties had gathered sufficient technical evidence indicating that the root of the problem is persistent packet loss/congestion within XO's network, which sits between our building and Veracity's infrastructure. Consequently, Veracity's engineering staff is pushing back on XO to get the problem corrected. Neither Technicolor nor Veracity have direct control over XO's network, although Veracity contracts with XO to provide "last mile" service for our connection, since no Veracity fiber is present in our building. We are hopeful that Veracity is successful in getting XO to correct the problem, and feel it would be reasonable to give them until August 31st to get the problem corrected. Technicolor has provided Veracity remote access to virtual machines within our building, so that they can perform any necessary testing 24/7 without relying on our involvement, although we are happy to assist with any troubleshooting should it be necessary. We maintain that the relationship with XO rests with Veracity, and that it is their responsibility to drive them to resolution. If resolution can be obtained by August 31st, we feel it would be reasonable to finish out the contract. If not, we feel that it would be reasonable for Veracity to release us from the contract, with no ETL or other penalties. Sincerely, Jim Harle

We had our troubleshooting session today with Chris (VP of Engineering for Veracity)...here is the post-mortem summary e-mail I sent: All, After extended testing with Chris today, he determined that the problem must be due to packet loss within the XO network. This packet loss is inconsequential with low-latency connections, but becomes more evident over more distant connections. This jives with what we’ve experienced with those XMission/UofU sites which were relatively fast, as the latency to those is low. Chris did his own packet captures, and the ones that I did also confirm the theory: Comcast-to-Technicolor LADC: Client-side suspected TCP retransmissions: 56 Server-side suspected TCP retransmissions: 0 Veracity-to-Technicolor LADC: Client-side suspected TCP retransmissions: 6,854 Server-side suspected TCP retransmissions: 4,022 After Chris left, I was concerned about the Ethernet run within our building walls, going from the XO device in the equipment room, to a wall jack in our suite. So we ran our own long Ethernet cable along the floor, and connected the Veracity router directly to the XO device. This exercise did not change the symptoms, so we can reasonably eliminate the cable run as a problem. Chris was going to push back with XO – and Chris, please correct anything I may have misstated. Thanks! Jim

Why would you want to hide that thing of beauty behind a nightstand?

So we met with Mr. Erb and other folks from his company on Tuesday afternoon. To make a long story short, they seem to be determined to solve our problem, and are “just as frustrated as we are.” We talked at length about tests that they and we have performed. I believe we generated some mutual respect, and a determination to fix the problem. It does seem that we are an isolated case for them, and that their service doesn’t suck for all of their customers. We are scheduled for a conference call on Friday, during which time we’ll be disconnecting our router and doing some detailed troubleshooting. They also requested some packet captures, which I provided today. I also installed an Ethernet switch between our respective routers, and plugged a vswitch from our ESXi hypervisor into it, so that I could give them a couple of VMs on the outside they could test remotely with. So there’s a Linux box and a Windows box configured with veracity user accounts. I told them they could do “whatever they want” with them. The BBB response to their response is due Monday, so I was going to hold off on that until then. Cheers! Jim

Awesome! Please post your test results for the 10/2 service. I will bet your download performance is better than our office, and we allegedly have a 30/30 symmetrical commercial connection. I just tested it while signed in, and got 4.5 down and 9.7 up, which is typical...but that's another saga

Is the computer you're testing with connecting wirelessly or via an Ethernet cable? You should always test via a wired connection to eliminate wireless variables. I think I misunderstood your comment about your router - I think you meant wireless router, as opposed to a device provided by the ISP. Hopefully your ISP is providing your service via a physical line.

I received a reply this morning...looks like I'll be meeting with them tomorrow, which is encouraging. Jim, Would you and your team be willing to meet with our CTO, COO and myself to dig deeper into this matter. Veracity does not want unhappy or disenfranchised customers. We’d like to meet on Tuesday at our offices in Draper, would this be amiable to you. If so, can you suggest a couple of time options for meeting? Marshall E. Erb

"full bars for router" implies wireless, what is the transport from ISP?

Here's the e-mail response I sent this morning: Dear Mr. Erb, I received your response to the BBB complaint I filed against your company, and have attached it here for reference. I trust you realize that is a public record. So to confirm that I’m understanding this correctly: Veracity Networks is unable, or unwilling, to provide us Internet service with speeds reasonably close to what we’re paying for. Veracity Networks will allow us to cancel the contract for Internet service, provided we pay the ETL for the “last mile” fiber provider (XO Communications). Even though no new fiber had to be laid to the building from which we are leasing office space, Veracity is paying XO a monthly fee for riding that fiber transport, which is then passed on to us as part of the monthly bill. Presumably, Veracity made a commitment with XO on how long that term would be, and cancelling early will trigger the ETL. You wish to collect that ETL from us, and excuse yourselves from any responsibility or liability. There is one word which comes to my mind in reaction to this: Stunning. Both Technicolor and Veracity agree that the problems we are experiencing are unrelated to the underlying XO circuit. Therefore, the problem must be originating from within Veracity’s network infrastructure, in terms of routing or peering with your upstream providers. However, Veracity continually fails to acknowledge this, conveniently hiding behind the notion that you can’t guarantee the performance of the “greater Internet.” While it is true you can’t control the performance of the “greater Internet,” or specific endpoints on it, you can, and do, control the size/type/quality of the connections to your upstream providers. From our perspective, these are woefully insufficient, with the two known exceptions being your connections to XMission and the University of Utah. It is clear that your technical staff is either incapable of determining root cause and correcting the problem, or unable to because of business decisions over which they have no control. The way I see it, there are two possibilities: Veracity maintains [mostly] poor/oversubscribed peering connections with their upstream providers, meaning that all of Veracity’s customers are experiencing the same problems we are, though may not be aware they are getting ripped off. There is some idiosyncrasy with the way our circuit terminates within Veracity’s network infrastructure, be that routing, a physical port or device, whatever. This would mean that the problem is isolated to our connection, and not impacting other customers. I really want to believe that it’s the second one, but I feel that if it were that, then all that would need to happen would be, you’d move our termination point to a different device or location (the same as one of your “working” customers), and we’d be off to the races. But that hasn’t happened, so it begs the question, is it really the first one? If it were that, it may not be “illegal” to maintain inadequate peering connections, but it would be [at the very least] highly unethical. Which is it? Asking us to pay the ETL is insulting, and I’m struggling with what it would take for you to face reality here. Do we need to approach the local news media? Do I need to publish my “story” on social networks so that all of my LinkedIn contacts can read it and potentially distribute it? What needs to happen for you guys to do the right thing? Some of my frustrated colleagues in the office want us to just stop paying the bill, let you sue us, and go with Comcast…which would be faster and cheaper. But personally, I think we should be willing to finish out the contract we originally committed to. While it is expensive, if it were performing as advertised, it would be meeting our needs, and we would be supporting the local economy. But the problem is, you are not holding up your end, it is impacting our business, and that is just plain wrong. What if the water pressure at your house was only averaging 25% of normal all of the time? That would make you pretty ticked, wouldn’t it? Disclaimer: I realize that this isn’t a “perfect” analogy, as one can’t choose between different suppliers of water; but please stay with me, the general points are the same. Me: I need to report a problem with the water supply to our house. The water pressure is very low, and has been that way since we moved in. We and our kids have to get up several hours earlier than normal, so that we have time to take the resultant long showers, so we’re not late for church. We had a plumber come in to take a look, and after measuring the flow, he said we’re only receiving an average of 25% of the water we should be. He said we should be able to receive twenty gallons per minute, but we’re only averaging five. City: Okay, that sounds odd. Are you sure you don’t have a leak somewhere? Perhaps a toilet is malfunctioning and running all the time? Me: No, we’ve checked for things like that, and everything appears normal. We’ve also analyzed the water bills from the city, and they verify that our water consumption has been very low. Incidentally, our bill is ten times higher than what our friends pay in different neighborhoods. Why is that? City: Well, your neighborhood requires us to install special main pipes, which connect your house to our water system. Those pipes are very high quality, made out of a titanium alloy, and capable of handling over a thousand gallons per minute, although we have that limited to twenty, as that’s what your currently paying for. However, unlike your friends, you are allowed to consume as much water as you want every month, for no additional charge. Much of what you are paying for is that special pipe. Me: Okay, that sounds odd. But whatever, please send someone out to troubleshoot, as we’re only getting five instead of twenty. City: Okay, we’ll send someone out. Days/weeks pass, as the city sporadically sends technicians out to troubleshoot. Each time they do, the technician has to disconnect the house from the water supply, which is disruptive and inconvenient. City: We haven’t been able to find a problem, so we changed the setting on your titanium pipe to allow thirty gallons per minute. Try it now. Me: That hasn’t made a difference; we’re still only averaging five gallons per minute. A different plumber came to the house and verified that. Will you send someone out again, please? City: We’ll check some things and get back to you. More days/weeks pass. City: Good news! We tested your titanium pipe, and it is functioning normally – we can push thirty gallons of water per minute through it just fine, when we hook up our water truck to it. Me: That’s great, but we’re still only getting five. Since the titanium pipe is not the problem, perhaps the problem is with your water supply not being good enough to handle the neighborhood’s capacity needs? City: Well, we can’t guarantee that your house, or anyone else’s, will always receive the maximum allotment. What if twenty people in the neighborhood are taking showers at the same time? We don’t have control over that. Me: Yes, I understand that, but shouldn’t your water system be built with enough capacity, so that your customers can receive reasonable amounts of water and pressure most of the time? City: What? Me: What? City: We turned your dial up to one hundred gallons per minute. Try it now. Me: Nope, we’re still only getting five. City: We don’t know what to tell you. Your titanium pipe is capable of thirty, and we’ve proven that. Me: Yes, but you can only deliver five, which is not acceptable. We’d like to switch to a different water provider. City: No problem, we’re here to help. Just pay us seven thousand dollars, and you can be on your way. Me: Seven thousand dollars?! For what? City: For the titanium pipe. Someone has to pay for that. That stuff is expensive! Tony (our Titanium supplier) is going to make us pay for it, whether you use it or not. So you need to pay for that. Me: But you’re only able to deliver five gallons of water anyway, not near the twenty we’ve been paying for (for months), and even further from the thirty and one hundred you set it to…what would happen if we needed and paid for one thousand? Why should we have to pay for your ineptitude? City: You don’t seem to understand how water delivery works. Pay us our money and move along. Thanks for reading. -Jim

Technicolor BBB Complaint.pdf So the ISP responded to my BBB complaint, and I am attaching their response here. I forwarded this to our legal, and he asked if this was an acceptable resolution, to which I responded: "It seems wrong to me. The fiber provider (XO Communications) already had a presence in our building before we moved in, and all that needed to be provisioned was a “port” on their equipment…no new infrastructure/fiber needed to be laid, and I believe Veracity confirmed this in an e-mail. I get that they must have a term agreement with XO so that they would get favorable pricing (which is then marked up and passed on to us as part of the monthly bill), but why should we be harmed because of a failure of Veracity to deliver services? The way I see it, both Technicolor and XO are being harmed by Veracity, so if we pay their ETL, Veracity suffers nothing for providing crappy service. The XO fiber link is like an eight-lane bridge that traverses a large body of water, and at the other end of that bridge, are small, congested roads leading to other places. Veracity is saying “there’s nothing wrong with the bridge,” which is true, but it is wrong to not take responsibility for the small roads on the other side. The traffic going across the big bridge can only go as fast as those small roads will allow." For me, this little battle is all about principle, and I no longer care what monetary resolution is established between our companies. What I do care about, is the likelihood that many other (if not all) of this ISP's customers are also receiving substandard service, and I want justice if that's the case. I'm still thinking about what my next move should be.

I agree that it doesn't seem right that subscribers should have to pay a rental fee for their device, when that device is used to bolster their WiFi hotspots for others. If you're not using their voice product (who needs land lines these days?), do what I do and purchase your own modem...your ROI is less than a year I believe.