Problems on Dallas Server ?

[Pgoodwin1]

The last couple of days, I've been registering really low speeds when using my normal default Dallas server. It's giving me speeds less than 1/3 of what I see on the Washington DC server. Is there a problem on that server?

 

Normally I'm between 10-15 Mbps down on that server. I was measuring as low as 3 tonight, and in the 5-8 range last night. Happens on more than one device (iPad and iMac)

I've rebooted my Airport Extreme, Cable Modem and Router - no help. Plus, when I use the Washington DC server, I see times that are pretty normal looking for this time of night on Time Warner Cincinnati

I'm not Multi threading

[Pgoodwin1]

Using the Alternate CDN Push Alternate Route to Dallas, the times were very fast. Faster than than using the Washington DC server by 30-50%, and 4-5 times faster than the standard Dallas server default

With a few more tests on the CDN Alt Route, I saw quite a variation in test results. Most of them were very good on download, but one was really low like the standard Dallas server numbers (down). I also saw some number using the CDN Alt route that were 1/2 of normal for upload speeds. The standard Dalla and Washington servers pretty much always give me about 0.8-1 Mbps, and the ALt CDN route was giving me about 0.35 a little bit ago for upload.

Dallas

Wed Aug 14 2013 @ 11:46:39 pm   448 kB 708 Kbps 89 kB/s 598569765972 Time Warner Cable  jyvFmt4 share  Wed Aug 14 2013 @ 11:46:31 pm   4.2 MB 4.78 Mbps 598 kB/s 598569765972 Time Warner Cable  a5rymhu share 

 

Washington

Washington

Wed Aug 14 2013 @ 11:46:12 pm     928 kB 1.03 Mbps 128 kB/s           Wed Aug 14 2013 @ 11:45:52 pm     13 MB 11.28 Mbps 1.41 MB/s

CDN Alt Push

 

Wed Aug 14 2013 @ 11:41:51 pm   448 kB 343 Kbps

 

Wed Aug 14 2013 @ 11:41:26 pm     12.8 MB 17.24 Mbps

[CA3LE]

I'm testing well on LTE

[Pgoodwin1]

Strange. This morning it's working fine on both the iPad and the iMac. Must have been a warp in the force.

Any idea what would cause that rapid fall off at the tail end of the test? Virtually every test I run does it. Smartest, or choosing file sizes, they all have that ramp down at the end that lowers the average speed value.

[Pgoodwin1]

Tonight is back doing the same thing.mtye Dallas server gives very slow Download times, where the CDN Alt Route and Washington DC server show what my equipment usually looks like for downloads.

Also note that the upload times on the CDN Alt server is roughly 1/2 what it shows on the Dallas and Washington DC servers

IMac:

The star is Dallas. The flag is Washington. The plain stack icon is the CDN Alt Route server.

Here's the similar iPad results:

The slow intermittent upload speed on the CDN Alt server seems to be intermittent. Every once in a while I get a normal upload speed on it (about 0.8-0.9 Mbps

[CA3LE]

There may be congestion along your route to that server.  I can tell you that it's not an error.

 

I have a new tool that will be released soon that will help highlight problem areas along your route.  Join the beta program if you'd like to try it out now.

[Pgoodwin1]

I just tried the TraceMy.net tool. It's already great. I've only spent minutes with it and don't claim yet to understand everything it's telling me. But, I got normal looking Up and Down speeds this morning on the Dallas server, so I know what the trace looks like under normal operating conditions. It showed a couple of 600+ millisecond ping numbers, and I don't know how good, bad, or typical that is for Time Warner Cincinnati; it sounds really long. I pulled up a Community graph and it looks like an order of magnitude lower ping numbers, but since my times were as fast as they are when everything is working right, I expect the 625 mS ping numbers were normal for my ISP. My TWC plan isn't one of their fast ones; it's just one step up from their minimum offered plan (if I remember right). For all I know, they program in delays for the cheap plans and remove them for the expensive ones, and everybody uses the same equipment.

I'll play with TraceMy.net some more soon, and maybe ask some specific questions

The one thing that confused me a little with the TraceMy.net results was the security for sharing info that came up. It talked about creating a TMN ID. I guessed that that message was for new users without a TMN ID that would be using the eventual released version, and it was telling them to sign up for a TMN account. I wasn't sure though that the data I was looking at with the IP addresses was something I should be sharing. As I said, I haven't played with it much yet, so I'm sure I'll figure stuff out after I use it more.

Your note comes up after a trace route test "Make this safe to share by removing identifying target information and converting the address into an anonymous TMN ID."

After I hit the link part of that "Make this safe to share", it displays a similar page with a temporary note saying that it was safe to share. But I couldn't tell what the difference was from the prior page that supposedly wasn't safe to share.

Note-9/3/13-I figured out the difference.

Edited September 4, 2013 by Pgoodwin1

[Pgoodwin1]

I'm still having issues when speed testing on the Dallas server. I get 3x the speed using the Washington DC server, which shows a speed more like I'm used to seeing during prime time evening hours on the Dallas aerver. When I do the TraceMy.net trace test, the ping times I see are very close to what I see when my speeds are registering normal - during non-prime time hours. I'm puzzled by the extremely low speeds I'm seeing using the Dallas server. Anyone have any idea what the issue(s) might be? This phenomenon has been going on for a while now-like 3 weeks or so.

Note that the Dallas server is identified with the star in the upper data picture. And the Washing DC server is the US flag.

Note. I get ping times like these when I'm registering close to 15 Mbps download from either server. When on the Dallas server, and my speeds are very low like the above ~3 Mbps, my ping times are virtually the same.

Edited September 4, 2013 by Pgoodwin1

[Pgoodwin1]

Again, this morning I'm getting the normal fast (for me) speed readings on the Dallas server....see the last two points on the chart/data list Note that the ping times are about the same as last night when I was experiencing the really slow download times.

Edited September 4, 2013 by Pgoodwin1

[HubCity]

I have the same problem with the Dallas server. For months I've been getting around 40 Mb/s and today I get less than 8 Mb/s. East and West coast tests are giving around 18-20 Mb/s.

[HubCity]

And today, speed is back to normal for me. Must have been some temporary routing/bottleneck slowdown.

[Pgoodwin1]

CA3LE. I still consistently get slow times on the Dallas server during the evening hours. When I switch to the Washington server, my download speeds are almost 2x what the Dallas server data shows. And when I use the CDN Alt Push Route to Dallas, it's roughly 3x. The key for the data below is the "star" is Dallas, the "flag" is Washington, and the ones with just the Apple are the CDN Alt Push Route to Dallas.

This phenomenon is repeatable every night between say 8 PM and midnight EDST. The TraceMy.net results look OK to me, but I don't know where that test takes me geographically.

Here's what the Dallas, Washington and CDN Push test curves look like.

Many times the Dallas server averages will be under 10 Mbps. Any thoughts about how to determine what the issue(s) might be?

Also, why do you think the CDN Push Alt server connection would be so much faster?

[Pgoodwin1]

Curiously again it's more normal in the morning

[CA3LE]

Curiously again it's more normal in the morningimage.jpg

 

I'm still working to understand TraceMy.net's results.  I don't think an address has to have bad response time to slow you down.  So until I fully understand what I built and can explain my findings... take those results with a grain of salt.  Right now it's only tracing to my main server in Dallas but when complete it will utilize my entire network.

 

CDN may perform better for you because the transaction is relayed to a server closer to you.  Look at the network map.  For some people this performs better... others will get better performance directly to TMN servers.

 

Here's my own result at about 8:30 the other night... I test just fine during those hours... but I am slightly slower than my best.

 

 

I think that Time Warner has some routes that are congested during peak hours in your area.  Other servers, like testing to CDN for instance, will take different routes... possibly bypassing the congestion.  My goal is to make TraceMy.net detect and idetify those problem areas... and give you the ammunition you need to complain to your ISP.

[CA3LE]

Curiously again it's more normal in the morningimage.jpg

 

I noticed that Tracy in KY (only a few hours from you) who's also routed through TWC is having drops during the same hours as you.  You might want to look at this topic https://testmy.net/ipb/topic/31124-hi-i-am-tracy-b-from-frankfort-ky/.  His issue is obviously way worse than yours but may be related.

 

-D

[Pgoodwin1]

Thanks CA3LE. I will take some time and study the network map. I too think there is congestion somewhere in TWC's path to the Dallas server. I've checked speeds using some speed test sites other than TMN, and while I don't think their speeds are at all indicative of any real speed numbers, I can see a difference in the results on thos sites - slower in the evening, just like I'm seeing here. I have an iPad app called OOKLA Speedtest, but I can't get results that I believe. I think there's huge variation in the various servers they have on their pick list. Some of them just must not have much bandwidth. I sometimes see 100% difference between different servers in the same city. I have had faster times using that app when I choose a server on the East coast and similar slower times choosing a server that is south of here (like the Dallas server is). But I don't have a lot of faith in their server system. On one of them, my download speeds were slower than my upload speeds.

And I will look at Trace in Ky. Thanks

[Sparticus]

This was tested using my iPhone hotspot on VZW 4G LTE w/ one bar of signal.

 

:::.. Download Speed Test Result Details ..:::

Download Connection Speed:: 10320 Kbps or 10.3 Mbps 

Download Speed Test Size:: 6 MB or 6144 kB or 6291456 bytes

Download Binary File Transfer Speed:: 1290 kB/s or 1.3 MB/s

Tested At:: http://TestMy.net Version 13

Validation:: https://testmy.net/db/xPLD8ir

TiP Measurement Summary:: Min 4 Mbps | Middle Avg 12.63 Mbps | Max 14.93 Mbps | 43% Variance

TiP Data Points:: 4 Mbps, 6.99 Mbps, 8.09 Mbps, 11.92 Mbps, 12.93 Mbps, 12.75 Mbps, 12.32 Mbps, 13.52 Mbps, 14.93 Mbps, 14.79 Mbps, 14.26 Mbps, 13.72 Mbps, 13 Mbps, 12.82 Mbps, 13.72 Mbps, 13.88 Mbps, 12.68 Mbps, 12.38 Mbps, 7.29 Mbps

More Stats:: https://testmy.net/quickstats/Sparticus https://testmy.net/compID/64370838453

Test Time:: 2013-11-03 22:47:28 Local Time 

Location:: Charlotte, NC US >> Destination:: Dallas, TX US

1MB Download in 0.79 Seconds - 1GB Download in ~13 Minutes - 184X faster than 56K

This test of exactly 6144 kB took 4.892 seconds to complete

Running at 145% of hosts average (Verizon Wireless https://testmy.net/hoststats/verizon_wireless) 

User Agent:: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71 [!]

[Pgoodwin1]

CA3LE. I've amassed quite a bit of data on your site here in the last 7 days. It's very repeatable that my download speeds slow down by a factor of 8 from here to the Dallas server, and by a factor of about 2 to the Washington server during the afternoons and evenings. By 1AM the times improve and by early morning the times are about what I'd call normal. So TWC definitely has a traffic bottleneck during high usage hours. I've confirmed the TMN test result on Speedtest.org where although their actual speeds aren't as believable as TMN, they show basically the same bottleneck (they happen to have a Dallas server and a NJ server that show almost the same results as here on TMN to the Dallas and Washington servers. I don't believe in their numbers per SE, but the trends versus hours of the day are the same. What's totally unbelievable is that TWC's speed test shows between 45 and 50 Mbps no matter what time of day it is. They don't indicate where the server is, but it must be in my front yard. It's total BS. I've got a whole slew of screen shots on my iPad of all the TMN results over the past 7 days. The story is clear, so it's time to call them and confront them with this issue. During peak usage hours, I could achieve the same download speeds I'm getting with a 10 Mbps cable modem. During low usage hours with the Arris gateway they recently gave me, I get over 40 Mbps regularly to the Washington server here and about 30 Mbps to the Dallas server. So I know there's nothing wrong with any of the equipment at my house. I'll let you know what they say after I talk to them.

[Pgoodwin1]

Technical questions. I get the trace below (pretty much the same regardless of the time of day). I see it originates at a Dallas server. Is that the same server that's used when testing down and up speeds? i.e. Is the path the same for the speed tests as it is for the TeceMy.net trace?

The reason I'm asking, is that I'm assuming that the test is a short duration one during each hop compared to say a 25 MB download speed test. I'm just guessing here that the ~22 mSec avg per leg is telling me that the path hardware latency is OK and that my issues with low speeds during high usage hours of the day is due to the number of users doing things that last considerably longer than the TraceMy test events. Even during the worst high Time Warner usage hours I get virtually the same TraceMy results even though my download speeds are 8x slower than off-peak usage hours.

Just trying to get educated here on what the data is telling me. I did the trace below right after registering a horribly slow download speed of 8 Mbps versus a 30+ Mbps speed during off hours.

Also, would this trace be considered normal or reasonable for a 50 Mbps max download service?

Edited November 9, 2013 by Pgoodwin1

[CA3LE]

TraceMy isn't really doing a test, it's a logged traceroute... it's just conducted regularly.  TraceMy is activated as soon as you hit any page outside of the forum (currently tracing only registered users... only beta testers and a handful of others have been granted access to see the data).  It's currently only running on my main server but this will expand soon.  I'm working on to make this do much more... the reason I haven't fully released it is because, first, I'm not done.  Second, I'm still trying to understand what I've built.

 

So far what I'm gathering is that you're not looking for hops that just have long response here and there... you're looking for hops that continually have longer response times.  Especially should be noted if the hop responds that way more than three consecutive times.  There really is no frequency to testing.  You're tested as soon as the queue runs.  It queues over and over and is based on your activity on TestMy.net.  Every page load (again, outside of the forum) resets your time to 2 minutes, if you stop interacting with the site... it stops interacting with you.  For that two minutes you'll be traced over and over as you reach your place in line.  No set time, it just runs when it's available.  This is done server-side and all you need to do to activate it is open TestMy.net.  It's silent and you can surf the site as you normally would, that side of the program actually has nothing to do with what you see on your screen. (it's pretty nifty)

 

What I'm finding is that just because you're displaying horrible speeds... doesn't mean that it's going to show in the trace.  Also, just because a hop responds slow... doesn't mean that your speeds will be degraded.

 

Check this out... here's an example using my own connection.

 

 

I obviously have a fast connection at home, it runs awesome... all the time.  Now look at the trace results about 2 minutes before my 12:35p test.

 

(hop 9 for me is the end of trace by the way but is not my actual IP...)

 

Yeah, that had a long response... if there was any issue during that time it wasn't noticeable to me.  I wasn't testing at that exact moment so who knows, it may have resulted in a little bobble.  Obviously you'd like to see nice even response times but for a hop to respond like that here and there... no worries, it happens.  Some routers de-prioritize ICMP traffic so it's normal in some situations to see spikes here and there.  But if a hop is continually doing that, especially in succession... there is most likely something going on with the hop.

 

Before I release it publicly I'll do a full writeup and I'll cite logged examples displaying what the real problems look like.  Here's a little example I randomly pulled (username sistem) of what MAY be an issue.

 

 

See the top blue line, that's hop 7 of 8.  It often responds slower than the final hop, and does it consecutively.  It also seems that when that happens it sometimes causes the final hop to respond slower....

 

 

Now look at this members result around the same time... dropped to half of his average.

 

 

Going back further in his results I find that when he has a lower than average result... it's often accompanied by slower than average response on those hops.

 

 

 

This is not always the case.  But if you if you experience slower than average speeds then look to your TraceMy results it can help determine what part of the route is possibly at fault.

 

Now, looking at your TraceMy.net results... I don't see anything alarming or stand out.  Researching the times when you display slower results shows very normal traces.  Sure there are spikes, but they don't correlate with those times... and they don't consecutively respond bad.  Like I said, just because there's an issue with a hop/router doesn't mean that it's going to respond bad and vice versa.  But when you start to see regular patterns, it can be a good indication. 

 

... as soon as I understand what I've built better and can put it all into words I will.  I'll also tell the program to look for the known patterns I'm finding and alert the user to a possible issue.  I'm also developing more in depth analysis of the route... realize that I only have a couple of days of programming time invested in that so far.  Right now it's in a gathering phase so... again... I can understand what I've built.  

 

(gee, you think that's enough ellipsis for one post, lol... ... ...)

[Pgoodwin1]

Thanks for the response CA3LE. I understand more than I did. And I agree, I don't see anything that looks indicative of an issue when I look at the TraceMy plots. They're actually pretty consistent regardless of time of day. I have seen a few long ones every now and then but they were always unique events and never repeated. And I haven't seen one of those in quite a few weeks.

[HubCity]

snip...What's totally unbelievable is that TWC's speed test shows between 45 and 50 Mbps no matter what time of day it is. They don't indicate where the server is, but it must be in my front yard. It's total BS. I've got a whole slew of screen shots on my iPad of all the TMN results over the past 7 days. The story is clear, so it's time to call them and confront them with this issue. During peak usage hours, I could achieve the same download speeds I'm getting with a 10 Mbps cable modem. During low usage hours with the Arris gateway they recently gave me, I get over 40 Mbps regularly to the Washington server here and about 30 Mbps to the Dallas server. So I know there's nothing wrong with any of the equipment at my house. I'll let you know what they say after I talk to them.

 

Suddenlink tried to pull the same thing on me some months ago.  Their tester never showed any slowdown and tech support would not take any action because their tester was consistant speed test during peak evening hours.  My speed at the time would drop to less then 4 Mb/s, which was about 10% of my normal speed.  I used my packet sniffer built into my old Sygate firewall and monitored the trafic log to determine where the test packets were coming from.  Then using reverse lookup on the IPs, I determined they came from within Suddenlink's own backbone of servers and never once came from outside their backbone, so the test never showed any slowdown due to peak evening usage.  TMN was giving me the true story of the speed slowdown.  I presented this information to the maintenance superviser as normal tech support was a waste of time.  Within two weeks, my speed during those hours was back to normal as apparently they changed their routing, added more equipment, or fixed the problem.  Obviously, Suddenlink was the cause of the bottleneck.  There was alot more history to this story, but I kept it short.

 

 

Good luck...

[Pgoodwin1]

Well I talked to Time Warner today, and basically they blamed the internet. The guy tested using what he termed a 400 Mbps business class T1 to the Dallas Server and got 50 Mbps and claimed that his effective speeds were the same 1/8 to 1/10 that I was experiencing. He basically said that the usage load problem was the Internet and not TWC. And that without specific instances of websites loading slowly, he couldn't help me. He did some ping testing of what he claimed was checking his system. He claimed that with all the testing he did on his end during the 15 minute phone call, there were no issues showing.

He mentioned that 75 customers are on my local network system, but the system has a capacity of 400 customers.

I've analyzed the TestMy results over the past 30 days. The first two pictures show the change in speed versus hour of the day using data I exported from TestMy and plotted in Excel. The next pictures are the typical performance - taken today, I included a TraceMy result, and the last picture shows what the TWC Speed Test shows within seconds of completion of the results in the prior two pictures.

Note the extreme slowdown during peak usage hours on the Dallas server path results. The trend on the Washington server is similar but doesn't degrade nearly as much during peak usage hours

NOTE - At 2 AM, my TestMy speeds are very close to the 43Mbps showing in the TWC speed test result. The 8 Mbps results today at 4 PM or so are typical for that time of day.

Edited November 14, 2013 by Pgoodwin1

[Pgoodwin1]

So basically Time Warner is telling me that the Internet system of the United States between Cincinnati and Dallas is a lot less capable than the Internet system between Cincinnati and Washington DC.

I don't know what kind of data I need to compile to convince them that there is a problem, if indeed it is their problem. I asked if there was a way I could send them data or pictures of data, and they said no, that I could chat with an online rep, or call in like I did.

Not sure I have the technical skills and software to isolate the problem. But I'll take any suggestions.

[TriRan]

You need to somehow get to higher tier tech support