Agreed. For the most part, the only problems I've ever seen with my connection were due to some sort of poor transit between some connection in my ISP's network to some connection who knows where. What does bother me however is, as you've mentioned, the networks were designed so long ago to where you've got many redundant systems tying into a single failure point, or many going into few which causes issues such as slow speeds, or, which Verizon is currently overdue for this year, routing outages in NYC that take out 95% of my Internet connectivity completely but do not destroy speeds. If it's not that, you've also got plenty of older systems that just have not been invested in and are having to handle loads that they can take, but don't allow for smooth failover. Or for that matter, old systems that choke down at night because they were set up with too much oversubscription (such as my local Cable Company's network/backbone).
Then of course, there are providers who just don't plan their networks for double growth (Moore's Law, in perspective) or upgrade them fast enough for whatever reason. I know my second ISP, Frontier Communnications is one of these. Their services are known to slow down at night due to lack of transit capacity or overloaded routers during the night time hours. They're starting to fix all of that up now that they're *FINALLY* bringing out the bigger guns. My Frontier connection before the days of streaming and YouTube in HD was pretty solid. Come all of the big streaming services like Netflix, and higher quality video, I started seeing speeds dropping from 3Mbps down to 1Mbps at night, perhaps lower depending on the day as the edge router at the ISP got overloaded, the old SONET rings not having been upgraded, and so on.
I do know that part of the problem here is with an ISP saying they don't guarantee speeds (and none can), but if it's something that happens routinely according to events, or time of day it's something they have to work to fix up.
Upgrades have been getting done though, which is fortunately something that is being done here in the East. The problem is with old last mile infrustructure and underutilized long haul cable, and of course poor routing to other areas.