Most likely this is Internet 'rush hour' congestion. It may only affect a portion of the sites you visit because to get out to different sites your host will route through different peers. Whenever an issue happens on a scheduled regularity and those times coincide with the time when most people get off work and hop online, chances are it's congestion. People dealing with this issue can pretty much place bets on when it will slow down every day.
Oversold intermediate peers are often to blame but this can sometimes be a temporary issue. Sometimes attacks across a host will force them to reroute more traffic than they normally would through alternate routes. Sometimes it's too much... it's like hooking a garden hose up to a fire hydrant. Only so much can get through, resulting in a trickle for everyone routing through the affected pipe. Hours when less people are online it's less apparent or a non-issue because those routes aren't hitting the critical point.
Keep a list of slow site addresses and email them to the support@ address for your provider. They most likely already know about the issue but in case they don't this can help them track down the problem route/peer.
Another thing to look into, less likely. Do you have any computers on the network that might be doing a scheduled task during that time. I've had visitors who wasted a bunch of time troubleshooting only to remember, "oh yeah, my other computer is set do something at the same time every day and it's congesting my internal network." ... or "it was my roommate, running tons of torrents.." ... make sure those variables don't exist before you flip out on your ISP.
If you resolve the problem please update this thread to help others.
