I have the same experience with 4G (cellular LTE) based broadband connections, such as when positioning a directional antenna. When the network is quiet (e.g. early on a weekend morning), there can be a large variation between what SpeedTest and TestMy reports if the antenna is not aimed correctly. Once the antenna is carefully aimed, the TestMy results climb up towards what Speedtest reports. It's similar also if there are swaying branches in line of sight as Speedtest will again ignore the brief dips as if the bandwidth is sustained.
If Speedtest measured road trip speeds, their speed test methodology would eliminate traffic lights, construction zones, slow vehicles, busy junctions and everything else that accounted for the slowest 30% of the journey.
If TestMy measured road trips, it would run a stopwatch from the moment of departure to the moment of arrival.
Of course like the Speedtest fanatics, there would be those that would argue the same for road trip measurements - "What if that construction zone was not there, those traffic lights were green, no accident on the route, ..."