Cryian

There was a reference to the wrong database server address.  Appeared on my end to be working because I hadn't selected one of those test types with predictive algorithm.  The regular tests I was selecting don't reach out to the database.
 
Thank you for pointing that out.  It just needed to be pointed to the correct address.
 
You're right, it's not rocket science.  It's computer science...
 
I'll add this to my monitoring (testing those database driven links every 1 minute) so if I make that mistake again I'll be alerted right away.  I thought I had manually checked for this but I was checking the wrong URLs.  I changed the reference during a smaller crash (that I caused on the backend by stopping a repair process)... it only went down for a few seconds and I needed to then change the IP back to the correct IP.
 
It takes a lot of time to reorganize TMN's databases when a crash this major happens.  Hundreds of millions of entries across hundreds of thousands of tables across dozens of databases.  It's not a simple process like most other websites may deal with... there is A TON of data and there are restrictions on how fast it can be imported.  I was pretty prepared for this incident.  Had a backup server on standby, had a new server built and on standby... restore procedure in writing... and it still took forever.  There are always unforeseen issues. 
 
Thank you for reporting this, please let me know if you see anything else that needs attention.