There have been no further problems since the last update.
As of 2020-01-12 22:20 UTC all Opal3 shell user home directories, PostgreSQL databases, and related control panel data have been restored.
All customer sites are back online and the server is operating normally at this time. We'll continue to monitor.
We were not able to recover shell user passwords, so Opal3 customers will need to reset their shell user passwords via the control panel. Instructions are available in our documentation: Changing a Shell User's Password
At approximately 17:40 UTC (9:40 AM Pacific) on 12 January 2020 we ran a live test of our upcoming migration tool on opal3, our shared hosting server in San Francisco.
A serious bug in the migration tool resulted in the loss of all customer home directories and PostgreSQL databases on opal3.
We're presently restoring opal3 from the most recent backup which was completed at 00:14 AM, about 7.5 hours prior to the incident.
We'll update this incident with more information when the restoration is complete.
