Past Incidents

We found and corrected a problem with database permissions and all database users should be able to connect now.

We're getting reports that some users still cannot connect to their databases so we are investigating that now.

Databases have been restored and should be fully operational.

The restore process is at 99% and the last databases are being restored.

The database restores are 66% complete.

The database restores are 51% complete.

The database restores are in progress and are ~25% complete.

The InnoDB data corruption is too bad to recover data from the database server. Tables are unreadable causing data dumps or repairs to the tables to be impossible.

We have started the database repair process from our backups that were 3-5 hours old when the server went offline. We will update this post with our progress.

The data corruption on the server is in the innodb data which means there is potentially corrupted data across multiple databases due to how MySQL/MariaDB store data. We are still trying to recover and repair the data in the MariaDB server. If the data is too corrupt to repair we will restore the data from the latest backups. The latest backups were taken between 3 and 5 hours (depending on the dump time) before the database server crashed.