Germany mail server offline. Friday 1st April 2022 08:30:00

At around 08:30 UTC on 1 April 2022 our DE incoming mail server stopped responding, so we initiated an emergency reboot.

The server did not reboot correctly and further analysis identified a failing disk associated with outdated disk controller firmware and a failed disk controller battery.

As of 11:00 UTC the disc controller was repaired and a filesystem check was initiated.

Further updates follow in reverse-chronological order.

All repairs are complete and all DE mailboxes have been restored.

If you are a DE mail customer and are still seeing issues with your IMAP/POP3 email then please email Opalstack support for further assistance.

Posted 2 years ago by sean

The repairs for message dates and mailbox permissions are 75-80% complete.

Posted 2 years ago by sean

The repairs for dates and mailbox permissions are still in progress. We expect them to be completed today.

Posted 2 years ago by sean

The restore is complete.

We're still working on lingering issues related to message dates and mailbox permissions which may take another day or two to fully resolve.

We'll continue to post updates until the issues are resolved.

Posted 2 years ago by sean

The restore is 99% complete.

Posted 2 years ago by sean

The restore is 99% complete.

Once the restore has completed, there may be sporadic service disruptions for individual mailboxes as we identify lingering issues and correct them.

Posted 2 years ago by sean

The restore is 98-99% complete.

Posted 2 years ago by sean

The restore is 93-99% complete.

Posted 2 years ago by sean

The recovery is 85-95% complete.

Posted 2 years ago by sean

The recovery is 80-90% complete.

Posted 2 years ago by sean

The recovery is 75-85% complete.

Posted 2 years ago by sean

The recovery is 70-80% complete.

Posted 2 years ago by sean

The recovery is 65-75% complete. Note that there are several parallel restore processes so the overall recovery progress doesn't reflect a percentage of the alphabet with regard to mailbox names.

Posted 2 years ago by sean

The recovery is 60-70% complete.

Posted 2 years ago by sean

The recovery is 45-60% complete.

Posted 2 years ago by sean

The recovery is 30-40% complete. We have to estimate with a range because of the parallel restore processes.

Posted 2 years ago by sean

The recovery is 24% complete.

Posted 2 years ago by sean

The recovery from backup is still in progress, albeit running slowly at only 12% complete.

We believe that we'll be able to recover most of the data from the failed server, so we're restoring that data in parallel with the backup recovery. This should speed up the overall recovery and provide the most recent data from prior to the failure.

Posted 2 years ago by sean

The recovery is in still progress.

We estimate that it is 11% complete and it is currently restoring mailboxes that being with "as".

Posted 2 years ago by sean

DE webmail access is now working.

The mailbox restore is still in progress.

Posted 2 years ago by sean

We're still noticing some login issues for webmail and are troubleshooting that now.

Posted 2 years ago by sean

We do not have high expectations for the outcome of the filesystem check, so we've deployed a new DE IMAP server and have re-created all DE mailboxes as new empty mailboxes on that server.

Customers should now be able to log in to our DE mail services to send and receive new mail. No changes to your email client settings should be required.

We'll begin restoring the original mailbox contents ASAP from the most recent backup made on 30 March 2022 (the failure occurred while the 31 March backup was still in progress). The restore may take several days to complete, so we'll keep this post updated until it is done.

We'll attempt to recover mail from 31 March 2022 when the filesystem repairs on the failed server are complete.

Posted 2 years ago by sean