All systems are operational

Stickied Incidents

9th March 2025

Web: Dallas OPAL4 MariaDB Outage

MariaDB on OPAL4 has crashed, we're working to identify the cause and solution now.

  • We've seen no further problems since our last update.

  • The issue with database permissions is resolved and the MariaDB service is back online and operating normally. We will continue to monitor.

  • We've discovered that there is still a problem with database permissions. We're working to resolve that at this time.

  • The data restoration has completed and we are continuing to monitor the server as usage increases.

  • The restoration is approximately 75% complete.

  • The data restoration is approximately 50% complete.

  • The MariaDB restore process is still running.

  • The database corruption is unrecoverable so we will proceed with restoring from backups. The most recent valid backups were from 1.5 hours before the server crashed so we'll proceed with the restore from there.

    The restoration process varies in completion time. It may take up to 6 hours or more. We'll keep this post updated.

    • Past Incidents

    1st October 2024

    Email: USA US SMTP blocked by Microsoft

    At this time messages sent from the US SMTP server to Microsoft email domains like hotmail.com, live.com, outlook.com, and their international variants are being blocked by Microsoft.

    The cause of the block has been identified and corrected and we are now waiting for Microsoft to remove the block.

    We'll update this post as soon as the block has been lifted or when there is other information to report.

  • Microsoft has lifted the block and their servers are now accepting mail from the US SMTP.

    • 30th September 2024

    No incidents reported

    29th September 2024

    Email: Germany DE mail SSL certificates were briefly expired

    Due to an unfortunate oversight on our end the SSL certificates for our DE mail servers expired earlier today at around 09:30 UTC, causing certificate warnings to appear in email client software using DE mailboxes and in our DE webmail system.

    Renewed certificates were rolled out approximately 1 hour later and there have been no issues since that time (~10:30 UTC). If you continue to see a certificate warning when accessing DE email systems then please restart your software and/or your device.

    28th September 2024

    No incidents reported

    27th September 2024

    No incidents reported

    26th September 2024

    No incidents reported

    25th September 2024

    No incidents reported