All systems are operational

Stickied Incidents

9th March 2025

Web: Dallas OPAL4 MariaDB Outage

MariaDB on OPAL4 has crashed, we're working to identify the cause and solution now.

  • We've seen no further problems since our last update.

  • The issue with database permissions is resolved and the MariaDB service is back online and operating normally. We will continue to monitor.

  • We've discovered that there is still a problem with database permissions. We're working to resolve that at this time.

  • The data restoration has completed and we are continuing to monitor the server as usage increases.

  • The restoration is approximately 75% complete.

  • The data restoration is approximately 50% complete.

  • The MariaDB restore process is still running.

  • The database corruption is unrecoverable so we will proceed with restoring from backups. The most recent valid backups were from 1.5 hours before the server crashed so we'll proceed with the restore from there.

    The restoration process varies in completion time. It may take up to 6 hours or more. We'll keep this post updated.

  • Past Incidents

    19th February 2024

    Delays in mail forwarding to Google

    Google had rate-limited 2 of our forwarding MX servers over the weekend resulting in forwarding delays for some users who forward mail to Gmail.

    We've corrected the cause of the rate limit and expect all pending mail to be delivered over the next several hours.

  • All of the pending forwarded mail has been delivered.

  • 18th February 2024

    No incidents reported

    17th February 2024

    No incidents reported

    16th February 2024

    No incidents reported

    15th February 2024

    Email: Germany EU mail system blocked by Hotmail

    Part of our EU mail infrastructure was blocked by Hotmail. The affected servers were:

    • smtp2.de.opalstack.com (outgoing mail)
    • mx1.de.opalstack.com (incoming mail and forwarding)

    We've already dealt with the issue and Microsoft have confirmed that the servers have been unblocked, but it may take a few more hours to take full effect.

    We'll update this post when we've confirmed that mail is no longer being blocked.

  • All blocks have been lifted.

  • mx2.de.opalstack.com was added to the block list as well, we have submitted the delisting request and are awaiting action from Microsoft.

  • 14th February 2024

    No incidents reported

    13th February 2024

    No incidents reported