All systems are operational

Stickied Incidents

9th March 2025

Web: Dallas OPAL4 MariaDB Outage

MariaDB on OPAL4 has crashed, we're working to identify the cause and solution now.

  • We've seen no further problems since our last update.

  • The issue with database permissions is resolved and the MariaDB service is back online and operating normally. We will continue to monitor.

  • We've discovered that there is still a problem with database permissions. We're working to resolve that at this time.

  • The data restoration has completed and we are continuing to monitor the server as usage increases.

  • The restoration is approximately 75% complete.

  • The data restoration is approximately 50% complete.

  • The MariaDB restore process is still running.

  • The database corruption is unrecoverable so we will proceed with restoring from backups. The most recent valid backups were from 1.5 hours before the server crashed so we'll proceed with the restore from there.

    The restoration process varies in completion time. It may take up to 6 hours or more. We'll keep this post updated.

  • Past Incidents

    17th September 2024

    No incidents reported

    16th September 2024

    No incidents reported

    15th September 2024

    No incidents reported

    14th September 2024

    Web: Washington D.C. Intermittent network connectivity to opal12

    At this time network connectivity to opal12.opalstack.com is intermittent and customer sites may be inaccessible. We're investigating at this time and will update when more information is available.

  • We've seen no further issues.

  • We've identified and blocked an abusive IP address that was saturating the network. Connectivity is stable at this time. We will continue to monitor.

  • Connectivity is failing again, we'll continue to troubleshoot.

  • opal12 is back online at this time. We suspect that a network issue upstream in the data center was the cause. We're awaiting confirmation and will continue to monitor.

  • 13th September 2024

    No incidents reported

    12th September 2024

    Web: Washington D.C. opal12 blocked by Malwarebytes

    At this time opal12.opalstack.com (Washington DC shared hosting) is being blocked by Malwarebytes. The cause was an outbound SSH attack from a compromised customer shell account which caused the server to be listed in a major block list. As a result visitors who use Malwarebytes may see a warning when visiting sites hosted on opal12.

    We're working with the block list operator to have the server delisted and expect to have this resolved within the next couple of days.

    In the meantime, you may be able to work around the problem by adding your site domain and/or server IP 207.244.121.250 to your Malwarebytes exception list.

  • opal12.opalstack.com is no longer blocked by Malwarebytes.

  • 11th September 2024

    No incidents reported