All systems are operational

Stickied Incidents

9th March 2025

Web: Dallas OPAL4 MariaDB Outage

MariaDB on OPAL4 has crashed, we're working to identify the cause and solution now.

  • We've seen no further problems since our last update.

  • The issue with database permissions is resolved and the MariaDB service is back online and operating normally. We will continue to monitor.

  • We've discovered that there is still a problem with database permissions. We're working to resolve that at this time.

  • The data restoration has completed and we are continuing to monitor the server as usage increases.

  • The restoration is approximately 75% complete.

  • The data restoration is approximately 50% complete.

  • The MariaDB restore process is still running.

  • The database corruption is unrecoverable so we will proceed with restoring from backups. The most recent valid backups were from 1.5 hours before the server crashed so we'll proceed with the restore from there.

    The restoration process varies in completion time. It may take up to 6 hours or more. We'll keep this post updated.

    • Past Incidents

    4th June 2024

    No incidents reported

    3rd June 2024

    No incidents reported

    2nd June 2024

    No incidents reported

    1st June 2024

    No incidents reported

    31st May 2024

    DNS outage (resolved)

    About 1 hour ago a failed DNS sync with our alternate DNS provider caused an outage in our DNS servers. The outage lasted approximately 1 hour during which customers may have experience trouble accessing Opalstack systems.

    The problem has been resolved and all DNS services are working normally at this time, however there may be lingering effects due to global DNS propagation. These issues will resolve themselves over the next few hours. You may be able to resolve them sooner for yourself by flushing your DNS and/or by power-cycling your devices.

    30th May 2024

    Web: San Francisco Intermittent high load on opal14

    At this time we are tracking an high load condition on opal14.opalstack.com (San Francisco shared hosting) which has been causing the server to become unresponsive on an intermittent basis. The problem appears to be caused by high memory usage by the system Apache service.

    We're working to identify the customer site that is causing the high memory usage and will work with them to resolve the issue once we've identified it.

  • We believe we identified the customer site that has been causing these issues and have taken action to mitigate it. We will continue to monitor a couple more days.

    • 29th May 2024

    No incidents reported