All systems are operational

Past Incidents

15th January 2020

No incidents reported

14th January 2020

No incidents reported

13th January 2020

No incidents reported

12th January 2020

Web: San Francisco Opal3 offline

Opal3 was offline.

  • There have been no further problems since the last update.

  • As of 2020-01-12 22:20 UTC all Opal3 shell user home directories, PostgreSQL databases, and related control panel data have been restored.

    All customer sites are back online and the server is operating normally at this time. We'll continue to monitor.

    We were not able to recover shell user passwords, so Opal3 customers will need to reset their shell user passwords via the control panel. Instructions are available in our documentation: Changing a Shell User's Password

  • At approximately 17:40 UTC (9:40 AM Pacific) on 12 January 2020 we ran a live test of our upcoming migration tool on opal3, our shared hosting server in San Francisco.

    A serious bug in the migration tool resulted in the loss of all customer home directories and PostgreSQL databases on opal3.

    We're presently restoring opal3 from the most recent backup which was completed at 00:14 AM, about 7.5 hours prior to the incident.

    We'll update this incident with more information when the restoration is complete.

  • 11th January 2020

    No incidents reported

    10th January 2020

    No incidents reported

    9th January 2020

    No incidents reported

    8th January 2020

    Web: Dallas Intermittent outages on opal4

    opal4.opalstack.com is experiencing intermittent outages. We're looking into it and will update this item when we have more information.

  • During last week's maintenance our upstream provider updated the server firmware on opal4. We've seen no further issues since that time.

  • The maintenance is complete and Opal4 is back online. We'll continue to monitor.

  • Opal4 is down for the emergency maintenance scheduled earlier today. The total expected downtime is less than one hour.

  • Opal4 will be going down this evening at 9PM US Central (2020-04-01 21:00 UTC-5) for emergency maintenance. The total downtime should be less than one hour.

    The ongoing issue on opal4 has been high system load caused by two separate types of attacks:

    1. High Apache RAM usage due to attacks against common targets like Wordpress sites
    2. High CPU usage by the system firewall under a high volume of SYN packets

    We've resolved the first problem by tuning our application firewall rules and by working with a couple of specific site owners that were receiving the brunt of the attacks.

    The second problem is part of what we're troubleshooting this evening.

    At first glance the SYN packet issue would seem like a common SYN flood attack but other shared servers in our infrastructure receive similar amounts of traffic and don't have any problems mitigating it.

    In the past 24 hours we've discovered key differences in Opal4's hardware compared to the other servers, so we're working with our upstream provider to sort that out.

    Tonight's downtime is at their request to allow them to perform hardware diagnostics in support of that investigation.

  • Opal4 just went through a brief spike in system load during which performance was degraded, but the system is back to normal at this time.

  • We've identified two distinct attack patterns responsible for the intermittent outages on opal4 and are putting measures in place to mitigate them.

  • This intermittent issue on Opal4 has started up again - we will continue to monitor as we work to resolve it.

  • We've seen no further issues in the past several hours.

  • The problem appears to have been caused by temporary high load due to high CPU usage by the system firewall.

    opal4 is stable at this time. We'll continue to monitor.