Some systems are experiencing issues

Stickied Incidents

12th September 2024

Web: Washington D.C. opal12 blocked by Malwarebytes

At this time opal12.opalstack.com (Washington DC shared hosting) is being blocked by Malwarebytes. The cause was an outbound SSH attack from a compromised customer shell account which caused the server to be listed in a major block list. As a result visitors who use Malwarebytes may see a warning when visiting sites hosted on opal12.

We're working with the block list operator to have the server delisted and expect to have this resolved within the next couple of days.

In the meantime, you may be able to work around the problem by adding your site domain and/or server IP 207.244.121.250 to your Malwarebytes exception list.

Past Incidents

2nd January 2023

No incidents reported

1st January 2023

No incidents reported

31st December 2022

No incidents reported

30th December 2022

No incidents reported

29th December 2022

DNS: USA Ongoing DNS DDOS Attack

Over the last few hours we've identified and mitigated an ongoing DDOS attack against our DNS infrastructure. This attack has also effected our Singapore and German DNS servers.

We're continuing to monitor the attack.

  • Post-mortem: this was a DNS flood attack which targeted all 9 of our regional name servers. The attack originated from several cloud provider subnets, including Google and Cloudflare.

    To mitigate the attack we had to temporarily block the provider subnets. This had some unintended and undesirable side effects:

    • The subnets included public DNS resolver infrastructure run by Google and Cloudflare, so blocking them prevented those resolvers from looking up DNS records on Opalstack name servers. As a result, people who use those resolvers were temporarily unable to access opalstack.com (including this status page) and customer domains that use our name servers.
    • The block also prevented Google from looking up DNS records related to email such as SPF and DKIM. As a result, mail sent (or forwarded) to Gmail from Opalstack's mail system was rejected by Google while the block was in place.

    The block was lifted over 24 hours ago and there have been no further issues since that time.

    Going forward, we'll refine our mitigation techniques to avoid blocking major providers when possible. We'll also move this status page to an externally-hosted domain to ensure that system status updates will be available regardless of the state of our infrastructure.

  • We've seen no further issues in the past several hours and consider this to be resolved.

  • The attack has subsided, we'll continue to monitor.

    • DNS: Singapore Delayed domain updates

    At this time (Thu Dec 29 02:17:55 UTC 2022) there is a backlog of domain updates which are waiting to be deployed to our name servers. Customers who are affected by this will see domains listed as "Pending" in their Opalstack dashboards.

    The actual processing delay is on our Singapore name servers. If your domain is using the US or DE name servers then they'll still appear as pending but the domains will actually be updated on the NS. Only domains that use the Singapore servers will actually be affected, and it's only a small number at this time.

    The cause of the issue appears to be network-related so we're working with our upstream providers to resolve this.

    We'll update this post when more information is available.

  • The backlog of domain updates is now clear.

    • 28th December 2022

    No incidents reported

    27th December 2022

    No incidents reported