All systems are operational

Past Incidents

4th January 2023

No incidents reported

3rd January 2023

No incidents reported

2nd January 2023

No incidents reported

1st January 2023

No incidents reported

31st December 2022

No incidents reported

30th December 2022

No incidents reported

29th December 2022

DNS: USA Ongoing DNS DDOS Attack

Over the last few hours we've identified and mitigated an ongoing DDOS attack against our DNS infrastructure. This attack has also effected our Singapore and German DNS servers.

We're continuing to monitor the attack.

  • Post-mortem: this was a DNS flood attack which targeted all 9 of our regional name servers. The attack originated from several cloud provider subnets, including Google and Cloudflare.

    To mitigate the attack we had to temporarily block the provider subnets. This had some unintended and undesirable side effects:

    • The subnets included public DNS resolver infrastructure run by Google and Cloudflare, so blocking them prevented those resolvers from looking up DNS records on Opalstack name servers. As a result, people who use those resolvers were temporarily unable to access opalstack.com (including this status page) and customer domains that use our name servers.
    • The block also prevented Google from looking up DNS records related to email such as SPF and DKIM. As a result, mail sent (or forwarded) to Gmail from Opalstack's mail system was rejected by Google while the block was in place.

    The block was lifted over 24 hours ago and there have been no further issues since that time.

    Going forward, we'll refine our mitigation techniques to avoid blocking major providers when possible. We'll also move this status page to an externally-hosted domain to ensure that system status updates will be available regardless of the state of our infrastructure.

  • We've seen no further issues in the past several hours and consider this to be resolved.

  • The attack has subsided, we'll continue to monitor.

  • DNS: Singapore Delayed domain updates

    At this time (Thu Dec 29 02:17:55 UTC 2022) there is a backlog of domain updates which are waiting to be deployed to our name servers. Customers who are affected by this will see domains listed as "Pending" in their Opalstack dashboards.

    The actual processing delay is on our Singapore name servers. If your domain is using the US or DE name servers then they'll still appear as pending but the domains will actually be updated on the NS. Only domains that use the Singapore servers will actually be affected, and it's only a small number at this time.

    The cause of the issue appears to be network-related so we're working with our upstream providers to resolve this.

    We'll update this post when more information is available.

  • The backlog of domain updates is now clear.