Post-mortem: this was a DNS flood attack which targeted all 9 of our regional name servers. The attack originated from several cloud provider subnets, including Google and Cloudflare.
To mitigate the attack we had to temporarily block the provider subnets. This had some unintended and undesirable side effects:
- The subnets included public DNS resolver infrastructure run by Google and Cloudflare, so blocking them prevented those resolvers from looking up DNS records on Opalstack name servers. As a result, people who use those resolvers were temporarily unable to access opalstack.com (including this status page) and customer domains that use our name servers.
- The block also prevented Google from looking up DNS records related to email such as SPF and DKIM. As a result, mail sent (or forwarded) to Gmail from Opalstack's mail system was rejected by Google while the block was in place.
The block was lifted over 24 hours ago and there have been no further issues since that time.
Going forward, we'll refine our mitigation techniques to avoid blocking major providers when possible. We'll also move this status page to an externally-hosted domain to ensure that system status updates will be available regardless of the state of our infrastructure.
