All systems are operational

Past Incidents

25th October 2020

Email: USA US SMTP blocked by outlook.com

Our US SMTP server smtp.us.opalstack.com is currently on a Microsoft block list used by the following domains and their non-US equivalents:

  • hotmail.com
  • live.com
  • msn.com
  • outlook.com

We're working to get the server delisted at this time.

  • Microsoft is no longer blocking our US SMTP.

  • Microsoft has responded saying that our US SMTP will be removed from the blacklist within the next 24-48 hours.

  • Microsoft have responded to our inquiry. It appears that the recent influx of WebFaction customers 🎉 has caused a sudden increase our outgoing mail volume and that this is what triggered the block.

    We've verified via Microsoft's own postmaster tools that there have been zero complaints regarding mail from our US SMTP and have provided this data to Microsoft for consideration.

    We hope to follow up with a resolution to this issue as soon as possible. Thank you for your patience.

  • 24th October 2020

    No incidents reported

    23rd October 2020

    No incidents reported

    22nd October 2020

    No incidents reported

    21st October 2020

    Web: multiple locations Let's Encrypt rate limit for opalstacked.com subdomains

    At this time the Let's Encrypt service is rate-limiting certificate issue and renewal requests for the entire *.opalstacked.com domain. This is the domain that we use to provide free subdomains to our hosting customers.

    We're working with the LE team to get a higher limit. In the meantime if you are attempting to issue a LE cert for a site that uses your opalstacked.com subdomain then it may fail. If this happens you'll receive a failure notice via email.

    Until the rate limit is raised then the alternative is to use one of your own registered domains on your site instead of your opalstacked.com subdomain. If you're not ready to point your domain at Opalstack, then you can create a testing subdomain and A record via your NS provider to point that subdomain at your Opalstack server.

    We'll update this issue as soon as the rate limit has been raised.

  • We've made some changes to the way Let's Encrypt works with the shared opalstacked.com domain and as a result the domain is no longer rate-limited. We'll be posting updated documentation with the details later this week.

  • Looks like we've hit our new limit, we'll get this corrected ASAP.

  • We've confirmed that our increased rate limits are in effect, Calling this fixed, thanks for your patience :-)

  • The LE service has increased our limit and at this time we are not seeing any rate-limiting for new LE certificates and renewals on any domains. We'll continue to monitor.

  • 20th October 2020

    No incidents reported

    19th October 2020

    Web: Dallas Opal4 offline

    Opal4 (Dallas shared hosting) is currently not responding to ping. We're looking into it at this time.

  • The issue was caused by a spike in system load due to temporary high network usage. The server is now stable.

  • 18th October 2020

    Web: multiple locations Intermittent PHP-FPM failures on Opal4 and Opal5

    We've been investigating intermittent failures of PHP sites hosted on Opal4 (Dallas) and Opal5 (Frankfurt) which result in "503 Service Unavailable" responses on websites. We'll post more details here as they become available.

  • We've rebuilt most of our PHP stack and this issue is now resolved. Updated documentation and maybe a blog post are on the way.

  • Over the last few weeks we’ve dealt with various PHP-FPM related problems. After trying various combinations of fixes to the server, apache, and PHP-FPM itself we’ve developed a new, more stable PHP-CGI stack to replace PHP-FPM. With PHP and WordPress powering large parts of the Internet (and businesses) having a stable, reliable stack for them is critical.

    We rolled the new stack out to all servers yesterday afternoon and found some edge cases we hadn’t found in our previous testing.

    We’ll be rolling those changes out again tonight. While we do there may be a brief downtime of less than 10 minutes across PHP based sites and the update happens. Once the sites are updated we’ll begin monitoring for any breakages that didn’t show up in our testing and fix them ASAP.

    We’ll follow up with a blog post on exactly what we’ve changed and how things are working in the future.

  • Our troubleshooting is going but we've improved the stability of PHP-FPM quite a bit over the past few days and are seeing no customer facing problems at this time. We'll leave this incident open until we're satisfied that we've got this issue nailed down.

  • There have been intermittent PHP-FPM outages on Opal5 in the past several hours. Troubleshooting is ongoing.

  • PHP-FPM for PHP 7.3 on Opal5 was briefly disrupted about 15 minutes ago. We'll continue to troubleshoot and monitor.

  • We've just deployed a potential fix and will continue to monitor.