October 21, 2016: DDOS attack

Earlier this morning (October 21, 2016), many major websites and companies were affected by a Distributed Denial Of Service (DDoS) attack. This attack was focused on Dyn, a major DNS provider. xMatters is among those companies impacted by the attack.

What happened?

On October 21, 2106, Dyn reported that they were dealing with a DDoS attack, beginning at approximately 4:10am PDT (11:10 UTC) and that it was believed to be resolved after a little more than two hours.

The attack resumed at 8:52am PDT, and has continued throughout the day. 

What is happening now?

Since approximately 8:52am, based on xMatters observations, there continue to be widespread, complete failures across different regions of the Internet. This continued to impact xMatters services and any other company using Dyn services.

Due to the way that DNS is designed, access to affected web sites, and the extent of the impact, will be highly inconsistent, even within regions. Many DNS services, including Dyn, are highly available and massively distributed. This often prevents total failure, but the possibility for inconsistent and localized behaviors exists - especially in DDoS attacks.

• Updated 1:50pm: At this time, Dyn and other affected services are reporting that the attack has ceased, but there continue to be some conflicting reports.
• Updated 3:23pm: Dyn has posted a statement indicating that the attack and resulting issues have been resolved.

What are we doing to resolve it?

Earlier this morning, when the extent of the attack became apparent, xMatters switched to an alternate DNS provider to mitigate the problems and work around the issues experienced by Dyn. Due to the nature of some DNS changes, it may take up to 48 hours for this change to be propagated throughout the Internet. xMatters has taken steps to reduce this time wherever possible.

If you or your customers are running and hosting DNS internally, you should be able to restore service by flushing the cache on the DNS servers hosted by the company, which may have cached entries for the name server records used by xMatters..

If Dyn is able to resolve the problem and halt the attack, service will be restored instantly and automatically. If they are unable to resolve the problems, you can expect delays to continue for up to 48 hours.

How will we prevent this in the future?

Internet security and reliability are separate but interconnected issues that are of utmost importance to xMatters and our customers. This attack has affected many of our clients, competitors, providers, and partners, and will be investigated thoroughly over the coming days and weeks.

While these sorts of attacks are difficult to predict and prevent, xMatters will be researching various ways to diversify our DNS providers and to maintain alternate solutions. More details will be available after an upcoming post mortem and root cause analysis.

Note: Any customers or other users who implemented the workaround published earlier today should now remove the entry from their hosts file.