Getting a handle on event floods

Event flood, alert storm, traffic surge, message torrent, memo tsunami - whatever you call it, it has happened or will happen to everyone in communications at some point. Sometimes, it just means your email inbox blows up and you have to keep hitting Delete until it all settles down. Other times, it means your phone is pinging madly with message after message about the same problem you're already trying to solve, causing distractions and delays.

The solution

Fortunately, the event flood control feature in xMatters reduces the number of events that are generated when you experience an event flood from one of your systems. The feature compares incoming event requests to recent events and suppresses correlated events that occur in too close succession to one another.

These suppressed events aren't queued for processing and don't count against your licensed number of events per minute. xMatters also sends you a notification when the suppression kicks in, and logs the details of the flood on the Events report so you can track how many events were suppressed and why.

Putting it into practice

The initial release of this feature included a company-level default Event Rate Filter for all inbound integrations, based on our analysis of real event flood data and actual customer experience, and designed to balance effective protection against event floods without impeding normal traffic. Customers were invited to opt-in to this default rule, which would automatically suppress incoming events from the same integration that targeted the same recipients if more than four events were injected within 60 seconds.

This new rule proved so successful among the customers who opted in that we're planning on implementing it on a much broader basis in our upcoming Defender release.

Beginning with the production deployment of the Defender release on February 12, 2019, the Event Rate Filter rule will be automatically applied to all new and existing inbound integrations.

This change will coincide with the release of the new Event Flood Control page in the xMatters web user interface (as described in the Defender Release Overview), which provides administrative control over the event flood control rules in your system. If you don't want to have event flood control active for an event source - for example, one you're using for testing - you can disable the rule individually for each communication plan.

Making it even better

While we think this default rule is the best way to protect your team and system from the stress of an event flood, we understand that not everyone will find it equally effective for their particular deployment. That's why we're working hard on coming up with a way for you to design and create your own flood control rules. Keep an eye on the Enduro Development Highlights if you'd like to track our progress!

If you have any questions about the implementation of the Event Rate Filter in your deployment, please contact xMatters Customer Support.