Today’s AWS outage highlights a critical question for all organizations: what’s your backup plan when your regular incident management tools are themselves experiencing disruptions? Slack, Zoom, PagerDuty, incident.io, FireHydrant, Rootly, and Atlassian (Jira, Opsgenie, Jira Service Management) and many others are currently reporting various issues due to the outage.
A big challenge during incidents like this is re-establishing communication among responders. If your usual platforms like Slack and Zoom are down, can you reliably transition to alternatives like Google Meet or Microsoft Teams? The key is to have a pre-defined fallback communication plan that’s known to everyone on the team, not just tucked away in an audit document that nobody remembers exists.
This plan needs to specify the exact meeting ID, channel, or other critical details for reconvening. This information must be accessible even when your primary communication channels and document stores are unavailable. Consider distributing laminated wallet cards with essential phone numbers and web URLs to key staff, or storing this information in a shared password vault that offers offline access. Alternatively, your IT department could ensure this crucial data is kept up-to-date in a file on every employee’s laptop.
Security is important. You have to assume that, over time, this fallback information might become accessible to unauthorized parties (press, competitors, hackers, etc.). Therefore, when using these reassembly mechanisms, avoid discussing highly confidential or sensitive matters in the main fallback channel. Instead, treat the fallback channel just as a rendezvous point, and use it to direct responders to move to specific secure channels where all participants can be carefully identified and authenticated.
These fallback mechanisms need to be exercised periodically, both to ensure that folks remember that they exist and to verify that the specific mechanisms still work as intended.
—
Incidents can be incredibly costly, impacting revenue, sales, reputation, employee productivity, morale, and turnover. Proactive preparation and continuous learning from every incident are vital. As an expert incident management consultant, advisor, and coach, I can help your organization develop these essential skills and prevent costly problems. Contact me today to discuss how I can support your team.
Recent Comments