04 · Macquarie Technology
A fifth of the data centre
A hardware fault hit about 20% of data-centre traffic, roughly 120 customers. Back online in about 90 minutes.
~90 minBack online
A hardware fault inside the data centre took roughly a fifth of the traffic with it, on the order of one hundred and twenty customers. The failure was physical, which makes the restore path clearer than a software mystery and the customer impact no smaller. People do not experience a failed component. They experience a service that has stopped.
I declared it and built one bridge: data-centre operations, network, the vendor who owned the hardware, and the person who would speak to customers. The first decision was to contain. If the design allowed traffic to move off the failed hardware, that is what we did. Debugging a dead box while customers are down is a laboratory exercise, not a restore.
Customer updates carried a number and a next time. ‘We are investigating’ is not an update a customer can use. We were back online in about ninety minutes, and online meant the traffic share had returned and a sample of customer flows worked. A cabinet with a healthy light and traffic still black-holed is not restored.
The review asked a harder question than what broke. It asked why one piece of hardware had been allowed to represent twenty percent of the traffic, when the failover had last been tested, and whether the next shift could run the same containment from a runbook without me in the chair. That is the difference between a good night and a repeatable one.
How I ran it
The order of decisions, written out so the clock is visible.
- 01
I declared a major on the hardware fault once the scope was clear: about twenty percent of data-centre traffic, roughly one hundred and twenty customers. Waiting for a perfect diagram would have spent the clock the customers were already on.
- 02
The bridge was data-centre operations, network, the hardware vendor, and customer communications. I kept the timeline so the technical room and the customer sentence did not drift apart.
- 03
Containment came before curiosity. Traffic was shifted off the failed hardware where the design allowed it. Where it did not, the path was failover or replacement. The dead component was not the primary piece of work while customers were down.
- 04
The customer update included the share of impact and the time of the next update. An investigation with no number leaves every customer assuming they are the one who is down.
- 05
The path was brought back and then checked against traffic, not against the cabinet. We sampled customer flows. A restored box that is not carrying those flows is a photograph of a fix.
- 06
At about ninety minutes the service was online and the check had been done. The bridge stood down after that, not when the vendor said the part looked fine.
- 07
The post-incident review covered sparing, the last real failover test, and a runbook the following shift could execute. One fault is an incident. The same fault being able to take a fifth of the traffic again is a design problem.