03 · Tabcorp
Victoria retail, rolled back
Stores across Victoria were dark after a firewall change. I directed the rollback, then put peer review and director sign-off on that class of change.
RollbackVictoria retail
Retail stores across Victoria went dark after a firewall change. In this business the shop is the product: if the venue cannot take a bet, the incident is already customer-facing, whatever the monitoring tile says. It was not one store and it was not the public website. The pattern was the retail network in one state.
A firewall fault tempts people into a forward fix — one more rule, applied live, while the venues are closed. That is how a bad change becomes two bad changes. I directed a rollback of the change that had just gone in, with network, the person who had implemented it, store operations, and the vendor if they owned the box, all on one bridge.
Stores were confirmed back from a sample of venues, on the till, not from a single successful ping. The update to stores said what had failed, what we had done, and what was actually up. Any region still dark kept a next-update time rather than a hopeful all-clear.
The night after the restore is where this result lives. Peer review, director sign-off, and an impact check that names retail were added to that class of network change. The next firewall change on the retail estate does not move without them. A control that stays in the review document, and never enters the change path, did not happen.
How I ran it
The order of decisions, written out so the clock is visible.
- 01
I treated the symptom as the retail network across Victoria until the bridge proved a narrower cause. One venue is a store problem. A state of venues is an incident.
- 02
The bridge was network, the engineer who had implemented the change, store operations, and the firewall vendor where they owned the device. One call. Store operations were there because they knew which venues were actually unable to trade.
- 03
The change was the suspect, so the decision was rollback rather than a new rule written under pressure. Shops that cannot trade do not get to be the test environment for the next idea.
- 04
Rollback ran in a stated order. We did not announce the state restored until a sample of venues had been checked at the till. A ping answers a different question from a customer standing at the counter.
- 05
The store update named the failure, the action taken, and what had been confirmed up. Where a pocket was still dark, the message carried the time of the next update instead of smoothing it over.
- 06
While the night was still obvious, the control was written down: peer review of the change, director sign-off, and an impact check that says retail rather than the vague word network.
- 07
That control was put into the change path. The following firewall change on retail could not proceed without it. Recognition from leadership followed the reduction in failed production changes, not the story of the night.