At approximately 14:35:07 on Wednesday the 30th of October our Auckland City network experienced a complete loss of service.
Our monitoring alerted our engineers to this immediately and we began investigating, however before we could determine the fault our network recovered from this event and traffic began flowing correctly at approximately 14:40:20 (5 minutes, 13 seconds later).
This failure was caused by one of our distribution switches becoming completely unresponsive, however our automatic fail-overs worked as planned and full service was quickly restored with no human intervention. While we are happy our network can recover as it did from a major hardware failure, we believe we can restore service even faster in the future based on some information we discovered in our logs.
In order to bring this device back into service we needed to complete some tests which were scheduled for the early hours of the 31st of October. These tests were completed without any downtime or impact to our customers. As part of this work we have made some further changes which we hope will improve the speed at which we can recover from this event should it happen in the future.
This is a fairly major event — one which we do not wish to repeat any time soon — and we are very sorry for any inconvenience this may have caused.