In the early hours of the morning on Friday 8 November, we performed a scheduled maintenance for some of our customers' servers that involved migrating them to a new hardware node - NZ-AKL2-62ROG0. This migration completed at 2:36am without incident.

At approximately 8:45am that morning, the destination node (NZ-AKL2-62ROG0) experienced an outage. Our monitoring picked up this outage and our engineers brought the node and guests back up again by 8:58am.

After this initial outage, our team began digging into the root cause. The outage presented itself as a full lockup of the node, and logs leading up to this point showed nothing untoward. Due to the migration to the node we had performed earlier that morning, our initial suspicion was the load resulting from the influx of guests had overloaded the server leading to the crash. Performance post-crash had improved and was looking normal.

At 2:23pm the node crashed again. We were able to recover the node by 2:29pm but due to the previous outage the decision was made to begin a partial rollback of the migration that had been performed the night before - we elected to keep Cloud Container servers that were migrated on the new node and move other servers back to their original node. This process was begun and emails were sent to affected customers, with a scheduled time of midnight to begin the switchover.

At 3:53pm the node went down again with the team bringing it back up by 4:01pm. A fourth outage occurred at 9:03pm with service restored by 9:11pm. The emergency maintenance then proceeded after midnight with the node and guests remaining stable since, hosting the Cloud Container servers that had previously been migrated across.

We're sorry for the downtime experienced as a result of these outages. We understand the impact an incident like this can have on your business and we're confident that the node will remain stable as it has since the partial rollback was completed. The team will be reassessing our maintenance processes to see if any improvements can be made, and we will continue investigating the root cause of the issue with further information provided as it comes to light.

If you have any questions or concerns related to this incident, please get in touch at support@sitehost.co.nz or give our team a call at 0800 484 537.