On Wednesday 26th September at approximately 9:30am we experienced performance issues on one of our North Shore hardware nodes - NZ-AKL2-9QQQZ9. The node appeared to be running out of memory and as a result the virtual servers on the node went offline. Our monitoring notified us of this immediately and we began to investigate, restarting the node and bringing the virtual servers back up. Unfortunately we experienced the same problem as before so began again, bringing up the virtual servers one by one to try and identify which server was causing the memory issues.

Initially, this resulted in all servers starting successfully and no recurrence of the memory issue was immediately apparent. Approximately one hour later however we saw the problem reappear with the virtual servers going down once more.

While tackling this latest outage our team noticed a corresponding uptick in network traffic, which upon further investigation pointed to a Distributed Denial-of-Service (DDoS) attack being performed against one of the virtual servers on the node. We then blocked this traffic upstream, and worked with the customer to reconfigure the targeted server to prevent the attack from continuing. Once this was complete the node and the virtual servers it hosts remained stable.

These incidents are very disruptive and we apologise for the downtime that customers experienced. This was quite a unique DDoS attack that presented itself in an unusual way which didn’t trigger our usual network monitoring or our upstream provider’s monitoring. As a result we will be making some changes to monitor for this style of attack moving forward which will help us identify problems faster, and in addition our team is reviewing what changes can be implemented to prevent these sorts of attacks from impacting our services in the future.

If you have any questions about this incident or would like further detail, please get in touch with us at support@sitehost.co.nz .