February 2017

Network Issues: North Shore
The team has implemented a configuration change on the affected node and we will continue to monitor the situation over the weekend in case of any reoccurrences.
Feb 24, 09:08-18:06 NZDT
Network issues: North Shore
The node networking has remained stable and we are marking this issue as resolved.
Feb 23, 09:27-11:31 NZDT
International Connectivity Disrupted
Our upstream has advised that all upstream connectivity should be restored.
Feb 13, 23:31 - Feb 14, 00:57 NZDT
[Scheduled] Software & Application Updates
This maintenance is now complete.
Feb 8, 07:00-07:36 NZDT

January 2017

Phone Lines Unavailable
Our phones are now back up and you can reach us through all the usual methods!
Jan 31, 09:48-10:32 NZDT
Mail Server Outage
The issue has now been resolved.
Jan 25, 15:22-15:53 NZDT
[Scheduled] Scheduled Cloud Container Maintenance
The scheduled maintenance has been completed.
Jan 24, 07:01-09:00 NZDT

December 2016

[Scheduled] Software & Application Updates
This maintenance is now completed.
Dec 14, 07:01-07:18 NZDT
[Scheduled] Node Security and Stability Updates - Auckland City
The scheduled maintenance has been completed.
Dec 8, 01:00-03:00 NZDT
[Scheduled] Software & Application Updates
This maintenance has been completed, please get in touch if you run into any issues.
Dec 7, 07:15-07:40 NZDT
[Scheduled] Node Security and Stability Updates - North Shore / Auckland City / Sydney
The scheduled maintenance has been completed.
Dec 7, 03:00-06:00 NZDT
Urgent Maintenance
This incident has been resolved.
Dec 1, 03:46-03:57 NZDT

November 2016

Urgent Maintenance
This incident has been resolved.
Nov 30, 05:54-06:08 NZDT
Urgent Maintenance
The hardware node is now back online and all the VM's booted. Sorry for any inconvenience caused.
Nov 28, 07:55-08:14 NZDT
[Scheduled] Software & Applications Update
The scheduled maintenance has been completed.
Nov 23, 07:02-07:45 NZDT
[Scheduled] Server Security Maintenance - Auckland City
The scheduled maintenance has been completed.
Nov 22, 23:00 - Nov 23, 01:00 NZDT
[Scheduled] Server Security Maintenance - Sydney / Auckland City / North Shore
The scheduled maintenance has been completed.
Nov 21, 23:00 - Nov 22, 01:00 NZDT
Cloud Containers Backup Cluster Stability
This incident has been resolved.
Nov 18, 09:51-14:29 NZDT
[Scheduled] Server Security Maintenance - Auckland City
The scheduled maintenance has been completed.
Nov 17, 23:00 - Nov 18, 01:00 NZDT
[Scheduled] Applications Update
The scheduled maintenance has been completed.
Nov 16, 07:01-07:30 NZDT
[Scheduled] Server Security Maintenance - North Shore
The scheduled maintenance has been completed.
Nov 15, 23:00 - Nov 16, 01:00 NZDT
[Scheduled] SiteHost Services Upgrade
The scheduled maintenance has been completed.
Nov 10, 23:00 - Nov 11, 01:01 NZDT
Cloud Container Node High Load
This incident has been resolved.
Nov 9, 14:26-17:37 NZDT
[Scheduled] Applications Update
This maintenance has been completed.
Nov 9, 07:00-07:24 NZDT
[Scheduled] Applications Update
These scheduled updates have now been complete and the Control Panel is now open again.
Nov 2, 07:01-07:14 NZDT
Isolated Network Congestion (DDoS Attack)
The blocking is working well and the attack is no longer affecting our servers, so we are marking the issue as resolved.
Nov 1, 17:06-18:05 NZDT
[Scheduled] Security Patching (Dirty COW)
The scheduled maintenance has been completed.
Oct 31, 23:01 - Nov 1, 01:01 NZDT

October 2016

[Scheduled] SiteHost Services Upgrade
The scheduled maintenance has been completed.
Oct 27, 23:00 - Oct 28, 01:36 NZDT
[Scheduled] SiteHost Services Upgrade
The scheduled maintenance has been completed.
Oct 20, 23:01 - Oct 21, 00:14 NZDT
Network Congestion (DDoS Attack)
The network has been stable for a few hours now and we believe this incident to be resolved.
Oct 19, 11:43-15:14 NZDT
[Scheduled] Cloud Container Updates (Auckland Central Facility)
The scheduled maintenance has been completed. Please contact support if you are experiencing any problems.
Oct 18, 23:01 - Oct 19, 02:02 NZDT
[Scheduled] Emergency Network Maintenance
The scheduled maintenance has been completed.
Oct 14, 04:01-06:00 NZDT
[Scheduled] Cloud Container Updates (North Shore Facility)
The scheduled maintenance has been completed. Please contact support if you are experiencing any problems.
Oct 13, 23:01 - Oct 14, 01:05 NZDT
[Scheduled] Applications Update
The scheduled maintenance has been completed. Please contact support if you are experiencing any problems.
Oct 12, 07:01-07:15 NZDT

September 2016

[Scheduled] Minor System Update
The scheduled maintenance has been completed.
Sep 30, 11:31-11:45 NZDT
[Scheduled] Server Security Maintenance - Auckland City (Extended)
The maintenance has now completed and servers should be returning to their previous states.
Sep 30, 01:45-04:42 NZDT
[Scheduled] Server Security Maintenance - Auckland City
The scheduled maintenance has been completed.
Sep 29, 23:01 - Sep 30, 01:30 NZDT
[Scheduled] Server Security Maintenance - North Shore
The scheduled maintenance has been completed.
Sep 27, 23:02 - Sep 28, 01:00 NZDT
[Scheduled] Network Maintenance
This maintenance was completed at 02:25
Sep 27, 11:11 NZDT
[Scheduled] Server Security Maintenance - Sydney
The scheduled maintenance has been completed.
Sep 22, 06:01-07:00 NZDT
[Scheduled] Applications Update
The scheduled maintenance has been completed. Please contact support if you are experiencing any problems.
Sep 21, 07:01-07:25 NZDT
[Scheduled] Applications Update
This maintenance has been completed. Please let us know if you experience any problems.
Sep 7, 07:15-07:51 NZDT

August 2016

[Scheduled] Network Maintenance
This work has been completed successfully if you notice anything unusual please feel free to get in touch.
Aug 25, 08:44 NZDT
[Scheduled] Applications Update
The scheduled maintenance has been completed.
Aug 24, 01:01-01:30 NZDT
[Scheduled] Applications Update
This maintenance has been completed.
Aug 17, 07:16-07:33 NZDT
[Scheduled] SiteHost Server Migrations
This maintenance has been completed. Please contact us if you experience any problems.
Aug 15, 22:00-00:00 NZDT
[Scheduled] Applications Update
This maintenance has been completed. If you run into any problems please get in touch.
Aug 10, 07:01-07:28 NZDT
[Scheduled] Applications Update
This maintenance has been completed. If you experience any problems please get in touch.
Aug 3, 07:01-07:26 NZDT

July 2016

Australia Extended Server Maintenance
Sydney VPS and Cloud servers are back online, we apologize for the abnormal length of this maintenance.
Jul 21, 02:26-04:50 NZDT
[Scheduled] Cloud Containers Maintenance
This maintenance is now complete. If you have any issues please get in touch.
Jul 20, 07:15-07:54 NZDT
Australia Network Issues
Connectivity should now be restored to all Sydney services, if you are still experiencing any issues please get in touch with our support team.
Jul 14, 11:57-13:16 NZDT
[Scheduled] Applications Update
The scheduled maintenance has been completed. Please contact support if you are experiencing any problems.
Jul 6, 07:00-07:28 NZDT

June 2016

[Scheduled] Applications Update
This maintenance is completed. If you experience any troubles please get in touch.
Jun 29, 07:00-07:56 NZDT
[Scheduled] Applications Update
This maintenance is now complete.
Jun 22, 07:15-07:28 NZDT
[Scheduled] Cloud Container Maintenance
The scheduled maintenance has been completed.
Jun 13, 23:00 - Jun 14, 02:00 NZDT
[Scheduled] Applications Update
This maintenance is now complete.
Jun 1, 07:15-07:42 NZDT

May 2016

Monitoring system network instability
This incident has been resolved.
May 19, 10:52-23:06 NZDT
[Scheduled] Applications Update
This maintenance is now complete.
May 18, 07:00-07:51 NZDT
[Scheduled] Applications Update
This maintenance is now complete.
May 11, 07:00-07:31 NZDT
API Outage
The API server is back online and all systems should be operational, if you notice any issues please do not hesitate to contact support@sitehost.co.nz
May 8, 10:40-11:04 NZDT
[Scheduled] Applications Update
This maintenance has been completed.
May 4, 07:15-07:24 NZDT
Urgent Network Maintenance
This incident has been resolved.
May 2, 00:45-02:57 NZDT

April 2016

[Scheduled] Applications Update
This maintenance has been completed.
Apr 27, 07:00-07:30 NZDT
IPv6 connectivity
This incident has been resolved.
Apr 26, 10:34-12:03 NZDT
[Scheduled] Network Maintenance
This work has been completed successfully if you notice anything unusual please feel free to get in touch.
Apr 25, 00:00-04:00 NZDT
[Scheduled] Applications Update
This maintenance is now complete and everything is back online. As always if you experience any issues please get in touch.
Apr 20, 07:01-07:20 NZDT
[Scheduled] Applications Update
The scheduled maintenance has been completed.
Apr 13, 07:00-07:59 NZDT

March 2016

[Scheduled] Cloud Container Maintenance
The scheduled maintenance has been completed.
Mar 30, 23:00 - Mar 31, 01:00 NZDT
Mail Delays
This incident has been resolved.
Mar 29, 14:43-17:48 NZDT
Windows VPS Node Outage
The node has continued to be stable since last night. Our engineers have scheduled some maintenance next week to make some changes to keep things that way! Any customers impacted by this maintenance will be contacted shortly.
Mar 24, 08:41-14:26 NZDT
[Scheduled] Applications Update
This maintenance has now been completed. If you spot any issues please let us know!
Mar 16, 07:00-08:01 NZDT
Intermittent Shared Hosting Issue (PHP5)
We've been experiencing some trouble with one of our Shared Hosting Platforms. We believe it is resolved now, if you're still having problems please contact us immediately.
Mar 15, 12:29 NZDT
[Scheduled] Applications Update
These updates have been completed and everything appears to be up and running smoothly.
Mar 10, 07:30 NZDT
Problem with inbound calls
This incident has been resolved.
Mar 4, 08:59-12:07 NZDT
[Scheduled] Applications Update
This maintenance has been completed and everything should be up and running smoothly.
Mar 2, 07:00-07:56 NZDT

February 2016

[Scheduled] Application Update
This update has been completed and everything should be running smoothly.
Feb 24, 07:00-07:30 NZDT
[Scheduled] Applications Update
This maintenance is now completed and everything should be running smoothly.
Feb 4, 07:00-07:23 NZDT
[Scheduled] Applications Update
These updates have been completed and everything should be up and running as normal.
Feb 3, 07:00-07:39 NZDT

January 2016

[Scheduled] Applications Update
These updates have been completed and everything should be up and running as normal.
Jan 28, 07:15-07:31 NZDT
[Scheduled] Network Maintenance
This Network Maintenance is now completed, if you notice any issues please get in touch to let us know.
Jan 26, 00:00-04:16 NZDT
[Scheduled] Applications Update
These updates have been completed and everything should be up and running as normal.
Jan 20, 07:30-07:39 NZDT
Intermittent Network Issue
This incident has been resolved.
Jan 8, 21:37 - Jan 9, 02:18 NZDT

December 2015

Intermittent Network Issue
This incident has been resolved.
Dec 21, 16:33-17:54 NZDT
[Scheduled] Critical Security Maintenance
The scheduled maintenance has been completed.
Dec 17, 23:30 - Dec 18, 05:30 NZDT
[Scheduled] Applications Update
The scheduled maintenance has been completed.
Dec 16, 07:15-08:00 NZDT
Yahoo Xtra Mail Delays
Mails are being relayed for xtra.co.nz since this morning
Dec 7, 17:51 - Dec 8, 12:38 NZDT
[Scheduled] Shared Hosting & Email Maintenance
This maintenance has been completed. If you are experiencing any unusual behaviour with your websites on shared hosting or email please get in touch.
Dec 4, 00:15-03:05 NZDT

November 2015

Partial Network Outage
This incident has been resolved.
Nov 16, 10:53-11:03 NZDT
[Scheduled] Cabinet Maintenance
The scheduled maintenance has been completed.
Nov 10, 23:00-02:10 NZDT
Intermittent Network Issues
Our upstream provider has resolved this issue and all network traffic should be back to normal. If you do experience any trouble please contact support.
Nov 9, 09:26-14:00 NZDT
[Scheduled] Critical Security Maintenance
This maintenance has been completed. If you are having any issues please contact support.
Nov 5, 23:00 - Nov 6, 01:19 NZDT
[Scheduled] Critical Security Maintenance
The scheduled maintenance has been completed.
Nov 2, 23:30 - Nov 3, 02:54 NZDT

October 2015

Phone Lines Unavailable
This incident has been resolved.
Oct 26, 16:43-17:11 NZDT
[Scheduled] Applications Upgrade
This maintenance has been completed.
Oct 22, 07:00-07:28 NZDT
Yahoo Xtra Mail Delays
This incident has been resolved.
Oct 16, 10:12-16:25 NZDT
Mail Delays
This incident has been resolved.
Oct 13, 09:23-13:39 NZDT
[Scheduled] Network Maintenance
Network maintenance has been completed with everything going as expected, if you notice anything unusual please get in touch to let us know.
Oct 9, 05:00-05:30 NZDT

September 2015

Shared Hosting Outage (PHP5.2)
Users may have experienced a 5 minute outage on our Shared Hosting PHP5.2 server. Our engineers have resolved this issue and it should be online again. If you are still experiencing any trouble on your website please feel free to get in touch with our support team.
Sep 30, 14:41 NZDT
[Scheduled] Applications Upgrade
This upgrade has been completed.
Sep 30, 07:15-07:56 NZDT
Shared Hosting Outage (PHP5.2)
Users may have experienced a 5-10 minute outage on our Shared Hosting PHP5.2 server. Our engineers haveresolved this issue and it should be online again. If you are still experiencing any trouble on your website please feel free to get in touch with our support team.
Sep 22, 11:20 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
Sep 14, 07:15-07:40 NZDT
International Traffic Congestion
Our monitoring alerted us to some international traffic congestion today between 10am-10:30am. Engineers quickly responded to the issue and are continuing to monitor the situation closely. While we don't expect any impact to customers if you experience any problems please contact support.
Sep 8, 10:51 NZDT
Recursive Name Servers Outage
Last night between 3am-4:30am we experienced an outage on our Auckland City recursive name servers. Engineers resolved the issue and we are now looking into the root cause and why the service didn't failover as expected.
Sep 8, 10:28 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
Sep 7, 06:45-07:17 NZDT

August 2015

Partial Website Outage
This incident has been resolved.
Aug 31, 15:40-17:02 NZDT
Network Issue
This morning we experienced a network issue that impacted a couple of ISP's ability to access our network. We believe this issue has now been resolved but please contact support if you are having any problems.
Aug 27, 07:45 NZDT
Yahoo Xtra Mail Delays
All mail to Yahoo/Xtra hosted addresses should now sending again. If you have any further interruptions feel free to get in touch with our support team. support@sitehost.co.nz
Aug 20, 12:15 - Aug 26, 10:26 NZDT
[Scheduled] Scheduled Maintenance
The scheduled maintenance has been completed.
Aug 12, 07:15-07:30 NZDT
Degraded Network Performance
This incident has been resolved.
Aug 5, 14:26-15:48 NZDT
[Scheduled] Scheduled Maintenance
The scheduled maintenance has been completed.
Aug 5, 06:15-06:30 NZDT
Partial Network Outage
An inbound DDoS attack caused packet loss and increased latency on our network. As soon as our engineers identified the targeted IP it was added to our upstream blackhole list which resolved the situation.
Aug 4, 13:27-14:14 NZDT
[Scheduled] Scheduled Maintenance
The scheduled maintenance has been completed.
Aug 4, 07:45-08:00 NZDT
Yahoo Xtra Mail Delays
We believe this issue is now resolved we have contacted Yahoo and hope to prevent these issues in the future. If you do experience any further problems please contact support.
Jul 29, 11:13 - Aug 3, 10:26 NZDT

July 2015

[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
Jul 27, 07:15-07:35 NZDT
Yahoo Xtra Mail Delays
We believe this issue is now resolved but we are trying to contact Yahoo to prevent these issues in the future. If you do experience any problems please contact support.
Jul 22, 14:12 - Jul 24, 15:48 NZDT
North Shore Facility Issue
We have isolated and addressed the issues regarding international traffic from our network. If you are still experiencing any service interruptions at this stage feel free to get in touch with our support team.
Jul 22, 13:49-15:42 NZDT

June 2015

[Scheduled] Scheduled Maintenance
The scheduled maintenance has been completed.
Jun 16, 23:00 - Jun 17, 01:00 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
Jun 9, 07:00-07:30 NZDT

May 2015

API Outage
Everything should be operating normally. If you encounter any further issues with the service, please contact support@sitehost.co.nz
May 30, 08:06-08:33 NZDT
[Scheduled] VENOM Security Patching
The scheduled maintenance has been completed.
Jun 15, 07:15 - May 28, 00:45 NZDT
Yahoo Xtra Mail Issue
The issue appears to be resolved and all the queued emails have been transferred to Yahoo's mail servers.
May 21, 14:13 - May 22, 10:00 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
May 19, 07:00-07:26 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
May 14, 07:00-07:14 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
May 6, 07:05-07:15 NZDT

April 2015

[Scheduled] North Shore Linux VPS Restart
The scheduled maintenance has been completed.
Apr 24, 00:30-01:30 NZDT
[Scheduled] Network Updates
The scheduled maintenance has been completed.
Apr 23, 23:00 - Apr 24, 00:00 NZDT
Cloud Hosting Issue
Engineers have brought the node back online and all impacted customers should be back online. If you're still experiencing problems please contact the support team.
Apr 23, 19:02-19:22 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
Apr 21, 07:01-07:42 NZDT
[Scheduled] Windows Security Patches
The scheduled maintenance has been completed.
Apr 20, 23:00-23:51 NZDT
Intermittent Network Issues
Network services are back to normal now as we have isolated the issue. Engineers are continuing to investigate the cause but there should be no customer impact at this stage. If you are experiencing problems please contact support.
Apr 8, 16:12-16:46 NZDT

March 2015

SMTP Server Blacklisted
This morning our SMTP server was blacklisted by some organisations due to a customers compromised website sending out spam emails. This caused some customer outgoing emails to be bounced back from their recipient. We identified the issue and have had the blacklisting reversed. We will also be making some policy changes to prevent this from happening again which will be communicated to impacted customers in due course.
Mar 26, 11:30 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
Mar 25, 07:15-08:00 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed. Please contact support if you are experiencing any problems.
Mar 18, 07:09-07:32 NZDT
[Scheduled] Security Patches
This maintenance was completed and all nodes have been restarted. If you're experiencing any problems please contact support.
Mar 17, 02:41 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
Mar 10, 07:00-07:12 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
Mar 3, 07:00-07:26 NZDT

February 2015

[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
Feb 19, 07:00-07:33 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
Feb 18, 07:00-07:40 NZDT
[Scheduled] Virtual Dedicated Server Maintenance
The scheduled maintenance has been completed.
Feb 4, 23:00 - Feb 5, 02:00 NZDT
[Scheduled] Security Patches
Work completed.
Feb 4, 22:01 - Feb 5, 00:21 NZDT
[Scheduled] Security Patches
The scheduled maintenance has been completed.
Feb 3, 22:00-23:00 NZDT

January 2015

[Scheduled] Security Patches
The scheduled maintenance has been completed.
Jan 29, 22:00-23:15 NZDT
North Shore Facility Problem
We believe the problems at our North Shore facility are now resolved, and servers should be online. As always please contact support if you're having any problems.
Jan 28, 14:36-14:41 NZDT
[Scheduled] Applications Upgrade
The scheduled maintenance has been completed.
Jan 28, 07:00-07:15 NZDT

December 2014

No incidents reported for this month.

November 2014

No incidents reported for this month.

October 2014

No incidents reported for this month.

September 2014

No incidents reported for this month.

August 2014

No incidents reported for this month.

July 2014

No incidents reported for this month.

June 2014

No incidents reported for this month.

May 2014

Unscheduled Network Outage
At approximately 23:36:43 on Tuesday the 21st of May the network at one of our Auckland City Data Centers experienced a complete loss of service. SiteHost engineers were on-site conducting what they considered to be a low risk network change which involved installing a line card into each of our redundant aggregation switches. This upgrade is a small part of a larger network upgrade we are undertaking to increase our network capacity and improve our reliability. Almost immediately following the installation of the first line card our aggregation switches began exhibiting issues which did not fail-over automatically and as a result engineers on-site immediately began manually intervening to resolve the connectivity issues with service being restored at approximately 00:08:20. According to the vendors supporting documentation, these line cards are hot swappable and other similar deployments showed no indication of issue when installing them in a production environment. Our experience overnight differs greatly with what has been documented and we are working with our vendor to identify the root cause of this issue and to help explain why we experienced a loss of service in this situation. It quickly became apparent that as a result of the impact to the aggregation switches, loop detection mechanisms in our access switches had disabled their uplink ports. In order to restore service our engineers needed to login to every switch and re-enable these ports. We are investigating further improvements that can be made in this area. This was a fairly major event for us, our last being just over six months ago, and as always we will work hard to ensure we learn from this and make sure it doesn’t happen again. That being said sometimes even with the best planning and intentions things don’t go as smoothly as we would like. We are confident that these network upgrades are well worth any short term pain though. The increased speed and capacity of our network has enabled us to offer better national links and lower costs for international bandwidth by 80%, as well as the increased reliability we expect to see when this work is complete. Even so, we are very sorry for the inconvenience this caused and we appreciate the support from our customers as we continue to work hard improving the services we offer you. Quintin Russ Technical Director - SiteHost
May 21, 00:08 NZDT

April 2014

No incidents reported for this month.

March 2014

No incidents reported for this month.

February 2014

No incidents reported for this month.

January 2014

No incidents reported for this month.

December 2013

No incidents reported for this month.

November 2013

No incidents reported for this month.

October 2013

Unscheduled Network Outage
At approximately 14:35:07 on Wednesday the 30th of October our Auckland City network experienced a complete loss of service. Our monitoring alerted our engineers to this immediately and we began investigating, however before we could determine the fault our network recovered from this event and traffic began flowing correctly at approximately 14:40:20 (5 minutes, 13 seconds later). This failure was caused by one of our distribution switches becoming completely unresponsive, however our automatic fail-overs worked as planned and full service was quickly restored with no human intervention. While we are happy our network can recover as it did from a major hardware failure, we believe we can restore service even faster in the future based on some information we discovered in our logs. In order to bring this device back into service we needed to complete some tests which were scheduled for the early hours of the 31st of October. These tests were completed without any downtime or impact to our customers. As part of this work we have made some further changes which we hope will improve the speed at which we can recover from this event should it happen in the future. This is a fairly major event — one which we do not wish to repeat any time soon — and we are very sorry for any inconvenience this may have caused.
Oct 30, 14:40 NZDT

September 2013

No incidents reported for this month.

August 2013

No incidents reported for this month.

July 2013

No incidents reported for this month.

June 2013

No incidents reported for this month.

May 2013

No incidents reported for this month.

April 2013

No incidents reported for this month.

March 2013

Unscheduled Network Outage
At approximately 01:08, Thursday the 7th of March, our monitoring alerted our operations engineers that there were problems affecting our network. We began investigating and discovered network issues resulting in intermittent packet loss and at times complete network loss affecting all customers at our Auckland City facility. At the time SiteHost engineers spoke to our upstream suppliers support staff who advised us the issues were potentially related to a single upstream network provider having issues on their network causing a range of issues across these diverse upstream networks. SiteHost maintain an out-of-band management network and access to this was also affected making diagnostics more difficult than we would have liked. Despite some connectivity being restored around 04:00 our engineers observed issues via some paths which we began attempting to diagnose. While diagnosing these issues we continued to experience brief periods where connectivity was degraded or completely lost. After some troubleshooting, our engineers restored connectivity at approximately 06:00. Upon further investigation the following day, it was discovered that at least 2 completely separate network devices on the SiteHost network were running but not allowing traffic to pass through. We are still working with our partners to understand why this happened. We are also still waiting on an incident report from one of our upstream suppliers to help us understand why their network went down. The SiteHost network is fault tolerant, using diverse paths and will fail over in the event a network device completely fails, unfortunately it was not possible for our network to failover when this behaviour was observed. Restarting the affected process restored these devices to be fully functional again, we have been monitoring this since and these issues have not reoccurred. This was perhaps the most challenging and certainly the longest network incident we have experienced since we began managing our own network 4 years ago, for the following reasons: - Multiple upstream failed concurrently, delaying access to begin diagnosing issues. - SiteHost out-of-band management network access was affected. - SiteHost network devices malfunctioned silently. We have learnt a lot from this incident and have a number of changes planned to improve the diversity, monitoring and ultimately the reliability of our network - some of these changes were easy and have already been done, others will take us several weeks to complete. We would like to take the time to apologise for this extremely inconvenient service interruption. We would like to remind customers to please feel free to follow @sitehostnz on twitter for regular updates in the event of any network issues in the future. If you made it this far, thank you for reading and thank you for your continued support. Quintin Russ Technical Director - SiteHost
Mar 7, 06:00 NZDT

February 2013

No incidents reported for this month.

January 2013

No incidents reported for this month.

December 2012

No incidents reported for this month.

November 2012

Unscheduled Network Outage
At approximately 11:11, Friday the 23rd of November, our monitoring alerted our operations engineers that there were problems affecting some routes out of our network. We began investigating and found that one of our upstreams had experienced a hardware failure causing a network outage and for some routes including all international transit these had not failed over correctly. Once our engineers had identified the fault service was restored at approximately 11:33. We also experienced several more faults with this upstream provider over the 23rd, 24th & 25th, but these would have gone largely unnoticed as our network failed over as expected on these occassions. Upon investigation into the root cause for our network not failing over it was determined a configuration change made a few weeks earlier was the cause & we have already adjusted our testing processes to ensure more thorough testing is carried out to prevent this from occuring again in the future. We would like to take the time to appologise for this extremely inconvenient service interruption. While no-one can guarantee their service will never be affected by downtime, what we can promise is that whenever something like this occurs, we will have a plan for restoring service, and we will carry this out as quickly as possible, while keeping you informed via email / twitter (please feel free to follow @sitehostnz on twitter for regular updates) in a timely fashion. SiteHost sincerely apologise to all impacted customers for any inconvenience caused.
Nov 23, 11:33 NZDT

October 2012

No incidents reported for this month.

September 2012

No incidents reported for this month.

August 2012

No incidents reported for this month.

July 2012

No incidents reported for this month.

June 2012

No incidents reported for this month.

May 2012

Unscheduled Network Outage
At approximately 07:20, Friday the 4th of May, our monitoring alerted our operations engineers that there were problems accessing our Auckland City network which is colocated in the ICONZ datacenter, and investigation started straight away. Initial indications suggested a power failure on multiple core network switches causing a complete network outage. Availability of these devices was restored at approximately 07:40 and most customers would have noticed service restored at this point. Once service was restored some customers may have noticed intermittent packet loss causing slow network performance which was corrected at approximately 08:40. The intermittent packet loss was due to a complete loss of communications between the high availability systems in our routers, causing high load & several other network related anomalies. Once these network issues were resolved, our engineers then began investigating the root cause. It was discovered that our colocation provider had some scheduled maintenance between 05:00 - 09:00 that morning, which our engineering team were advised was network related and due to the redundant nature of the SiteHost network was not expected to have any significant impact. Unknown to our engineers the maintenance also included restarting a single (1) access switch which is located inside our non-shared cabinets. It is understood following several conversations that their technician mistakenly restarted multiple (2) SiteHost managed switches which are not in any way owned or operated by our colocation provider. This basic oversight had significant consequences for us as while our team has worked hard building a fault tolerant network it was not possible for the network to recover from the removal of multiple devices. SiteHost have strict processes in place for access of equipment in our cabinets & have been let down by our provider in communicating what work was being carried out. We have reiterated to our suppliers that we expect to be informed of any & all access to our locked cabinets & will be looking at any changes we can make to ensure this occurs in the future. In addition to this, our team has in the last few months been upgrading our core network, this work has been completed at our North Shore facility and we expect to be completed with this upgrade in the next few weeks. This upgrade will see our core network components housed in separate cabinets which when combined with a number of other planned improvements should make it significantly harder to repeat the service interruption you experienced last Friday. We would like to take the time to appologise for this extremely inconvenient service interruption. While no-one can guarantee their service will never be affected by downtime, what we can promise is that whenever something like this occurs, we will have a plan for restoring service, and we will carry this out as quickly as possible, while keeping you informed via email / twitter (please feel free to follow @sitehostnz on twitter for regular updates) in a timely fashion. SiteHost sincerely apologise to all impacted customers for any inconvenience caused.
May 4, 07:20 NZDT