At approximately 11:11, Friday the 23rd of November, our monitoring alerted our operations engineers that there were problems affecting some routes out of our network. We began investigating and found that one of our upstreams had experienced a hardware failure causing a network outage and for some routes including all international transit these had not failed over correctly. Once our engineers had identified the fault service was restored at approximately 11:33.
We also experienced several more faults with this upstream provider over the 23rd, 24th & 25th, but these would have gone largely unnoticed as our network failed over as expected on these occassions. Upon investigation into the root cause for our network not failing over it was determined a configuration change made a few weeks earlier was the cause & we have already adjusted our testing processes to ensure more thorough testing is carried out to prevent this from occuring again in the future.
We would like to take the time to appologise for this extremely inconvenient service interruption. While no-one can guarantee their service will never be affected by downtime, what we can promise is that whenever something like this occurs, we will have a plan for restoring service, and we will carry this out as quickly as possible, while keeping you informed via email / twitter (please feel free to follow @sitehostnz on twitter for regular updates) in a timely fashion.
SiteHost sincerely apologise to all impacted customers for any inconvenience caused.