Good Morning, 

I wanted to start by experssing our sincerest apologies about the length of the outage and the lack of communication and updates. We will be having a full review with the supplier and providing feedback on the way this has been handled. 

We have been provided with the following summary of the issue and breakdown of the progress overnight as follows. I am now asking for a more recent update and confirmation of when the full Reason For Outage report will be provided.

A further update will provided on this before the end of the day

Issue Summary: Access to NMC has just been restored following some routing changes and resetting Virtual Machine (VM) environment.

There are a pair of Cisco routers that support the SPN (Service Provider Network) in Global Switch (GS). Both of these devices had failed earlier in the day. One of the Cisco devices was brought back in to service at 2pm but service restoration for end users was gradual. As they did this restoration issues were identified with a port that connects out to Netapps Storage. This was shut down. Customer services were restored, and broadband services began to connect again via GS. Some MPLS customers did not come back and this is still being investigated. 

The plan is to bring the second device back starting at 8pm. Therefore customers may see connections drop intermittently during this period. The change window will be 8pm through to 2 am.

Issue Timeline:

Broadband (Currently In Monitoring)

P1 - Multiple Connections Down - Service Partner planned works have ran further than expected causing large outages across the country. This is believed to have caused a knock on affect onto our systems. We're currently investigating to get these connections back online.
19 May 2022 at 08:00 - Incident raised by Nasstar Customer Services 1 day 38 mins. ago
19 May 2022 at 17:27 (15 hrs 11 mins ago) - Investigation is ongoing but Nasstar engineers are restoring services gradually. We expect to see more recovery through the evening and will place the next update here at 8.30pm
19 May 2022 at 17:33 (15 hrs 5 mins ago) - Partial service is now restored and will continue to increase during the evening. Between 8-12pm this evening services will be interrupted as emergency engineering works are completed.
20 May 2022 at 07:10 (1 hr 28 mins ago) - Services have been restored following a major network incident impacting our Global Switch location. We apologise for the disruption has had on your services. There may be a small number of individual customer services that have not recovered fully. Please log these in the normal way and we will work to resolve as soon as possible.
20 May 2022 at 07:12 (1 hr 26 mins ago) - Service: 'Leased Line' has been updated to 'Monitoring'
20 May 2022 at 07:12 (1 hr 26 mins ago) - Service: 'Broadband' has been updated to 'Monitoring'
20 May 2022 at 07:12 (1 hr 26 mins ago) - If you are continuing to experience service disruption, then please contact our service desk or your client service manager.
21 May 2022 at 08:00 (39 mins ago) - This incident is now resolved.