News Categories
News
Simplified Hosted PBX - One Way Audio - 24/07/2019 16:40
Posted by Adam Howard on 25/07/19 - 13:58

OUTAGE REPORT

Good Afternoon, We would like to start by offering our apologies for the impact to service for resellers and customers late yesterday afternoon. After discussions with our platform partner we have a full report on the incident which is as follows.

Third-party DNS Server failure

Start

24/07/2019 16:40

End

24/07/2019 17:25

Duration

0:45

Impact

High

Customers affected

Few

Products/services

Calls

 

  1. Executive Summary

We use a third-party DNS service, Quad9, with Google as a back-up. Quad9 had a failure and Google rate limited the number of accesses which could be made. This caused a number of knock-on effects for any services which use domain names rather than IP addresses (which is most of our services).

We were not expecting to be rate limited on our back-up server, however mitigation is fairly simple as described in detail below. We therefore do not expect a repeat of this issue.

  1. Detailed explanation

We use two nameservers: Quad9 DNS (9.9.9.9) and Google Public DNS (8.8.8.8). Unfortunately, 9.9.9.9 suffered some issues, which caused a cascade affect which affected the following:

  1. DTMF Requests began to slow down/fail due to REST calls not resolving
  2. Outbound calls may have failed if they were using domains due to the REST calls failing
  3. The slow REST calls caused queues which meant there were also voice issues present

 

We failed over to the secondary nameserver, however, we did not realise that Google enforces a QPS (Queries Per Second) policy and so stopped responding to us.

 

  1. Mitigation

 

To combat this, we will introduce Cloudflare (1.1.1.1) as our 2nd server and have Google as our 3rd. Cloudflare do not enforce a QPS limit

 

We will also add static host file entries for our REST requests, that way, in the event that the DNS fails (for whatever reason) REST requests would not suffer since our own HOSTs files would take over.


Comments (0)