Blackboard Learn unavailable
Incident Report for Wentworth Institute of Technology
Postmortem

Incident Details

3:25 PM ET

A scheduled routine change to our DNS configuration was initiated. An error in the execution of this change resulted in misconfigured DNS entries causing domain resolution errors for

blackboard.com domains. This change propagated across the internet over time, causing Blackboard services to be inaccessible.

6:30 PM ET

The Engineering team was able to identify the root cause and restore the DNS configuration using a backup file.

6:40 PM ET

Access to all products was restored.

Incident Root Cause

A change to the DNS configuration was made that resulted in the misconfiguration of several internet facing

Blackboard DNS zones. Once configuration of the DNS zones was corrected access to all impacted Blackboard

services was restored. At no point were the actual systems/environments down, however, they were

inaccessible from the internet for some time. No data loss occurred as a result of this event.

Incident Resolution

Resolution required restoration of the proper DNS configuration using a backup file.

Recommended or Preventive Action

Blackboard continues to review and validate staffing policies, training, and change management

procedures with our infrastructure maintenance service provider.

Our service provider has executed this change type successfully on multiple occasions.
Corrective action with the responsible staff has been taken by the management team.

Posted Oct 22, 2019 - 12:41 EDT

Resolved
Due to a vendor related service issue, Blackboard Learn was unavailable for approximately 1 hour.
Posted Oct 07, 2019 - 18:00 EDT