Morning of 3 August 2019, alarms notified me that there was an issue with system name resolution. After troubleshooting what was going on, we discovered two of our name servers were not responding to queries. We manually changed the resolvers on the server immediatley, and all started working fine. Subsequently, the ansible playbooks were updated with the new name servers.
After we began the process of notifier users that the service disruption was over, it was noted our IRC server had split from the network, due to resolution issues. We reconnected to hub.tilde.chat, and that was resolved.
Total time of the disruption was 53 minutes. We apologize for the disruption, and are working to ensure this doesn't happen in the future.