- PostmortemPostmortem
A small typo in a DNS record created due to the manual setup steps of another bot-host made this server unreachable for a prolonged period of time. Our processes after the initial report of the downtime focused on identifying the issue on our hosting infrastructure, instead of reviewing recently taken actions which could have highlighted configuration changes in our DNS configuration. The processes have been made optimized for this case and call for a proper review of recent changes to infrastructure in addition to the already existing issue identification steps.
- ResolvedResolved
This issue has been resolved, Bot-Host #17 is reachable again. Please note that hosted bots were not affected.
This incident was caused by a misconfiguration of DNS records, made in error when setting up infrastructure for another Bot-Host. We apologize for the inconvenience caused.
As no bots were offline, this incident is ineligible for our voluntary refund policy.
- MonitoringMonitoring
We've rolled out a fix for this issue and are monitoring results. Affected customers should be able to access their bots again. Hosted bots were not affected in this incident.
- IdentifiedIdentified
We've identified a potential misconfiguration that might cause this issue. We're rolling out a fix.
- UpdateUpdate
We've identified that connectivity issues are limited to Bot-Host #17 - Bots hosted on Bot-Host #17 are not affected.
We're continuing to investigate the root cause of this issue.
- InvestigatingInvestigating
We've received reports of an increased number of connections to Bot-Hosts failing. We're investigating.
