ScootKit - Network issues on Bot-Manager & Backend – Incident details

All systems operational

Network issues on Bot-Manager & Backend

Resolved
Partial outage 70 %
Started 4 months agoLasted about 1 hour

Affected

SCNX

Partial outage from 3:28 PM to 3:28 PM, Degraded performance from 3:28 PM to 3:28 PM, Partial outage from 3:28 PM to 3:43 PM, Degraded performance from 3:43 PM to 4:17 PM

SCNX Backend

Degraded performance from 3:28 PM to 3:28 PM, Partial outage from 3:28 PM to 3:43 PM, Degraded performance from 3:43 PM to 4:17 PM

SCNX Bot-Manager

Partial outage from 3:28 PM to 3:43 PM, Degraded performance from 3:43 PM to 4:17 PM

Updates
  • Resolved
    UTC
    Resolved

    We have fully resolved the network issues affecting the Bot-Manager and Backend.

    After a thorough investigation, we determined that the root cause was a DNS server failure at our hosting provider. This failure led to internal connection timeouts, which temporarily impacted the SCNX dashboard and some bots located on servers at that specific provider.

    Our systems are now fully stable, and we are no longer seeing any failed requests. In total, the downtime for this incident was less than 45 minutes. We deeply apologize for the inconvenience and frustration this disruption caused to your experience.

    If your services were impacted during this window, we invite you to claim an SCNX Care Voucher. You can file a request by visiting: https://scnx.app/user/downtime-refund

  • Update
    UTC
    Update

    After reviewing user reports, we can confirm that a few bots attempting to start during the incident were stopped automatically due to network failures on Bot-Host #38. Most of these bots have now automatically started, but some bots - particularly bots that users attempted to manage during the downtime via the dashboard - might need to be started manually due to the failure affecting the network process.

  • Monitoring
    UTC
    Monitoring

    The rate of failing requests has dropped below 1%, meaning that most requests now work. The dashboard should work now without issues again, but we're continuing to monitor the situation and will update this page with further information once available. Hosted Bots were not affected.

  • Identified
    UTC
    Identified

    We've adjusted some networking systems and we're seeing a partial recovery, with now less than 10% of requests failing. We're still working on getting this resolved as soon as possible.

  • Update
    UTC
    Update

    Quick additional note: It seems like only around 70% of requests fail, meaning that reloading the page might allow you to create a temporary connection. Still, please avoid spamming this to avoid ratelimits blocking your device.

  • Investigating
    UTC
    Investigating

    We're seeing an increased amount of internal traffic between Bot-Manager and Backend failing. We're investigating this. Hosted Bots are not affected and will continue to work normally, only the dashboard is affected.