ScootKit - Issues with multiple servers – Incident details

We've restored full functionality of all services.

Bot-Host #11 has been restored, and we're monitoring the situation and rolling back the emergency patch to restore full functionality.

All other Bot-Hosts except Bot-Host #11 are reachable again. We're still working on this situation and are waiting for our hosting provider to address the issues affecting Bot-Host #11.

Our hosting provider has restored access to "some servers", including Bot-Host #1 and some core components.

Due to the emergency patch being applied, Early Access continues to be unavailable for now.

Additionally, some internal services, such as our mailserver, can be reached again.

Bots that aren't on any affected server can now be started again thanks to an emergency patch. Please note that Early Access features are temporarily not available due to the mentioned core component failure.

We're restoring access to SCNX and have updated internal systems to ensure that the failed core component won't impact data integrity. Certain features, such as dcserver.link configurability can't be used due to the failed core component.

Meanwhile, our hosting provider didn't share any new details regarding their "cooling issues".

We've received reports that due to core infrastructure being affected on at least on of the servers, using SCNX might lead to messages being overwritten by corrupt values, leading to issues in the future. In the interest of data integrity, we've blocked SCNX access for now, while we wait for our hosting provider to resolve this issue.

We've identified some of these components as possible single points of failure in the past and took measures to prevent such failure, but our investigation shows that one core component used to delivery experiments can cause SCNX to assume a state that is incompatible with data written by a newer state released using the experiment service.

We've discovered that due to a core system being located on at least one of the affected servers, restarting your bot will lead to your bot being offline. Stopped bots can't be started and restarts will lead to temporarily making your bot offline.

Avoid restart your bots. Bots can't be started until this incident has been resolved, spamming the restart button doesn't change this unfortunately.

Our hosting provider has confirmed that this issue is caused by an "unexpected cooling issue" and they are actively working on resolving this as soon as possible. This incident is outside of our control.

Please note that only services marked red on status.scootkit.net are affected, other Bot-Hosts are not affected. Do not switch Bot-Hosts while this incident is ongoing to avoid corruption of your bot files.