Affects
- UpdateFebruary 28, 2025 at 1:52 AMCompletedFebruary 28, 2025 at 1:52 AM
Maintenance for this has been completed today. We've found the root cause to these Layer 2 issues relating to the router appliance itself. Luckily we have built the network to be redundant, so we will be shifting traffic over to a working router while we work on getting this defective one replaced.
Again this maintenance will continue to take place at 9:00 AM Central time to avoid dropping players in the peak hours of the day. In the mean time some changes have been made to lessen the effects of these spikes.
To clarify, we would've had this done today, however the backup router was still in the process of being setup and was paused while Onsite was being developed. We needed a few prerequisites completed from our upstream before we could shift any traffic onto it. Which is why we are continuing this tomorrow as it went too far into prime time today to justify dropping traffic. - CompletedFebruary 27, 2025 at 11:00 PMCompletedFebruary 27, 2025 at 11:00 PM
Maintenance has completed successfully
- In progressFebruary 27, 2025 at 3:00 PMIn progressFebruary 27, 2025 at 3:00 PMMaintenance is now in progress
- PlannedFebruary 27, 2025 at 3:00 PMPlannedFebruary 27, 2025 at 3:00 PM
Hello everyone, our network team has identified the root cause of the frequent short drops that have plagued the network over the past few days. We understand the severity of these drops and have identified the root cause. To permanently resolve these issues, we will be conducting emergency maintenance at 9:00 AM Central Time. During this maintenance, you will experience brief disruptions as mass changes will be done. We apologize in advance for any inconvenience this may pose to you, especially over these past few weeks where we have suffered instability from one of our upstream providers as well as the network maintenance we conducted to upgrade our core late last month. To be frank, the timing could not have been worse.
For transparency, the issue stems from a Layer 2 issue inside our VXLAN-EVPN network, which is affecting our spine switches that processes essentially all traffic on our network. As a result, this was felt widespread and not just related to one upstream or one cabinet. This issue appears to have sprung from the preparation of on-boarding our on-prem protection suite into our core. We have been working at a fast pace to get that online and operational in hopes of shifting all traffic over in order to provide stability.
We know the past few weeks have been difficult, and we recognize the importance of a reliable network for our customers and are tirelessly working to improve our offerings.