Summary of Impact
On September 25, some users experienced limited access to the EV Portal. While certain screens loaded successfully, others displayed the message: "The service is not available." This issue affected the usability of the portal for a portion of users during the incident window.
Timeline of Events
- 08:00 CEST: A new version of EV Portal was deployed to production.
- 10:15 CEST: Another version was deployed.
- 12:40 CEST: An unusual accumulation of messages was observed in the service bus.
- 12:45 CEST: Scaling actions were taken. Message volume began to decrease gradually.
- 13:00 CEST: Limited availability of EV Portal was detected.
- 13:23 CEST: EV Portal web site was restarted, resolving the issue.
- 15:00 CEST: A new version was deployed.
Root Cause
The issue was caused by new feature that was deployed earlier in the day. It was designed to send real-time notifications to users and worked well under normal conditions. However, the system was overwhelmed when we received an unexpectedly high volume of messages, which caused the temporary outage.
Resolution and Mitigation
The notification functionality was rolled back to stabilize the system. The portal returned to normal operation after web site restart.
Next Steps and Improvements
To prevent similar issues in the future, we are reviewing the architecture to avoid direct calls from high-volume message handlers to services hosted within the EV Portal. Alternative approaches will be explored to ensure scalability and reliability during peak loads.