Date of incident: 2024-09-25
Date reported internally: 2024-09-25Date reported to customer: 2024-09-25
Summary of Impact
In the EV Portal sending/writing messages to charge stations stopped working properly. After confirming such an operation (clicking Save button) entities were shown in a syncing state that was not completed. The action initiated by the user on the screen was lost. Also, some action on the screens led to the display of error messages.
The impact was mostly limited to user initiated actions in EV Portal. Automated message flows from and to charge stations as well as well as on external interfaces (OCPI, OICP, Platform API) was not impacted.
Detailed Timeline of Events
15:44 Our infrastructure team decommissioned the old service bus namespace as a finalization step of the maintenance conducted on 2024-09-11.
16:22 The problem was reported by our support department as a critical issue to the responsible team.17:45 The problem was solved and EV portal resumed full functionality.
Confirmed Root Cause of the Incident
We recently did a migration of a critical service bus namespace to a higher service tier to help us absorb the growth of our platform. An alias to the old namespace should have been kept on the new namespace as a fallback. It appears that this behavior has been silently removed between the previous namespace migrations we did and this one. As a result, some of our services could no longer connect to the correct service bus namespace.
We have opened a ticket with Microsoft to investigate the issue. We are not dependent on this for an issue fix but would like to understand how this could have happened. There is no public announcement of a change in behavior.
Mitigation and Resolution
The affected services were quickly restored by updating the namespace connection details manually.
Lessons Learned for System and Process Improvements
Our current understanding is that the issue has been caused by an undocumented/unannounced change in behavior of the upgrade procedure at Microsoft. We are in contact with Microsoft support to better understand why the behavior was different this time. Any learning from this that we could apply to improve our internal processes we will regard with interest.