Date of incident: 2025-05-14
Date reported internally: 2025-05-14
Date reported to customer: 2025-05-14
Summary of Impact
Transaction service, responsible for handling Start Transaction messages from charge stations started to fail with an Error Response.
This does not mean that Charging Sessions could not be started locally on the charge stations.
The impact of this was that no CDR could be created.
Detailed Timeline of Events
All times refer to CEST time zone (UTC+2)
- 00:20: Transaction Database switched partitions starting to cause errors in processing StartTransaction messages.
- 02:20: First customer reports the issue
- 07:12: Internal chat was created.
- 07:32: Message posted on status page to inform customers.
- 08:02: Development team informed the team that a fix was already prepared for the issue.
- 08:16: The fix is deployed to GFX PROD to mitigate the issue
- 08:17: We see the errors have stopped and TransactionEvents is processing StartTransaction messages properly again
- 08:22: All systems back to normal operating condition. Message on status page that incident is over.
Confirmed Root Cause of the Incident
The Physical Database partition storing the Transactions started to reach its limit, that resulted in not accepting requests to store new transactions.
Mitigation and Resolution
We have changed our partition key so that it acts more even (and in order to avoid similar issues in the future).
Lessons Learned for System and Process Improvements
Alerts need to be improved to have a better and faster view on critical issuesImprovement with our external emergency landline need to be done