Date of incident:
2025-07-28 (15:00 - 15:51 CEST)
2025-07-29 (16:10 – 17:10 CEST)
2025-07-30 (16:40 – 16:50 CEST)
Date reported internally: 2025-07-29
Date reported to customer: 2025-07-30
Summary of Impact
Between July 28 and July 30, 2025, the OCPI service experienced multiple periods of degraded performance and availability.
The impact of this was slow API responses, OCPI data (e.g. meter values, sessions, CDRs) not being transmitted and timeouts on endpoints at ocpi.greenflux.com.
Detailed Timeline of Events
2025-07-28
- 12:54 UTC – Partner reported timeouts and remote charging failures
- Afternoon – Partner reported network-wide timeouts on Greenflux-connected OCPI networks
- Evening – Customer identified discrepancies in session data
2025-07-29
- 16:10 CEST – Initial performance degradation observed on OCPI service.
- 16:17 CEST – 2nd line support reported issues loading OCPI subscriptions via the Portal
- 16:32 CEST – Investigated OCPI subscription APIs; observed delayed responses and increased error rates
- 17:10 CEST – System performance started to recover.
2025-07-30
- 16:40 CEST – Performance degradation reported again by customers
- 16:50 CEST – System recovered.
Confirmed Root Cause of the Incident During the incident windows, the service experienced an unexpected spike in traffic/load, auto scaling didn't work, and the service couldn't handle that load.
Mitigation and Resolution Review and optimize OCPI-related database queries to reduce likelihood of SQL Server timeouts.
Lessons Learned for System and Process Improvements Improvement in monitoring and timeout alerts are needed to have a better and faster view on critical issues.