Summary of incident
On July 7, 2026, a Microsoft Azure Service Bus topic used for processing charging transactions and related events reached its maximum storage capacity. As a result, new messages could no longer be accepted, causing delays and interruptions in the processing of transaction events, CDR generation, token synchronization, and several related services.
The issue was resolved by clearing the affected queue and deploying a permanent software fix. Following the resolution, extensive recovery activities were performed to reprocess missed events and restore impacted data wherever possible.
Customer Impact
Between 09:30 and 11:39 UTC on July 7, 2026, some platform events were not processed as expected.
The following services may have been affected:
- Charging session start and stop events were delayed or not processed
- Charge Detail Records (CDRs) for GreenFlux-owned charge points were not generated in a timely manner
- Token updates received through the Platform API were not synchronized to connected systems and external CPOs via OCPI
- A subset of incoming OCPI and Hubject CDRs may not have been processed immediately
- Mileage submissions through the Charge Assist API experienced processing delays
Timeline
Time (UTC) | Event |
09:30 | Service Bus topic reached maximum capacity and stopped accepting new messages |
10:00 | Automated monitoring alerts were triggered due to abnormal CDR volumes and increased processing latency |
10:45 | Investigation commenced |
11:39 | Dead-lettered messages were removed from the affected subscription, restoring message processing |
17:58 | Permanent software fix deployed to prevent recurrence |
Root Cause
The incident was caused by a defect in a recently deployed component responsible for online fraud detection.
Under specific conditions, messages containing incomplete information were automatically moved to a dead-letter queue (DLQ). Due to an unexpectedly high volume of such messages and the absence of automated cleanup procedures, the dead-letter queue continued to grow until the Service Bus topic reached its maximum configured capacity.
Once the capacity limit was reached, publishers were no longer able to write new messages to the topic, resulting in the interruption of downstream event processing.
The relevant functionality had been deployed on July 2, 2026, and the accumulation of dead-lettered messages gradually increased until the capacity threshold was reached five days later.
Resolution
Immediate Mitigation
To restore service GreenFlux:
- Cleared the dead-letter queue associated with the affected subscription
- Restored message processing immediately after capacity became available
Permanent Fix
A software change was deployed on July 7, 2026 that modified the Online Fraud Detection service to prevent the creation of large volumes of dead-lettered messages under similar circumstances.
Data Recovery Activities
Following service restoration, GreenFlux performed extensive recovery actions to process missed events and recover affected data.
Recovery activities included:
- Reprocessing dead-lettered messages from affected subscriptions
- Recovering and replaying missed transaction start and stop events
- Reprocessing delayed CDRs through dedicated recovery workflows
- Replaying backlogged OCPI-related messages and other queued events
As a result, more than 14,000 CDRs were successfully recovered and processed.
A small number of CDRs could not be automatically recovered. Most of these records were already associated with pre-existing validation or business-rule issues that required separate investigation.
Preventive Actions
To reduce the likelihood and impact of similar incidents in the future, GreenFlux has implemented or initiated the following improvements:
- Enhanced monitoring and alerting for Service Bus topic capacity utilization
- Additional monitoring of dead-letter queue growth
- Review and validation of dead-letter queue cleanup procedures across services
- New operational tooling to simplify the replay of missed transaction events
- Improvements to resilience and error-handling mechanisms in publishing services
- Review of Service Bus capacity configuration to better accommodate abnormal message volumes
- Simplification and strengthening of recovery and replay processes
Current Status
The incident was fully resolved on July 7, 2026. The permanent fix has been deployed, recovery activities have been completed, and additional monitoring and resilience improvements are being implemented to prevent recurrence.
We apologize for the disruption and appreciate your patience while our teams worked to restore normal service and recover affected data.