Summary
On 12 July 2026, GreenFlux experienced a service disruption caused by an outage in Microsoft Azure Event Hub services in the West Europe region. This external platform issue affected the processing of messages between several GreenFlux components.
As message processing slowed, a related messaging component reached its configured capacity limit, resulting in temporary message backlogs. This led to intermittent failures for some charging session start requests and delays in the availability of reporting data.
Our investigation confirmed that charging session completion data was processed successfully and that no significant transaction or CDR data was lost.
Customer Impact
During the incident:
- Some charging session start requests were processed with delays or experienced temporary failures.
- Reporting and downstream data processing were delayed.
- Completed charging sessions continued to be recorded successfully.
- A subset of charging session stop transactions was later identified as requiring reprocessing due to an undiscovered software defect triggered by the Event Hub outage.
- No significant loss of charging transaction data was identified. Most affected transactions and CDRs were successfully recovered through targeted reprocessing activities.
The issue was resolved, all affected services recovered successfully, and the remaining edge cases continue to be reviewed separately.
Timeline
Time (CEST) | Event |
12 Jul 2026 20:12 | Monitoring detected a sharp increase in queued messages, indicating delays in message processing. At the same time, production logging capabilities were partially unavailable, which limited visibility into the underlying issue. |
12 Jul 2026 20:22 | A messaging component reached its configured capacity limit, causing additional message backlog growth. |
12 Jul 2026 20:39 | Engineers increased the capacity of the affected messaging component, restoring normal message flow and resolving the immediate service impact. |
13 Jul 2026 10:27 | Further investigation identified failures when publishing events to Azure Event Hub. |
13 Jul 2026 10:34 | Microsoft Azure Service Health confirmed an Event Hub platform issue affecting the West Europe region. |
13 Jul 2026 (afternoon) | Logging and monitoring systems were progressively restored, allowing a more detailed investigation. |
14–15 Jul 2026 | Engineering teams completed a detailed analysis and confirmed that charging session completion processing had largely succeeded and that no significant transaction or CDR data loss had initially been identified. |
21 Jul 2026 15:32 | The Support Duty Team reported a related issue to the development team responsible for the original incident, triggering a renewed investigation. |
21 Jul 2026 17:00 | A previously undetected defect was identified in the StopTransaction processing flow. Under specific conditions, failures when publishing to Azure Event Hub prevented transaction records from being fully updated, causing processing to stop before CDR generation could be completed. |
22 Jul 2026 15:00 | Engineering implemented a dedicated maintenance endpoint and custom recovery tooling to reprocess affected StopTransaction events and reconstruct missing transaction processing records. |
22 Jul 2026 17:30 | Recovery and reprocessing activities were completed for both GreenFlux and Eneco environments. Validation confirmed successful recovery of the vast majority of affected transactions and CDRs. |
Root Cause
The primary cause of the incident was a Microsoft Azure Event Hub platform outage in the West Europe region. This resulted in intermittent failures when GreenFlux services attempted to publish events, slowing the processing of messages across several systems.
As message processing slowed, queued messages accumulated and eventually caused one of the messaging components to reach its configured capacity limit. This increased the operational impact and contributed to delays in processing certain transactions.
During the post-incident investigation, a secondary software defect was identified in the StopTransaction processing workflow. Under specific failure conditions caused by unsuccessful Event Hub publishing attempts, transaction processing could terminate before the transaction document was fully updated. As a result, some charging sessions were left in an incomplete processing state and required subsequent reprocessing to generate their associated CDRs.
Resolution
To restore normal operations, GreenFlux:
- Increased the capacity of the affected messaging component to accommodate the backlog.
- Restored monitoring and logging capabilities to support investigation and validation.
- Verified transaction and charging session data integrity.
- Monitored system recovery following Microsoft's mitigation of the Azure Event Hub outage.
- Investigated and corrected the StopTransaction processing defect exposed by the outage.
- Implemented dedicated maintenance functionality and recovery tooling to safely reprocess affected transactions.
- Executed targeted reprocessing of impacted charging sessions and regenerated missing CDRs where required.
Recovery Results
Following the investigation and recovery effort, GreenFlux identified and reprocessed transactions affected by the Azure Event Hub outage.
A total of 12,825 transactions were identified as potentially impacted. Initial analysis showed that 5,176 GFX transactions had already been successfully processed and converted into CDRs prior to recovery activities.
Using dedicated recovery tooling, GreenFlux reprocessed the remaining affected transactions and successfully generated or recalculated 5,826 CDRs . During validation, 426 CDRs were flagged for additional review.
Following the recovery process, all identified transactions were either successfully recovered, had already been processed correctly before recovery activities began, or were flagged for further investigation. Only 13 GFX transactions remained unresolved due to pre-validation failures caused by missing authentication identifiers.
These results confirmed that the vast majority of charging session data affected by the Event Hub disruption was successfully recovered and processed.
Preventive Actions
To further improve resilience, GreenFlux is implementing the following actions:
- Enhance monitoring and alerting for message queue growth and capacity utilization.
- Review and adjust capacity settings for critical messaging components.
- Improve visibility and alerting for dependencies on external cloud services.
- Strengthen operational monitoring to ensure critical alerts remain available even when logging systems are degraded.
- Review platform resilience measures and recovery procedures for future third-party service disruptions.
- Improve failure handling and resiliency within StopTransaction processing when external dependencies are unavailable.
- Expand post-incident validation procedures to detect incomplete transaction processing states more quickly.
- Develop additional automated recovery capabilities for transaction reprocessing when dependency failures occur.
Conclusion
This incident was triggered by an external Microsoft Azure Event Hub service outage. While the outage caused temporary processing delays and affected some charging session start requests, a later investigation also revealed a software defect in the StopTransaction processing workflow that was exposed by the same event.
Following detailed analysis, GreenFlux implemented corrective measures, recovered affected transactions, and successfully regenerated the vast majority of impacted CDRs. Additional monitoring, resiliency improvements, and recovery mechanisms are being implemented to reduce the impact of similar third-party service disruptions in the future.