Date of incident: 2025-2-18
Date reported internally: 2025-2-18
Date reported to customer: 2025-2-18
Summary of Impact
Inability to process authorization requests for any payment terminal integrations.
Detailed Timeline of Events
- 14:00 A large batch of patch operation message was done through the platform API with incorrect setting
- 14:39 First customer noticed the problem
- 16:13 The first hotfix for our Dev team is deployed
- 16:33 The second hotfix for our Dev team is deployed
- 16:54 The configuration is fixed in our Token Service
- 16:55 The issue is resolved
Confirmed Root Cause of the Incident
- Unexpected data in Charge Station settings that authorization could not handle
- CPO-EMSP mapping was updated incorrectly
Mitigation and Resolution
- Applied hotfix to handle empty string for the configuration property
- Fixed contactless mappings
Lessons Learned for System and Process Improvements
Alerts need to be improved to have a better and faster view on critical issues
Contactless-related configuration do not follow Developer review and developers are prone to making mistakes. That should be changed.