Channel Manager Degraded Performance
Incident Report for Mews
Postmortem

Firstly, we'd like to apologize for problems with channel manager reservation processing during night from December 6th. We'll learn from it and improve the system, infrastructure monitoring and our internal processes in order to prevent occurrences of similar issues in the future.

Cause

During planned update of the system, a performance regression was introduced into a component that is responsible for identification of travel agency emails within channel manager module. Reservation processing was therefore slowed down significantly for smaller properties and failing in bigger properties.

Action

As soon as engineers were notified about the issue, they started investigating it, found the root cause, fixed it and updated the system.

Next steps
  • We will introduce more robust performance regression tests into our test suite in order to detect performance issues before the system update even happens.
  • We will significantly improve monitoring of our infrastructure and consequent alerting in order to be aware of such issues as soon as they happen.
  • We will review internal issue escalation processes, so that similar issues would be reported to engineering team earlier, than it was in this case.
Posted Dec 08, 2017 - 17:28 CET

Resolved
This incident has been resolved.
Posted Dec 07, 2017 - 11:59 CET
Investigating
We are currently investigating a possible issue with the connection between Mews & the Channel Managers. It is possible you may see a delay in your reservations being delivered.
Posted Dec 07, 2017 - 10:08 CET