Dear Customers,
We identified performance degradation during peak hours (2021.05.31 - 2021.06.02)
Starting from 2021.05.31 we observed performance degradation of request processing by api-gateway cluster. Issues normally started around 14:30 CET reaching its peak at 15:40 CET and going back to normal at 17:00 CET. All subsystem behind api-gateway cluster reported normal processing time while internal api-gateway processing time increased by 10 times. As a result, customers observed increased processing time on every operation or blank screens during UI load.
Issue occurred throughout 2021.05.31 - 2021.06.02 between 14:30-17:00 CET. Due to irregular nature of this incident, we were unable to identify root-cause during first 2 occurrences. On 2021.06.02 we identified poorly performing cache service containing some very old entries that never expire (up to 3 years old). This subsystem is heavily utilized by api-gateway and normally responds under 1 millisecond, stalled entries in cache resulted in response time up to 100 milliseconds. After flushing content of this cache, everything was back to normal on 2021.06.02 16:19 CET.
This issues affected all requests coming via api-gateway (SMS5 UI and agent sales), WEB and GDS channels were not affected.
We will continue our internal analysis to make sure that such issue doesn’t occur again.
Kind regards
Worldticket Team