Service disruption

Incident Report for WorldTicket Operational Status

Postmortem

One of our virtual machines hosting the primary mail server (MX) for outgoing emails was lost during the internal migration process performed by our hosting vendor (unable to boot, unable to access via KVM etc.)

This wasn’t detected in a timely manner, because this primary MX was also used to send alarms by our monitoring system.

Please see a timeline report :

2021.11.04 08:41:00 UTC.

In order to mitigate this issue we had to switch to our backup mail MX server.

2021.11.04 08:41:00 UTC

After switching to the secondary server we noticed new emails sent normally, but some email vendors deferred accepting emails and those were queued on our side, up to 10% of total volume mostly by outlook hosted emails. This happened because our backup server was a cold backup and it’s IP didn’t have any reputation and it was immediately greylisted by some servers.

2021.11.04 10:06 UTC

We assigned the IP of the primary email server to the secondary email server and all queued emails were sent out.

2021.11.04 15:00 UTC

All ticket confirmation emails were sent. We are working on reactivating sendgrid email transport for customers that were using it.

We are now working on improving the reliability of our email transport subsystem.

Posted Nov 05, 2021 - 10:45 CET

Resolved

This incident has been resolved.

Posted Nov 05, 2021 - 09:22 CET

Update

Dear customers

One of our virtual machines hosting the primary mail server (MX) for outgoing emails was lost during the internal migration process performed by our hosting vendor (unable to boot, unable to access via KVM etc.) This wasn’t detected in a timely manner, because this primary mail server MX was also used to send alarms by our monitoring system.

Please see a timeline report :

2021.11.04 08:41:00 UTC.
In order to mitigate this issue we had to switch to our backup mail MX server.

2021.11.04 08:41:00 UTC
After switching to the secondary server we noticed new emails sent normally, but some email vendors deferred accepting emails and those were queued on our side. This happened because our backup server was a cold backup and it’s IP didn’t have any reputation.

2021.11.04 10:06 UTC
We assigned the IP of the primary email server to the secondary email server and all queued emails were sent out.

We aim to resend all ticket issue confirmation emails for tickets issued between 2021.11.04 00:00:00 UTC and 2021.11.04 10:06:00 UTC by the end of the day

Final update regarding resending emails for issued tickets, will be posted on our status page when the task is completed.

We are sorry for the inconvenience

Best regards

Worldticket team

Posted Nov 04, 2021 - 12:09 CET

Monitoring

Dear Customers,

Due to IBM mail server breakdown no mails have been sent from SMS. We are now using backup server and investigating resend possibilities.

We will come back with further updates.

Thank you for your understanding

Best regards,

Worldticket Team

Posted Nov 04, 2021 - 10:15 CET

Investigating

Dear Customers,

We are currently experiencing a service disruption which may cause emails not being sent from SMS. Our Operations team is working to identify the root cause and implement a solution.

We will send an additional update as soon as possible.

We apologize for the inconvenience

Best Regards,

Worldticket Team

Posted Nov 04, 2021 - 09:20 CET

This incident affected: SMS Modules (Reservations, Queues, Pricing, Payment & Ticketing, Inventory & Schedules, Security (Roles & Users), Reporting, Localization).