What happened ?

We had a severe load issue on our Bots backend services preventing Bots from being functional on your websites.Conversations handled exclusively by humans were still functional. However, if a bot intervened in the engagement flow used by visitors, the conversation could not take place.

This load issue was caused by an unscheduled self-cleaning script performed on the Bots database engine. This cleaning was performed on a table with a large amount of data. As a consequence, critical queries needed by Bots were not able to perform in reasonable time, making the whole Bots system degraded.

This issue happened on October 26th between 11:10 to 15:47 CEST.

Resolution

Once started, the self-cleaning script cannot be stopped and must be completed. So we looked for alternative solutions.

In order to mitigate and restore the Bots services, we performed following actions: 

• Upgrade Bots database engine to a higher-performance instance type in order to let Bots critical queries to be fully executed. This action took several hours to complete. This partly explains the duration of the incident.
• Deploy a new version with patches to reduce the bots' dependence on the overloaded database.

Actions for the future

• (Done) The bots database engine upgrade significantly reduced the permanent load. We have more capacity to handle heavy loads.
• (Done) We have identified and cleaned up the data table at the origin of the self-cleaning.
• (In progress) We are setting up new probes to detect loaded databases and avoid self-cleaning script launches.