Incident

We had a lag issue on our Bots backend services preventing Bots managed by iAdvize from being functional on your websites.Conversations handled exclusively by humans were still functional. However, if a bot intervened in the engagement flow used by visitors, conversations stopped at the first stage of the bot scenario.

This lag issue occurred following the release of a version containing the first building blocks of a feature that will soon be available. This release successfully passed all our validation protocols. However, the increase in load that followed the release generated a significant lag in the incoming conversation ingestion service. As a consequence, these incoming conversations exceeded the maximum execution time and were discarded from processing.

This issue happened twice on November 29th : - 16:45 to 17:22 CEST - 17:27 to 17:37 CEST 

Resolution

In order to mitigate and restore the Bots services, we performed following actions: 

• Manually clean up the event overflow in the incoming conversation ingestion service 
• Rollback of the release that introduced the lag

Actions for the future

• (Done) Add parallelization processes on the events consumers in order to be reactive in cause of lags on bots
• (Done) Put a limit on the events publisher in order to prevent possible lags on bots