Incident

We have encountered instabilities in the loading of the discussion panel and mobile app for agents.

Instead of loading the expected interface on desktop, a blank screen was displayed to the agents.Regarding mobile app, an error was displayed as soon as agent IDs were entered.These regressions prevented agents from processing the incoming flow of conversations. 

These instabilities occurred :
- August 11th : 9:35 > 10:42 CEST

Reasons

We are currently working on some major innovations that will be live in a few weeks' time.The first pieces of this in-depth work are starting to be delivered in production. Although inactive for our end users, they impact the source code already in production.The incident came from one of these supposedly transparent deliveries. It introduced a new environment variable required to start the discussion panel and mobile app. 

On our build and test environments, no problems were detected as this variable is instantiated automatically on the discussion panel and mobile app login.On production servers, due to the fact that all new pieces are not online yet, the variable has not been instantiated as expected. It generated an internal error.

Resolution

We solved the problem by taking several successive actions : - Our probes and automated non-regression tests detected the incident within minutes of deployment to production. We therefore proceeded to a rollback of the release. This action did not restore the solution, as a database schema update accompanied the release.- We had to urgently develop a hotfix to bypass the database schema update.

Actions for the future

• (Done) (Tech) Add new Unit tests on build and test environments in order to check newly added environment variable integrity 
  
• (Done) (Tech) Remove dependencies between in progress developments but not visible yet and initial discussion panel/mobile apps loading