On September 16th we experienced service disruptions between 10:55am and 12:55pm (UTC+2) on our platform, impacting all iAdvize services.
We quickly identified the problem was located on our software's orchestrator cluster. For an unknown reason it was causing orchestrated jobs some unwanted restarts or stops.
After further investigations we discovered this misbehaviour was caused by a bug on our software's orchestrator cluster. As a consequence of this bug all our instances were stuck in a restart loop.
After identifying the problem, the following corrective actions have been performed:
When the restart of all our services was complete, everything went back to normal at 12:55pm (UTC+2) and agents could receive incoming contacts.
The bugfix we applied will prevent this issue from happening in the future, no further actions have been identified after this incident.