On October 1st 2018 we experienced service disruptions on our social platform between 00:00 and 11:30 (UTC+2).

During this incident we were unable to provide conversations on social channels (Facebook, Twitter, Whatsapp and Text Messages).

Cause of the incident :
The message broker was unreachable due to a network incident in one of our provider. This incident happened on Monday 1st, at 00:00 (UTC+2) and lasted about 30 minutes.
During these 30 minutes, some of our social services depending on this message broker were unhealthy and tried to restart automatically.
As they were unable to get healthy again, they unfortunately stayed in this state.

After identifying the problem, the following corrective actions have been performed:

• restart the social backend services that were unhealthy, so they can recover a proper connection to the message broker, since it was reachable at this time.

Everything went back to normal at 11:30 (UTC+2).

Other actions have been identified to prevent this issue from happening in the future:

• We will enhance our alerting system so these errors would be immediately notified, and actions taken quickly.