Bot and Livechat down

Incident Report for Proto

Postmortem

The release of June 13, 2023 included additions to our Gateway websocket handlers, which ended up dramatically affecting the Gateway, and Gateway websocket services. These services are responsible for various channel/chatbot communication, and our livechat feature.

These problems were not replicable during our previous phases of testing environments, and we have reverted the platform to its previous version before our release on June 13, 2023 until we can better address this issue.

We will update our testing environments to replicate the issues raised in this incident, and hope to deliver our features in a more safe, reliable manner.

We deliver our sincerest apologies for the incident, and will make sure that the same issue does not happen again.

Posted Jun 14, 2023 - 09:10 PDT

Resolved

We have identified that Bot and Livechat are now functional.
However, Livechat may be experiencing some slowness. We have created another incident and will continue to update.

Posted Jun 13, 2023 - 08:10 PDT

Monitoring

We have reverted our latest release. We are observing that all parts of the platform are back to normal except Livechat, which is taking longer to load than usual. We are continuing to monitor our systems for signs of full recovery.

Posted Jun 13, 2023 - 07:45 PDT

Update

We are continuing to work on a fix for this issue.

Posted Jun 13, 2023 - 07:30 PDT

Identified

We have noticed that bots and livechat are not operating as expected. We are currently reverting our latest release.

Posted Jun 13, 2023 - 07:30 PDT

This incident affected: Platform (Chatbots, Livechat).