Livechat affected by release
Incident Report for Proto
Postmortem

Summary:
On September 8, 2023, following a release to production at approximately 5:00 AM, an issue was reported where users were unable to connect to the Livechat functionality. This disruption affected all clients meeting certain criteria.

To maintain the integrity and security of our systems, we cannot disclose specific details about the release and the exact nature of the incident at this time.

Immediate Actions Taken:

  1. Upon detection of the disruption, the engineering team promptly rolled back the release.
  2. After rollback, it was verified that the Livechat functionality and associated APIs were restored and operating normally.

We are committed to understanding the root cause of this incident and implementing necessary measures to prevent a recurrence in future releases.

We deeply regret the inconvenience caused to our clients and users. We sincerely appreciate your patience and understanding as we worked to resolve this issue.

Thank you for your continued trust and support.

Posted Sep 07, 2023 - 15:16 PDT

Resolved
We have confirmed that the recent release to production has caused Livechat to be disconnected for users that meet certain conditions.
Specific details will be included in the post-mortem of this incident.

The Proto engineering team has rolled back the release, and have confirmed that the issue is no longer occurring.

We would like to deliver sincere apologies for all users and clients affected by this incident.
Posted Sep 07, 2023 - 13:00 PDT