JSON API Action Failure Due to Proxy Downtime

Root Cause

Azure reported an unexpected failure on the physical host of the virtual machine running the proxy service. As a result, the VM was automatically redeployed to a different host by Azure’s auto-recovery system.

After redeployment, the proxy did not automatically restart, leading to continued proxy unavailability. Since the JSON API Action in the chatbot platform is configured to route all outbound requests through this proxy for static IP purposes, all such requests failed with connection errors (e.g., 403, Connection Refused, or Timeout).

Resolution Steps

1. Azure Auto-Recovery Triggered:
   Azure initiated a VM redeployment on June 29 due to a detected hardware failure.
2. Post-Redeployment Validation Missed:
   The proxy process did not automatically restart after the VM was moved and booted.
3. Manual Intervention Performed:
   On June 30, engineers identified the root cause and manually restarted the proxy service.
4. Service Verified:
   Successful outbound proxy calls were verified using curl and internal platform tests.

Preventive Actions

Lessons Learned

• Auto-recovery of infrastructure by Azure does not guarantee application-level readiness.
• Critical services such as proxy layers should be monitored with service-level health checks post-recovery.
• JSON API actions should have retry/backoff or graceful degradation paths in case of proxy unavailability.