Ping/Pong WebSocket Timeout Errors in Genesys Cloud

[fparejam]

Introduction We have been encountering intermittent WebSocket connection issues in our Genesys Cloud IVR integration. The error consistently reports a timeout while waiting for a "pong" response. Below, we provide a detailed analysis of the issue and seek insights into its root cause and potential solutions.

---

Issue Analysis The logs reveal recurring occurrences of WebSocket timeout errors, specifically with error code 408 indicating that the system "Timed out waiting for pong response." Below are anonymized log illustrating the problem:

Log Trace 1

2025-02-05 11:37:44,911 - [Session-ID] - [Module] - INFO - Received text data: {"version":"2","id":"[Unique-ID]","type":"ping","seq":51,"position":"PT85.8S","parameters":{"rtt":"PT0.018108612S"},"serverseq":16}
2025-02-05 11:37:44,911 - [Session-ID] - [Module] - INFO - Sent ping: version='2' id='[Unique-ID]' type='pong' seq=17 parameters=PongParameters() clientseq=51
2025-02-05 11:37:44,940 - [Session-ID] - [Module] - INFO - Received text data: {"version":"2","id":"[Unique-ID]","type":"error","seq":52,"position":"PT90.8S","parameters":{"code":408,"message":"Timed out waiting for pong response"},"serverseq":16}
2025-02-05 11:37:44,940 - [Session-ID] - [Module] - DEBUG - Received message: {'version': '2', 'id': '[Unique-ID]', 'type': 'error', 'seq': 52, 'position': 'PT90.8S', 'parameters': {'code': 408, 'message': 'Timed out waiting for pong response'}, 'serverseq': 16}
2025-02-05 11:37:44,940 - [Session-ID] - [Module] - INFO - Received text data: {"version":"2","id":"[Unique-ID]","type":"close","seq":53,"position":"PT90.8S","parameters":{"reason":"error"},"serverseq":16}

Log Trace 2

2025-02-05 11:49:42,744 - [Session-ID] - [Module] - INFO - Received text data: {"version":"2","id":"[Unique-ID]","type":"ping","seq":44,"position":"PT32.4S","parameters":{"rtt":"PT0.019276122S"},"serverseq":7}
2025-02-05 11:49:42,744 - [Session-ID] - [Module] - INFO - Sent ping: version='2' id='[Unique-ID]' type='pong' seq=8 parameters=PongParameters() clientseq=44
2025-02-05 11:49:42,775 - [Session-ID] - [Module] - INFO - Received text data: {"version":"2","id":"[Unique-ID]","type":"error","seq":45,"position":"PT38.4S","parameters":{"code":408,"message":"Timed out waiting for pong response"},"serverseq":7}
2025-02-05 11:49:42,775 - [Session-ID] - [Module] - DEBUG - Received message: {'version': '2', 'id': '[Unique-ID]', 'type': 'error', 'seq': 45, 'position': 'PT38.4S', 'parameters': {'code': 408, 'message': 'Timed out waiting for pong response'}, 'serverseq': 7}
2025-02-05 11:49:42,776 - [Session-ID] - [Module] - INFO - Received text data: {"version":"2","id":"[Unique-ID]","type":"close","seq":46,"position":"PT38.4S","parameters":{"reason":"error"},"serverseq":7}

---

Key Observations

• Consistent Error Code: The error message Timed out waiting for pong response with 408 status is observed across multiple instances.
• Server Sequence Mismatch: Despite updating the server sequence on our end, the Genesys server continues responding with the previous sequence, indicating a possible desynchronization or unprocessed update.
• Session Termination: The WebSocket connection is closed shortly after the timeout error.

---

This issue does not occur consistently, as multiple successful ping/pong exchanges take place during runtime before the timeout occurs...

We appreciate any insights or recommendations to help resolve this issue. Thanks!

[fparejam]

Log Trace 3

2025-02-06 08:12:36,051 - [Session-ID] - [Module] - DEBUG - payload='{"version":"2","id":"[Unique-ID]","type":"playback_completed","seq":12,"position":"PT11.6S","parameters":{},"serverseq":4}'
2025-02-06 08:12:36,051 - [Session-ID] - [Module] - DEBUG - payload='{"version":"2","id":"[Unique-ID]","type":"close","seq":13,"position":"PT14.2S","parameters":{"reason":"end"},"serverseq":4}')
2025-02-06 08:12:36,051 - [Session-ID] - [Module] - DEBUG - payload='{"version":"2","id":"[Unique-ID]","type":"ping","seq":14,"position":"PT14.2S","parameters":{"rtt":"PT0.017759511S"},"serverseq":4}'
2025-02-06 08:12:36,051 - [Session-ID] - [Module] - DEBUG - payload='{"version":"2","id":"[Unique-ID]","type":"error","seq":15,"position":"PT14.2S","parameters":{"code":408,"message":"Timed out waiting for pong response"},"serverseq":4}'

Mismatch Between Documentation and Reality

The Genesys Cloud documentation on Pings and Keepalive states:

"Pings are not sent during the open transaction or close transaction."

However, in Log Trace 3, a ping message appears after the connection is supposed to be closed. This does not align with the documented behavior.

Is this intended behavior? Am I missing something? Any insights would be appreciated.

[fparejam]

@tim.smith any help would be appreciated