Format to send back audio to genesys via websocket : Audio Connector

Genesys Cloud Webhooks
1

How can we send audio back to Genesys via WebSocket?

I'm using the Azure TTS service, and after generating the audioData, I convert it to a Uint8Array before sending it to Genesys. However, during a call, the audio I hear is distorted. To troubleshoot, I wrote the audioData to a file, and it played back clearly.

For context, I'm using this repository as the foundation for my audioConnector: Genesys AudioConnector Server Reference Implementation.

here is the sample code:

async getAudioBytes(text: string): Promise<Uint8Array> {
        return new Promise((resolve, reject) => {

            this.synthesizer.speakTextAsync(
                text,
                (result: sdk.SpeechSynthesisResult) => {
                    if (result.reason === sdk.ResultReason.SynthesizingAudioCompleted) {    
                        const audioData = new Uint8Array(result.audioData);
    
                        fs.writeFile("output.wav", audioData, (err) => {
                            if (err) {
                                console.error("Error writing to file:", err);
                                reject(err);
                            } else {
                                console.log(`Audio content written to file: output.wav`);
                            }
                        });
    
                        resolve(audioData);
                    } else {
                        console.error("Speech synthesis failed. Reason:", result.errorDetails);
                        reject(new Error(result.errorDetails));
                    }
                    this.synthesizer.close();
                },
                (err: any) => {
                    console.error("Error during synthesis:", err);
                    this.synthesizer.close();
                    reject(err);
                }
            );
        });
    }

can anyone help me here with this?

Thanks!

1 Like
2

Hi, @imran, are you able to get audio buffer data through audio connector and pass to azure STT. If yes does azure STT converted it to text

3

Yes, I am receiving the audio buffer from Genesys and successfully performing speech-to-text . However, for Azure STT to function, you’ll need to convert the audio buffer you receive into one of the formats supported by Azure Cognitive Services.

4

this has been resolved, please refer this thread

5

Hi , thank you for your response, just would like to check which runtime env that you were using because when i try using nodejs it doesnt work to convert the mulaw format to PCM
So can you please let me know to which format you have converted and if possible can you please provide any reference.

thanks

6

Hey Varun, i am using TS, below is the code snippet for audio conversion hope it helps.

import { WaveFile } from 'wavefile';

export class AudioConverter {
    static convertMuLawToPCM(data: Uint8Array): Uint8Array {
        const wav = new WaveFile();
        wav.fromScratch(1, 8000, '8m', data);
        wav.fromMuLaw();
        return wav.toBuffer();
    }
}
7

Hi @imran , thank you for answering and it was helpful, but can you also please provide any TS reference for speech to text in handling streams of audio buffer data that was getting from audio connector .

thanks.

8

Hey @varun , it seems like you're having a tough time figuring this out. I think you need to specify the audio format for the input stream you're creating recognizer with Azure STT.

      const audioFormat = AudioStreamFormat.getWaveFormatPCM(8000, 16, 1);
      const pushStream = AudioInputStream.createPushStream(audioFormat);
      const audioConfig = AudioConfig.fromStreamInput(pushStream);
      const recognizer = new SpeechRecognizer(speechConfig, audioConfig);
  
      recognizer.recognized = (s, e) => {
        const result = e.result;
        if (result.reason === ResultReason.RecognizedSpeech) {
          const transcript = result.text;
          console.log(`RECOGNIZED: ${transcript}`);
        }
      };
9

Thank you for your answers @imran it was very helpful

10

I am seeing that Genesys drops the Websocket connection after open event. Wondering if this could be the cause. @imran Are you aware if Genesys has an error log where I can see what could be wrong with my Audio Connector server?

11

@spaliwal Genesys will close the socket connection if it does not receive a response from the server. To keep the connection alive, you should at least respond to ping messages or send some audio data.

12

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.