Predictions text to speech throws error related to ArrayBuffer (React Native)

[hoangtam101]

Hi everybody,

I trying to create an example using aws-amplify to handle text-to-speech.
So I follow the document: https://docs.amplify.aws/lib/predictions/text-speech/q/platform/react-native/#set-up-the-backend

Then I run the example I got the message [Error: FileReader.readAsArrayBuffer is not implemented]

Here is my code for calling Predictions to convert text to speech.
Predictions.convert({ textToSpeech: { source: { text: 'Hello. Text to speak', }, voiceId: 'Ivy', }, }) .then(result => {}) .catch(err => console.log({ err }));

Can anyone help me?

Thank a lot.

[nadetastic]

Hi @hoangtam101 thank you for opening this issue - Question i have for you is, are you doing anything to consume the result from Predictions.convert()?

[hoangtam101]

Hi @nadetastic For example I just want to code is work and log the result to know the data.
So the current code does not work, I still got an error message above.

Thanks,

[hoangtam101]

Hello @nadetastic
Can you give me an example or code to run it success?

I stuck with it.
WARN Possible Unhandled Promise Rejection (id: 0): Error: FileReader.readAsArrayBuffer is not implemented

[nadetastic]

@hoangtam101 I spent some time digging into this and found a potential problem that affects how Predictions.convert() for text-to-speech works in React Native. In short the library uses ArrayBuffer to create a blob from the response - and the issue is React Native doesn't support ArrayBuffer conversions very well.

Im transferring this over to the Javascript repo to better continue investigating this issue.

[nadetastic]

Hi @hoangtam101 we've identified this as a bug with the library and are continuing to investigate a fix for this. In short, the implementation in place uses ArrayBuffer conversions which works as expected for web, however RN BlobManager doesn't supports ArrayBuffer conversions very well.

amplify-js/packages/predictions/src/Providers/AmazonAIConvertPredictionsProvider.ts

Lines 141 to 157 in 28bc873

	try {
	const data = await this.pollyClient.send(synthesizeSpeechCommand);
	const response = new Response(data.AudioStream as ReadableStream);
	const arrayBuffer = await response.arrayBuffer();
	const blob = new Blob([arrayBuffer], {
	type: data.ContentType,
	});
	const url = URL.createObjectURL(blob);
	return {
	speech: { url },
	audioStream: arrayBuffer,
	text: input.textToSpeech.source.text,
	} as TextToSpeechOutput;
	} catch (err) {
	return Promise.reject(err);
	}
	}

I'll update this issue once I have some additional info.

[kyokchyan]

Any updates on this issue with ArrayBuffer?

[Umoren]

I think there's a similar problem in React with the SpeechToText functionality.

async function startRecording() {
        console.log('start recording');
        audioBuffer.reset();

        window.navigator.mediaDevices.getUserMedia({ video: false, audio: true }).then((stream) => {
            const startMic = new mic();

            startMic.setStream(stream);
            startMic.on('data', (chunk) => { // error: process is not defined
                var raw = mic.toRaw(chunk);
                if (raw == null) {
                    return;
                }
                audioBuffer.addData(raw);

            });

            setRecording(true);
            setMicStream(startMic);
        });
}

[cwomack]

Hello, @hoangtam101, @Umoren, and @kyokchyan 👋. Wanted to update this issue and let you all know that we'll be consolidating this into #13322 in order to better focus on a more general investigation into the support/bugs with Predictions and React Native in v5 of Amplify. If you have any additional context/feedback, feel free to comment on that issue and follow it for updates.