WhisperX crashes unexpectedly throughout usage

[SammyVincent-Dasion]

Hello,

I have been developing an API that uses WhisperX during a crucial part of audio processing. However, WhisperX crashes unexpectedly throughout usage (maybe after an hour or so of testing). The only thing that will fix the bug is to restart my docker container or sometimes fully restart my PC.

Some specifics:

• I am writing my API in python using uvicorn with FastAPI
• I am running the entire project in Docker
• This bug is persistent (though not replicatable) across devices and compute types, throwing different errors for each (on one device without a gpu, it simply terminates, returning -9, but no specific error, on another device with CUDA on an A500 it either gives me an "all CUDA capable devices are busy or unavailable" or a "CUDA failed with error unknown error”)
• I initially thought this might have to do with memory usage because that first no-gpu pc has 8Gb of memory, and the A500 has 4Gb. I am using the base version of WhisperX by the way.
• Upon switching devices and compute types and having the same issues (but slightly different), I now suspect that the error has something to do with my docker setup or my method of installing WhisperX.
• I will attach some of my files to help in identifying the error.

My Docker Compose File:

version: '3.7'
services:
  example_api:
    container_name: api
    #command: nvidia-smi
    restart: always
    build: ./api
    image: nvidia/cuda:12.3.1-base-ubuntu20.04
    environment:
      - ENV_TYPE=DEV
      - PYTHONDONTWRITEBYTECODE=1
    ports:
      - 8000:80
    volumes:
      - ./api/src/:/src/
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

My Dockerfile:

FROM python:3.8

RUN mkdir /src

COPY ./setup.py /
COPY ./start.sh /
COPY ./requirements.txt /
COPY ./src/ /src/

# Conform shell script to linux line endings
RUN sed -i 's/\r//g' /start.sh
RUN apt-get update && apt-get install ffmpeg -y
RUN pip install --upgrade pip; exit 0
RUN git clone https://github.com/m-bain/whisperX.git
RUN cd whisperX && pip install -e .
RUN pip install -e .

EXPOSE 80

CMD ["/bin/bash", "/start.sh"]

My actual usage of whisperX in Python:
proc = subprocess.run(['whisperx', 'tmp/output/input_trimmed.wav', '--model', 'base', '--output_dir', 'tmp/output/{}/transcripts'.format(file_name)])

Any help would be much appreciated. My next steps are probably to mess around with my Docker setup or my WhisperX installation. I could also switch my second PC to CPU compute type because this one has 32Gb ram and that would eliminate an ylingering concerns about memory.

[brvier]

Maybe with a crash log ?

[SammyVincent-Dasion]

This is a picture of what I get when I try running Whisperx in an interactive terminal. I hope this is what you meant by crash logs because I am not aware of where Whisperx keeps its own logs if they exist.

[SammyVincent-Dasion]

I tested out using --device cpu and --compute_type int8 and that seems to work fine if i just execute as a standalone command through my docker's shell, but when I make that change in my app, downstream stuff breaks unless I set it to also use cpu.

[dahifi]

@SammyVincent-Dasion Just an FYI that there is an existing ASR whisper project that you may want to look at: ahmetoner/whisper-asr-webservice#125.

I am not having any problems running this, I'm currently using docker on a Win10 install and a 3070Ti.

Hopefully that'll help you.