Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add base requirements #238

Merged
merged 2 commits into from
Mar 23, 2024
Merged

Add base requirements #238

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
83e2efd
Merge pull request #237 from tattle-made/hotfix
duggalsu Mar 22, 2024
810a45e
fix: Added pinned pip package
duggalsu Mar 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Please create a new Discussion [here](https://github.com/tattle-made/tattle-api/
```
# Install locally in venv
$ cd src/api/
$ pip install -r requirements.txt
$ pip install --require-hashes --no-deps -r requirements.txt
```


Expand Down Expand Up @@ -67,8 +67,8 @@ Please create a new Discussion [here](https://github.com/tattle-made/tattle-api/
```
# Install locally in venv
$ cd src/api/core/operators/
$ pip install -r image_vec_rep_resnet_requirements.txt
$ pip install -r vid_vec_rep_resnet_requirements.txt
$ pip install --require-hashes --no-deps -r image_vec_rep_resnet_requirements.txt
$ pip install --require-hashes --no-deps -r vid_vec_rep_resnet_requirements.txt
..
# Create the docker containers
$ cd src/api/
Expand Down Expand Up @@ -127,13 +127,13 @@ Note:
```bash
$ cd src/
$ pip install --upgrade pip-tools
$ TMPDIR=<temp_dir> pip-compile --verbose --generate-hashes --emit-index-url --emit-find-links requirements.in
$ TMPDIR=<temp_dir> pip-compile --verbose --allow-unsafe --generate-hashes --emit-index-url --emit-find-links requirements.in

# Updating operators
$ cd src/core/operators/
# The link for torch is required since PyPi only hosts the GPU version of torch packages.
$ TMPDIR=<temp_dir> pip-compile --verbose --generate-hashes --emit-index-url --emit-find-links --find-links https://download.pytorch.org/whl/torch_stable.html vid_vec_rep_resnet_requirements.in
$ TMPDIR=<temp_dir> pip-compile --verbose --generate-hashes --emit-index-url --emit-find-links --find-links https://download.pytorch.org/whl/torch_stable.html audio_vec_embedding_requirements.in
$ TMPDIR=<temp_dir> pip-compile --verbose --allow-unsafe --generate-hashes --emit-index-url --emit-find-links --find-links https://download.pytorch.org/whl/torch_stable.html vid_vec_rep_resnet_requirements.in
$ TMPDIR=<temp_dir> pip-compile --verbose --allow-unsafe --generate-hashes --emit-index-url --emit-find-links --find-links https://download.pytorch.org/whl/torch_stable.html audio_vec_embedding_requirements.in
```

#### Modify generated `requirements.txt` for platform specific torch packages
Expand Down Expand Up @@ -199,7 +199,7 @@ torchvision==0.17.0+cpu; platform_machine=='x86_64' \
This is useful to update dependencies e.g. when using `pip-audit`

```bash
$ TMPDIR=<temp_dir> pip-compile --verbose --generate-hashes --find-links https://download.pytorch.org/whl/torch_stable.html --upgrade-package <package>==<version> --upgrade-package <package>
$ TMPDIR=<temp_dir> pip-compile --verbose --allow-unsafe --generate-hashes --find-links https://download.pytorch.org/whl/torch_stable.html --upgrade-package <package>==<version> --upgrade-package <package>

```

Expand Down
5 changes: 4 additions & 1 deletion src/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,10 @@ WORKDIR /usr/app
RUN python -m venv /usr/app/venv && chown -R python:python /usr/app/venv
ENV PATH="/usr/app/venv/bin:$PATH"

RUN pip install --no-cache-dir --upgrade pip
# base requirements file
COPY --chown=python:python base_requirements.txt /usr/app/base_requirements.txt
RUN pip install --no-cache-dir --require-hashes --no-deps -r /usr/app/base_requirements.txt

RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install-recommends vim curl
# RUN apt-get install -y ffmpeg
# RUN apt-get update && \
Expand Down
5 changes: 3 additions & 2 deletions src/Dockerfile.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,14 @@ RUN python -m venv /home/python/app/venv \
# Set venv in path
ENV PATH="/home/python/app/venv/bin:$PATH"

# Copy core and operator requirements
# Copy base, core and operator requirements
COPY --chown=python:python base_requirements.txt /home/python/app/base_requirements.txt
COPY --chown=python:python requirements.txt /home/python/app/requirements.txt
COPY --chown=python:python ./core/operators/vid_vec_rep_resnet_requirements.txt /home/python/app/core/operators/vid_vec_rep_resnet_requirements.txt
COPY --chown=python:python ./core/operators/audio_vec_embedding_requirements.txt /home/python/app/core/operators/audio_vec_embedding_requirements.txt

# Run pip install
RUN pip install --no-cache-dir --upgrade pip \
RUN pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/base_requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/core/operators/vid_vec_rep_resnet_requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/core/operators/audio_vec_embedding_requirements.txt
Expand Down
1 change: 1 addition & 0 deletions src/base_requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pip==24.0
12 changes: 12 additions & 0 deletions src/base_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile --allow-unsafe --generate-hashes base_requirements.in
#

# The following packages are considered to be unsafe in a requirements file:
pip==24.0 \
--hash=sha256:ba0d021a166865d2265246961bec0152ff124de910c5cc39f1156ce3fa7c69dc \
--hash=sha256:ea9bd1a847e8c5774a5777bb398c19e80bcd4e2aa16a4b301b718fe6f593aba2
# via -r base_requirements.in
4 changes: 3 additions & 1 deletion src/benchmark/audiovec/Dockerfile.audio_vec_embedding
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@ WORKDIR /usr/app
RUN python -m venv /usr/app/venv && chown -R python:python /usr/app/venv
ENV PATH="/usr/app/venv/bin:$PATH"

RUN pip install --no-cache-dir --upgrade pip
# base requirements file
COPY --chown=python:python base_requirements.txt /usr/app/base_requirements.txt
RUN pip install --no-cache-dir --require-hashes --no-deps -r /usr/app/base_requirements.txt

# audio requirments file
COPY --chown=python:python ./core/operators/audio_vec_embedding_requirements.txt /usr/app/core/operators/audio_vec_embedding_requirements.txt
Expand Down
4 changes: 3 additions & 1 deletion src/benchmark/audiovec/Dockerfile.audio_vec_embedding_graviton
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@ WORKDIR /usr/app
RUN python -m venv /usr/app/venv && chown -R python:python /usr/app/venv
ENV PATH="/usr/app/venv/bin:$PATH"

RUN pip install --no-cache-dir --upgrade pip
# base requirements file
COPY --chown=python:python base_requirements.txt /usr/app/base_requirements.txt
RUN pip install --no-cache-dir --require-hashes --no-deps -r /usr/app/base_requirements.txt

# audio requirments file
COPY --chown=python:python ./core/operators/audio_vec_embedding_requirements.txt /usr/app/core/operators/audio_vec_embedding_requirements.txt
Expand Down
5 changes: 4 additions & 1 deletion src/benchmark/imgvec/Dockerfile.image_vec_rep_resnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,10 @@ WORKDIR /usr/app
RUN python -m venv /usr/app/venv && chown -R python:python /usr/app/venv
ENV PATH="/usr/app/venv/bin:$PATH"

RUN pip install --no-cache-dir --upgrade pip
# base requirements file
COPY --chown=python:python base_requirements.txt /usr/app/base_requirements.txt
RUN pip install --no-cache-dir --require-hashes --no-deps -r /usr/app/base_requirements.txt

COPY --chown=python:python ./core/operators/image_vec_rep_resnet_requirements.txt /usr/app/core/operators/image_vec_rep_resnet_requirements.txt
RUN pip install --no-cache-dir --require-hashes --no-deps -r /usr/app/core/operators/image_vec_rep_resnet_requirements.txt
COPY --chown=python:python ./core/operators/image_vec_rep_resnet.py /usr/app/core/operators/image_vec_rep_resnet.py
Expand Down
5 changes: 4 additions & 1 deletion src/benchmark/vidvec/Dockerfile.vid_vec_rep_resnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,10 @@ WORKDIR /usr/app
RUN python -m venv /usr/app/venv && chown -R python:python /usr/app/venv
ENV PATH="/usr/app/venv/bin:$PATH"

RUN pip install --no-cache-dir --upgrade pip
# base requirements file
COPY --chown=python:python base_requirements.txt /usr/app/base_requirements.txt
RUN pip install --no-cache-dir --require-hashes --no-deps -r /usr/app/base_requirements.txt

# video requirments file
COPY --chown=python:python ./core/operators/vid_vec_rep_resnet_requirements.txt /usr/app/core/operators/vid_vec_rep_resnet_requirements.txt
RUN pip install --no-cache-dir --require-hashes --no-deps -r /usr/app/core/operators/vid_vec_rep_resnet_requirements.txt
Expand Down
4 changes: 3 additions & 1 deletion src/benchmark/vidvec/Dockerfile.vid_vec_rep_resnet_graviton
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@ WORKDIR /usr/app
RUN python -m venv /usr/app/venv && chown -R python:python /usr/app/venv
ENV PATH="/usr/app/venv/bin:$PATH"

RUN pip install --no-cache-dir --upgrade pip
# base requirements file
COPY --chown=python:python base_requirements.txt /usr/app/base_requirements.txt
RUN pip install --no-cache-dir --require-hashes --no-deps -r /usr/app/base_requirements.txt

### AWS Graviton Optimization ###

Expand Down
13 changes: 8 additions & 5 deletions src/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile --generate-hashes requirements.in
# pip-compile --allow-unsafe --generate-hashes requirements.in
#
blinker==1.7.0 \
--hash=sha256:c3f865d4d54db7abc53758a01601cf343fe55b84c1de4e3fa910e420b438d5b9 \
Expand Down Expand Up @@ -1203,7 +1203,10 @@ zope-interface==6.1 \
--hash=sha256:fddbab55a2473f1d3b8833ec6b7ac31e8211b0aa608df5ab09ce07f3727326de
# via gevent

# WARNING: The following packages were not pinned, but pip requires them to be
# pinned when the requirements file includes hashes and the requirement is not
# satisfied by a package already installed. Consider using the --allow-unsafe flag.
# setuptools
# The following packages are considered to be unsafe in a requirements file:
setuptools==69.2.0 \
--hash=sha256:0ff4183f8f42cd8fa3acea16c45205521a4ef28f73c6391d8a25e92893134f2e \
--hash=sha256:c21c49fb1042386df081cb5d86759792ab89efca84cf114889191cd09aacc80c
# via
# zope-event
# zope-interface
5 changes: 3 additions & 2 deletions src/worker/audiovec/Dockerfile.audio_worker
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,13 @@ RUN python -m venv /home/python/app/venv \
# Set venv in path
ENV PATH="/home/python/app/venv/bin:$PATH"

# Copy core and operator requirements
# Copy base, core and operator requirements
COPY --chown=python:python base_requirements.txt /home/python/app/base_requirements.txt
COPY --chown=python:python requirements.txt /home/python/app/requirements.txt
COPY --chown=python:python ./core/operators/audio_vec_embedding_requirements.txt /home/python/app/core/operators/audio_vec_embedding_requirements.txt

# Run pip install
RUN pip install --no-cache-dir --upgrade pip \
RUN pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/base_requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/core/operators/audio_vec_embedding_requirements.txt

Expand Down
5 changes: 3 additions & 2 deletions src/worker/audiovec/Dockerfile.audio_worker_graviton
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,13 @@ RUN python -m venv /home/python/app/venv \
# Set venv in path
ENV PATH="/home/python/app/venv/bin:$PATH"

# Copy core and operator requirements
# Copy base, core and operator requirements
COPY --chown=python:python base_requirements.txt /home/python/app/base_requirements.txt
COPY --chown=python:python requirements.txt /home/python/app/requirements.txt
COPY --chown=python:python ./core/operators/audio_vec_embedding_requirements.txt /home/python/app/core/operators/audio_vec_embedding_requirements.txt

# Run pip install
RUN pip install --no-cache-dir --upgrade pip \
RUN pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/base_requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/core/operators/audio_vec_embedding_requirements.txt

Expand Down
5 changes: 3 additions & 2 deletions src/worker/hash/Dockerfile.hash_worker
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,12 @@ RUN python -m venv /home/python/app/venv \
# Set venv in path
ENV PATH="/home/python/app/venv/bin:$PATH"

# Copy core and operator requirements
# Copy base, core and operator requirements
COPY --chown=python:python base_requirements.txt /home/python/app/base_requirements.txt
COPY --chown=python:python requirements.txt /home/python/app/requirements.txt

# Run pip install
RUN pip install --no-cache-dir --upgrade pip \
RUN pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/base_requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/requirements.txt

#####################################
Expand Down
5 changes: 3 additions & 2 deletions src/worker/hash/Dockerfile.hash_worker_graviton
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,12 @@ RUN python -m venv /home/python/app/venv \
# Set venv in path
ENV PATH="/home/python/app/venv/bin:$PATH"

# Copy core and operator requirements
# Copy base, core and operator requirements
COPY --chown=python:python base_requirements.txt /home/python/app/base_requirements.txt
COPY --chown=python:python requirements.txt /home/python/app/requirements.txt

# Run pip install
RUN pip install --no-cache-dir --upgrade pip \
RUN pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/base_requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/requirements.txt

#####################################
Expand Down
5 changes: 3 additions & 2 deletions src/worker/vidvec/Dockerfile.video_worker
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,13 @@ RUN python -m venv /home/python/app/venv \
# Set venv in path
ENV PATH="/home/python/app/venv/bin:$PATH"

# Copy core and operator requirements
# Copy base, core and operator requirements
COPY --chown=python:python base_requirements.txt /home/python/app/base_requirements.txt
COPY --chown=python:python requirements.txt /home/python/app/requirements.txt
COPY --chown=python:python ./core/operators/vid_vec_rep_resnet_requirements.txt /home/python/app/core/operators/vid_vec_rep_resnet_requirements.txt

# Run pip install
RUN pip install --no-cache-dir --upgrade pip \
RUN pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/base_requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/core/operators/vid_vec_rep_resnet_requirements.txt

Expand Down
5 changes: 3 additions & 2 deletions src/worker/vidvec/Dockerfile.video_worker_graviton
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,13 @@ RUN python -m venv /home/python/app/venv \
# Set venv in path
ENV PATH="/home/python/app/venv/bin:$PATH"

# Copy core and operator requirements
# Copy base, core and operator requirements
COPY --chown=python:python base_requirements.txt /home/python/app/base_requirements.txt
COPY --chown=python:python requirements.txt /home/python/app/requirements.txt
COPY --chown=python:python ./core/operators/vid_vec_rep_resnet_requirements.txt /home/python/app/core/operators/vid_vec_rep_resnet_requirements.txt

# Run pip install
RUN pip install --no-cache-dir --upgrade pip \
RUN pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/base_requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/requirements.txt \
&& pip install --no-cache-dir --require-hashes --no-deps -r /home/python/app/core/operators/vid_vec_rep_resnet_requirements.txt

Expand Down
Loading