Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poetry #707

Draft
wants to merge 35 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
d14c0d6
Remove unneeded semi-colon
MatthijsBurgh Jan 31, 2024
99c734d
Bump desired pip version to 23
MatthijsBurgh Jan 31, 2024
a47c284
Use bash array correctly
MatthijsBurgh Jan 31, 2024
9d4a239
(bootstrap) no escaping needed in echo's
MatthijsBurgh Feb 1, 2024
d826e82
(check-pip-pkg-installed-version.py) small update
MatthijsBurgh Feb 5, 2024
782b172
Fix typos
MatthijsBurgh Feb 5, 2024
0ebdfd1
(tue-env) add include system sites arg for venv
MatthijsBurgh Feb 5, 2024
a823986
(bootstrap) lower-/uppercase for local/global vars
MatthijsBurgh Feb 5, 2024
b32defa
(bootstrap) add arg to include system args in venv
MatthijsBurgh Feb 5, 2024
1be0954
(tue-make) run colcon on system interpreter
MatthijsBurgh Feb 5, 2024
641bb43
(tue-get) run scripts with system interpreter
MatthijsBurgh Feb 5, 2024
ef656c0
(tue-get) consistent case styling
MatthijsBurgh Feb 5, 2024
fd933a5
(tue-get) implement pip install site
MatthijsBurgh Feb 5, 2024
f8124c7
(tue-get) PyYaml>=6
MatthijsBurgh Feb 5, 2024
d3e9c17
Extend .git-blame-ignore-revs
MatthijsBurgh Feb 5, 2024
1a66fd2
(bootstrap) consistent case styling
MatthijsBurgh Feb 5, 2024
23c70b6
Extend .git-blame-ignore-revs
MatthijsBurgh Feb 5, 2024
1867072
Bump desired pip version to 24
MatthijsBurgh Feb 5, 2024
84019c7
targetdir -> target_dir
MatthijsBurgh Feb 5, 2024
a09c725
Fix typo
MatthijsBurgh Feb 5, 2024
8f86ad0
Use var for reused file name
MatthijsBurgh Feb 7, 2024
6c2a75a
targetdir -> target_dir
MatthijsBurgh Feb 7, 2024
5e916f7
Fix typo's and var/function names
MatthijsBurgh Feb 7, 2024
80bd342
(docs) markdownlint
MatthijsBurgh Feb 7, 2024
b56f9af
(docker) Remove equal sign, when no default value
MatthijsBurgh Feb 7, 2024
07f5712
Fix typo's
MatthijsBurgh Feb 7, 2024
79b39f2
catkin/colcon run outside the venv
MatthijsBurgh Feb 9, 2024
7a8cf31
Fix typo
MatthijsBurgh Feb 9, 2024
dae83ac
(tue-install-ros) small logic fix
MatthijsBurgh Feb 9, 2024
276a6b5
(tue-get) better target args handling
MatthijsBurgh Feb 9, 2024
4601fbc
(tue-get) split pip list for different site-packages
MatthijsBurgh Feb 9, 2024
749fc9b
First working integration of poetry into tue-env
MatthijsBurgh Feb 9, 2024
c575230
(tue-get) fix update of all targets
MatthijsBurgh Feb 12, 2024
626f1e6
(tue-get) add debug logging to target resolving
MatthijsBurgh Feb 12, 2024
d23a61c
(tue-get) print debug info of creating the state file
MatthijsBurgh Feb 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .git-blame-ignore-revs
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,7 @@
d71897d20d9816b2c86bb23d2d8ba78a24622a8d
# (bootstrap) use curly brackets for vars
5c2e670137b009ec55eff5f101575bfc3652ed6c
# (tue-get) consistent case styling
ef656c09024fbdb65b56a80bc40e0fa15d7b8a2d
# (bootstrap) consistent case styling
1a66fd24500c4b9d8252eba22f7fe7fb277cefa0
10 changes: 5 additions & 5 deletions ci/build-package.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,24 +50,24 @@ fi
if [[ "${ROS_VERSION}" == 1 ]]
then
echo -e "\e[35m\e[1mCompile the package (catkin build -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCATKIN_ENABLE_TESTING=OFF)\e[0m"
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}"/src/"${PACKAGE}" && /usr/bin/python3 "$(command -v catkin)" build --this --no-status -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCATKIN_ENABLE_TESTING=OFF'
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}"/src/"${PACKAGE}" && (deactivate &>/dev/null; /usr/bin/python3 "$(command -v catkin)" build --this --no-status -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCATKIN_ENABLE_TESTING=OFF)'
else
echo -e "\e[35m\e[1mCheck for default mixin repo (colcon mixin list)\e[0m"
MIXIN_REPOS=$(docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && colcon mixin list | grep -v "^- "' | tr -d '\r' | awk -F ": " '{print $1}')
MIXIN_REPOS=$(docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && (deactivate &>/dev/null; /usr/bin/python3 -m colcon mixin list) | grep -v "^- "' | tr -d '\r' | awk -F ": " '{print $1}')
if ! echo -e "${MIXIN_REPOS}" | grep "^default$" -q
then
echo -e "\e[35m\e[1mAdd the default mixin repo (colcon mixin add default https://raw.githubusercontent.com/colcon/colcon-mixin-repository/master/index.yaml)\e[0m"
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && colcon mixin add default https://raw.githubusercontent.com/colcon/colcon-mixin-repository/master/index.yaml'
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && (deactivate &>/dev/null; /usr/bin/python3 -m colcon mixin add default https://raw.githubusercontent.com/colcon/colcon-mixin-repository/master/index.yaml)'
else
echo -e "\e[35m\e[1mDefault mixin repo already exists\e[0m"
fi

echo -e "\e[35m\e[1mUpdate colcon mixins (colcon mixin update)\e[0m"
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && colcon mixin update'
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && (deactivate &>/dev/null; /usr/bin/python3 -m colcon mixin update)'

echo -e "\e[35m\e[1mDeleting the merged install directory\e[0m"
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && rm -rf install'

echo -e "\e[35m\e[1mCompile the package (colcon build --mixin rel-with-deb-info build-testing-off)\e[0m"
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && colcon build --packages-up-to "${PACKAGE}" --mixin rel-with-deb-info build-testing-off --event-handlers desktop_notification- status- terminal_title-'
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && (deactivate &>/dev/null; /usr/bin/python3 -m colcon build --packages-up-to "${PACKAGE}" --mixin rel-with-deb-info build-testing-off --event-handlers desktop_notification- status- terminal_title-)'
fi
2 changes: 1 addition & 1 deletion ci/delete-docker-tags.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ def _get_authorization_token(self, username, password):
Actually gets the authentication token

Raises:
AuthenticationError: didn't login right
AuthenticationError: didn't log in correctly
"""
resp = self._session.post(
f"{DH_API}/users/login/",
Expand Down
10 changes: 5 additions & 5 deletions ci/test-package.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,21 +50,21 @@ if [[ "${ROS_VERSION}" == 1 ]]
then
# Build test targets
echo -e "\e[35m\e[1mBuild test targets of this package (catkin build --this --no-deps -DCATKIN_ENABLE_TESTING=ON)\e[0m"
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}"/src/"${PACKAGE}" && /usr/bin/python3 "$(command -v catkin)" build --this --no-status --no-deps -DCATKIN_ENABLE_TESTING=ON'
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}"/src/"${PACKAGE}" && (deactivate &>/dev/null; /usr/bin/python3 "$(command -v catkin)" build --this --no-status --no-deps -DCATKIN_ENABLE_TESTING=ON)'

# Run unit tests
echo -e "\e[35m\e[1mRun tests on this package (catkin test --this --no-deps -DCATKIN_ENABLE_TESTING=ON)\e[0m"
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}"/src/"${PACKAGE}" && /usr/bin/python3 "$(command -v catkin)" test --this --no-status --no-deps -DCATKIN_ENABLE_TESTING=ON'
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}"/src/"${PACKAGE}" && (deactivate &>/dev/null; /usr/bin/python3 "$(command -v catkin)" test --this --no-status --no-deps -DCATKIN_ENABLE_TESTING=ON)'
else
# Build test targets
echo -e "\e[35m\e[1mBuild test targets of this package (colcon build --packages-select ${PACKAGE} --mixin rel-with-deb-info build-testing-on)\e[0m"
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && colcon build --packages-select "${PACKAGE}" --mixin rel-with-deb-info build-testing-on --event-handlers desktop_notification- status- terminal_title-'
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && (deactivate &>/dev/null; /usr/bin/python3 -m colcon build --packages-select "${PACKAGE}" --mixin rel-with-deb-info build-testing-on --event-handlers desktop_notification- status- terminal_title-)'

# Run unit tests
echo -e "\e[35m\e[1mRun tests on this package (colcon test --packages-select ${PACKAGE} --executor sequential)\e[0m"
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && colcon test --packages-select "${PACKAGE}" --executor sequential --event-handlers desktop_notification- status- terminal_title- console_cohesion+'
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && (deactivate &>/dev/null; /usr/bin/python3 -m colcon test --packages-select "${PACKAGE}" --executor sequential --event-handlers desktop_notification- status- terminal_title- console_cohesion+)'

# Check test results
echo -e "\e[35m\e[1mCheck test results (colcon test-result --verbose)\e[0m"
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && colcon test-result --verbose'
docker exec -t tue-env bash -c 'source ~/.bashrc; cd "${TUE_SYSTEM_DIR}" && (deactivate &>/dev/null; /usr/bin/python3 -m colcon test-result --verbose)'
fi
22 changes: 14 additions & 8 deletions dockerfiles/tue-env.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,18 @@ FROM base as builder
# Build time arguments
# BRANCH is the target branch if in PULL_REQUEST mode else it is the test branch
ARG CI=false
ARG BRANCH=
ARG BRANCH
ARG PULL_REQUEST=false
ARG COMMIT=
ARG REF_NAME=
ARG COMMIT
ARG REF_NAME
# Default is empty and gives ROS1, for ROS2 use --build-arg ROS_VERSION=2
ARG ROS_VERSION=
ARG ROS_DISTRO=
ARG ROS_VERSION
ARG ROS_DISTRO
# Default is empty and will result in the default targets repo
ARG TARGETS_REPO=
ARG TARGETS_REPO
ARG CREATE_VENV=false
ARG OAUTH2_TOKEN=
ARG VENV_INCLUDE_SYSTEM_SITE_PACKAGES=false
ARG OAUTH2_TOKEN

# Inform scripts that no questions should be asked and set some environment
# variables to prevent warnings and errors
Expand Down Expand Up @@ -79,7 +80,12 @@ RUN --mount=type=ssh,uid=1000 sed -e s/return//g -i ~/.bashrc && \
export COMMIT=$COMMIT && \
export REF_NAME=$REF_NAME && \
# Run the standard installation script
source bootstrap.bash --ros-version="${ROS_VERSION}" --ros-distro="${ROS_DISTRO}" --create-virtualenv="${CREATE_VENV}"--targets-repo="${TARGETS_REPO}" && \
source bootstrap.bash \
--ros-version="${ROS_VERSION}" \
--ros-distro="${ROS_DISTRO}" \
--create-virtualenv="${CREATE_VENV}" \
--virtualenv-include-system-site-packages="${VENV_INCLUDE_SYSTEM_SITE_PACKAGES}" \
--targets-repo="${TARGETS_REPO}" && \
# Make tue-env to be available to the environment
source ~/.bashrc && \
# Install target ros
Expand Down
28 changes: 19 additions & 9 deletions docs/CI_Travis_Setup.md
Original file line number Diff line number Diff line change
@@ -1,42 +1,52 @@
# Setting up SSH support on Travis CI
The travis configuration needs changes to recurse submodules from private gitlab repository.

The travis configuration needs changes to recurse submodules from private gitlab repository.

Steps to enable ssh support with travis and gitlab are as follows:

## On the local PC
1. Generate a ssh keypair to be used with CI
```

1. Generate an ssh keypair to be used with CI

```bash
mkdir -p ~/travis
ssh-keygen -t rsa -b 4096 -N "" -f ~/travis/deploy_key
```

1. Add the ssh key to travis
```
2. Add the ssh key to travis

```bash
gem install travis
cd <REPOSITORY>
travis encrypt-file ~/travis/deploy_key --pro --add
```

The last command will automatically generate and set two environment variables `encrypted_<STRING>_key` and
`encrypted_<STRING>_iv`in the repository settings of travis. They are not to be removed.

1. Change the value of `-out` option in `before_install: - openssl` to `./deploy_key`
3. Change the value of `-out` option in `before_install: - openssl` to `./deploy_key`

1. Commit the updated .travis.yml file and the generated file `deploy_key.enc`
4. Commit the updated .travis.yml file and the generated file `deploy_key.enc`

### For targets repository

1. Append the following lines to .travis.yml under `before_install`
```

```bash
- eval "$(ssh-agent -s)"
- chmod 600 ./deploy_key
- ssh-add ./deploy_key
```

### For package repository

1. Add the following lines to .travis.yml under `install`
```

```bash
- SSH_KEY_PRIVATE=$(cat deploy_key)
- bash install-package.sh --package=$PACKAGE --branch=$TRAVIS_BRANCH --commit=$TRAVIS_COMMIT --pullrequest=$TRAVIS_PULL_REQUEST --ssh --ssh-key="$SSH_KEY_PRIVATE"
```

## On GITLAB

1. In the repository Settings > CI/CD > Deploy Keys, add the public key from the file ~/travis/deploy_key.pub
Loading