Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#BE-3905 Added FAQ for Cleaning Temp Bucket on MinIO #616

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
227 changes: 227 additions & 0 deletions docs/self-hosted-appcircle/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ sidebar_position: 4
sidebar_label: Troubleshooting & FAQ
---

import SpacetechExampleInfo from '@site/docs/self-hosted-appcircle/configure-server/\_spacetech-example-info.mdx';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

# Overview

This section is designed to help you quickly find answers to common questions and provide you with a better understanding of Appcircle server and runner.
Expand Down Expand Up @@ -292,6 +296,229 @@ If you are sure that **PAT** has the required permissions, you should check the

For more details about configuring the outbound requests, you can refer to the [Outbound Requests](/build/manage-the-connections/adding-a-build-profile/connecting-to-gitlab#outbound-requests) section.

### Checking and Cleaning the MinIO Temporary Buckets

Appcircle stores the build outputs, like logs, APKs, and IPA files, in the MinIO buckets.

If you want to check the MinIO bucket sizes, you can follow the commands below.

Copy link
Contributor

@csonuryilmaz csonuryilmaz Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add a caution or info that says this FAQ can be used on Appcircle server after MinIO migration.

Add a link to MinIO migration page for the snsd concept.

- Navigate to the Appcircle server directory.

```bash
cd appcircle-server
```

- Get the MinIO container name.

<SpacetechExampleInfo/>


<Tabs
defaultValue="docker"
groupId="container-engine"
values={[
{label: 'Docker', value: 'docker'},
{label: 'Podman', value: 'podman'},
]}>

<TabItem value="docker">

```bash
docker ps --filter "name=snsd" --format "{{.Names}}"
```

</TabItem>

<TabItem value="podman">

```bash
podman ps --filter "name=snsd" --format "{{.Names}}"
```


</TabItem>

</Tabs>

- Add the `jq` dependency:

```bash
export PATH=$PATH:$(pwd)/deps/bin
```


- Retrieve the MinIO secret key.

```bash
yq '.minio.secretKey' projects/spacetech/export/.global.yaml
```

<Tabs
defaultValue="docker"
groupId="container-engine"
values={[
{label: 'Docker', value: 'docker'},
{label: 'Podman', value: 'podman'},
]}>

<TabItem value="docker">

- Configure the `mc` client.

:::tip
Please replace `yourMinioSecretKey` with your own MinIO secret key.
:::

```bash
docker exec -it spacetech-snsd-1 mc config host add local http://localhost:9000 admin yourMinioSecretKey
```

- Get the bucket sizes.

```bash
docker exec -it spacetech_snsd_1 mc du -d 2 local
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐞 Container name is wrong here.

Screenshot 2024-11-05 at 13 51 46

```

</TabItem>

<TabItem value="podman">

- Configure the `mc` client.

:::tip
Please change the `yourMinioSecretKey` with your own MinIO secret key.
:::

```bash
podman exec -it spacetech_snsd_1 mc config host add local http://localhost:9000 admin yourMinioSecretKey
```

- Get the bucket sizes:

```bash
podman exec -it spacetech_snsd_1 mc du -d 2 local
```

</TabItem>

</Tabs>

Sample output should be as follows:

```text
0B 0 objects appcircle-local-resource-agent-cache
0B 0 objects appcircle-local-resource-backup
3.4GiB 1542 objects appcircle-local-resource-build
132MiB 89 objects appcircle-local-resource-distribution
1.3GiB 125 objects appcircle-local-resource-publish
241MiB 22 objects appcircle-local-resource-store
128KiB 29 objects appcircle-local-resource-storesubmit
224KiB 17 objects appcircle-local-resource-temp
5.1GiB 1824 objects
```

:::caution
You shouldn't edit the buckets manually except the `appcircle-local-resource-temp` one.
:::

If you see that the temporary bucket is too large, like more than 10 GB, files 7 days old or older can be cleaned with the following method:

- Create the MinIO cleanup file.

```bash
vi clean-minio.sh
```

- Paste the content below into the file and save it. Don't forget to edit the `MINIO_SECRET_KEY` variable according to yours.

<details>
<summary>Click to see the code block.</summary>

<p>

```bash
#!/bin/bash
MINIO_ENDPOINT="http://localhost:9000"
MINIO_ACCESS_KEY="admin"
MINIO_SECRET_KEY="yourMinioSecretKey" # Change this for your needs.
BUCKET_NAME="appcircle-local-resource-temp"
THRESHOLD=$(date -d "7 days ago" +%s)
mc config host add myminio $MINIO_ENDPOINT $MINIO_ACCESS_KEY $MINIO_SECRET_KEY
OBJECTS=$(mc ls myminio/$BUCKET_NAME --json | jq -r '.key')
for OBJECT in $OBJECTS; do
LAST_MODIFIED=$(mc stat myminio/$BUCKET_NAME/$OBJECT --json | jq -r '.lastModified')
LAST_MODIFIED_SECONDS=$(date -d "$LAST_MODIFIED" +%s)
if [ $LAST_MODIFIED_SECONDS -lt $THRESHOLD ]; then
echo "Deleting $OBJECT"
mc rm "myminio/$BUCKET_NAME/$OBJECT"
fi
done
```

</p>
</details>

- Make the script executable.

```bash
chmod +x clean-minio.sh
```

<Tabs
defaultValue="docker"
groupId="container-engine"
values={[
{label: 'Docker', value: 'docker'},
{label: 'Podman', value: 'podman'},
]}>

<TabItem value="docker">

- Copy the script to the MinIO container.

```bash
docker cp clean-minio.sh spacetech-snsd-1:/
```

- Copy the `jq` tool to the MinIO container.

```bash
docker cp deps/bin/jq spacetech-snsd-1:/usr/local/bin/jq
```

- Run the cleanup script to delete old files:

```bash
docker exec -it spacetech-snsd-1 /clean-minio.sh
```


</TabItem>

<TabItem value="podman">

- Copy the script to the MinIO container.

```bash
podman cp clean-minio.sh spacetech_snsd_1:/
```

- Copy the `jq` tool to the MinIO container.

```bash
podman cp deps/bin/jq spacetech_snsd_1:/usr/local/bin/jq
```

- Run the cleanup script to delete old files:

```bash
podman exec -it spacetech_snsd_1 /clean-minio.sh
```

</TabItem>

</Tabs>

## Appcircle Runner FAQ

### We are facing a self-signed certificate error on builds.
Expand Down
Loading