Making a better mirror

[marekm72]

I'd like to keep all stuff mirrored and backed up just in case anything ever disappears (or as an alternative to try while debugging network issues like recently vyos-cloud-init GH repo clone failing), preferably in one place (with subdirectories) for this project as well as vyos, accel-ppp, vyos-contrib, maybe others.
It seems cgit has better support for subdirectories than gitweb, and both can co-exist on the same server so I'm setting up cgit (not ready yet) - https://git.amelek.net/cgit/ - but running into some issues.

1. Many more repos so it would be nice if the "description" files were all created automatically with correct contents.
2. The script can be made to backup any github organization (vyos, accel-ppp) but doesn't work for a single user (marekm72, dd010101).
3. Broken links on the generated page, something misconfigured (investigating).
4. Found much more powerful tool https://github.com/gabrie30/ghorg but it also leaves default "description" files, and mirrors organizations but not single users (despite documentation saying it should work).
   BTW, https://github.com/vyos-contrib is another VyOS user building their own APT repo for LTS images, may be good to work together, exchange bug fixes etc.

[dd010101]

I updated the script (now named github-mirror.sh). To be more flexible and to support both orgs and users. Thus now you can use this single script to mirror both vyos and dd010101 or any other GitHub org/user. The script is simple so I don't see why to use more complicated solution written in Go, if you can use simple bash script for your use-case. This way you can fix or modify the script by yourself easily and you don't rely on another project to fix some edge-case/bug for example.

The usage now looks like this:

#!/usr/bin/env bash
set -e

githubMirror="/opt/vyos-jenkins/extras/mirror/github-mirror.sh"
export ROOT_PATH="/var/lib/git"

export NAMESPACE="vyos"
export GITHUB_KIND="org"
export GITHUB_SUBJECT="vyos"
$githubMirror

export NAMESPACE="dd010101"
export GITHUB_KIND="user"
export GITHUB_SUBJECT="dd010101"
$githubMirror

Read the github-mirror.sh for details.

I also added the description support.

The URLs in your cgit are broken because you are mixing gitweb and cgit and incompatible git backend nginx path routing (the git backend is catching the links for the cgit, that's why it's broken). I don't see the point of mixing gitweb and cgit since they provide the same functionality. Thus I updated git-mirror.md to use cgit instead of gitweb, this gives you correct nginx vhost for cgit as well.

BTW: The https://github.com/vyos-contrib doesn't look like anything we do. The packer modifies existing ISO (thus you need subscription) and the vyos-apt looks like development tool for current branch - it builds only portion of the packages from vyos-build and thus it's useless for sagitta or equuleus since those use whole array of packages from all over the place.

So far I didn't see any other public implementation that would allow you to create replacement for the VyOS repositories. I know only of people that do have their own private build system of some kind that does this - so there are alternative implementations but not public and/or easily portable for use by anyone.

[marekm72]

Thanks, https://git.amelek.net/ now changed to use cgit, the only reason for mixing was that I didn't want to break the existing working gitweb until the new cgit works properly - it should now (I hope, please test).
I see vyos-apt is for current, but that may become the new LTS later at some point (but is kind of a moving target now, as they announced VyOS Stream and deprecating Jenkins without more details what the replacement will be - hopefully not some proprietary software).

[dd010101]

Looks good. Both the cgit and as well the git backend daemon work as intended.

So far there is no sign of any progress for the new build system and this is also confirmed by the fact they still write new packages via Jenkinsfile thus I don't expect the change to arrive anytime soon - based on the priority and the experience with history of some vyos.dev tasks - this could take years... The same for the current becoming next stable... I mean you surely have your own experience with the speed of resolution of some vyos.dev tasks...

The stream-thing is also strange - I saw sagitta-stream and circinus-stream branches in the GitHub repositories but that seems to be gone now... I see them in my backups so indeed they made them and then removed some time later? Thus I have no idea what is the progress, they said they have the stream in progress and we should expect rolling from stream soon but the circinus-stream disappeared as well... 🤔

[dd010101]

@marekm72
There was weakness in the cgit configuration. The snapshots=tar.gz zip option needs to be specified before scan-path otherwise it doesn't do anything and the option to download source as archive from cgit is missing.

The /etc/cgitrc should be like this:

clone-prefix=https://git.some.tld

css=/cgit.css
logo=/cgit.png
root-title=My mirror
root-desc=Welcome to my mirror
max-repo-count=0

snapshots=tar.gz zip
section-from-path=1
scan-path=/var/lib/git/repos

virtual-root=/

You may also want to change the root-title and root-desc to something more specific.