Replies: 14 comments 78 replies
-
What do they use to build automatically? |
Beta Was this translation helpful? Give feedback.
-
Some more information about how they likely will use the new scripts. Thus how they plan to replace other parts of Jenkins. https://github.com/vyos/vyos-build/blob/current/.github/workflows/trigger_rebuild_packages.yml By looking at this it seems they will use GitHub workflows to trigger a build via the new build scripts. The It would be trivial to write piece of python that would do the same thing - scan change-set and look for changes and trigger build for specific package if applicable and send it to reprepro. That's why we don't even need to worry about how the VyOS team does it. It's easy to write standalone replacement that doesn't use GitHub workflows. The new build scripts doesn't seem to be complete tho since they work with what vyos-build repository contains, thus it doesn't include any other package that is being built from it's own repository thus I don't believe the new system can build all packages yet. Perhaps this is testing phase before it's used to cover every package? |
Beta Was this translation helpful? Give feedback.
-
I wonder why they want to keep the action secret |
Beta Was this translation helpful? Give feedback.
-
We could make the new implementation like this:
I'm not worried about the missing pieces since we would need to rewrite those anyway (unless you want to use GitHub workflows - I don't). I want script that you can call where you want. I take this as opportunity to ditch the CI/CD - if we are getting rid of Jenkins we should get rid of the GitHub workflows/any other CI/CD too. This wasn't quite possible with Jenkins since part of the build.py is inside of the Jenkins pipelines. Thus there was no unified way to build packages standalone. The great part is, that when the VyOS gives you universal build scripts, in the shape of the build.py, then it's easy to write your own scheduler/builder around it without burden of release-to-release maintenance - once you write it, there is no need to change it since the part that changes with VyOS development is the definitions under build.py. It's still early days tho. It makes sense to start any work after the system is complete - when you can build all circinus packages and assemble usable reprepro repository until then I see the current state as experimental phase. |
Beta Was this translation helpful? Give feedback.
-
New development T6754: Delete Jenkins build packages - seems like current branch is transitioning to the new GitHub Workflow + Python build system for good. So far no progress for circinus nor sagitta. The equuleus likely won't get the new build system ever. This is tied to the change of APT repository vyos/vyos-build#780 - this tells me that they now have whole repository built by the GitHub Workflow (for current branch). It's still mystery how they build packages outside the main vyos-build repository. Even the current branch has still bunch of dependencies on packages that have their own repository and I didn't see any build script as the vyos-build has for the standalone packages. I'm missing something? For example https://github.com/vyos/ipaddrcheck was updated recently by adding new GitHub Workflow and this Workflow points to vyos/.github/.github/workflows/trigger-rebuild-repo-package.yml and this Workflow points to the mysterious They could invoke the build.py directly for "simple builds" but this script expects package.toml configuration and I don't see any .toml file that would have this configuration for these packages. So they use some other build system like invoking dpkg-buildpackage directly or what? Doesn't make much sense to me, likely I'm missing something... Do they have two separate build systems (like the one build.py and another one in pure GitHub Workflow/other hidden scripts? The new repository has these packages:
And these need to come from somewhere... I'm blind or it's just not public? EDIT: I just noticed that what both packages (ipaddrcheck, vyos-1x) share is that they don't have custom build, they invoke the Jenkins library with null command and this has fallback to dpkg-buildpackage. Thus for these packages they use the same method, that the hidden GitHub Workflow just calls |
Beta Was this translation helpful? Give feedback.
-
BTW, there is a new organization https://github.com/VyOS-Networks and it has no public repositories. Spotted in https://vyos.dev/T6708 with a comment: "All PRs created in /vyos/* repos should be mirrored to /VyOS-Networks for backporting into LTS branches." First they say:
and then they seem not too happy someone actually has done this, how nice... |
Beta Was this translation helpful? Give feedback.
-
It's interesting they are pointing out the replacement of Jenkins. Yet the mentioned workflow points to hidden workflow that does the build. I don't see the script that calls build.py, so currently the build scripts are incomplete, partially hidden away - even for |
Beta Was this translation helpful? Give feedback.
-
One worrying possibility - may or may not be true, for now this is pure speculation, hopefully not true but can't be excluded - the more closed development (some source and scripts not public, available only to those with $$$ subsciptions) means it's harder to notice when some malicious code is quietly added. As the xz backdoor has shown, it might happen even in fully open source projects, but still is easier to pull off if fewer eyes can see inside, and this project is much more complex. They might get some secret offer they couldn't refuse (say from some Russian oligarchs) who will help the project financially, but want some backdoor added in return to make it possible to spy on the users. So all these changes could be intended just to hide some potential future malicious changes. As for poking the bear, I think the bear is watching us anyway, or the reaction wouldn't be so quick (a few minutes, and my reply was removed too, even though I was just helping the fellow user who turned out to be from the same city where I was studying, said nothing offensive or off-topic or defaming the project). And the fact they get so nervous even just discussing things might mean they are indeed not honest - I had a gut feeling that something is suspicious, but was giving them the benefit of the doubt. Such threats should not be under-estimated, as a side note it's not just the bear but also the panda too (as I recently learned, adding a "wifi stick "to an on-grid PV inverter gives the server in China complete control over the inverter, not just statistics to show the user nice graphs, but also write access to any registers including remote firmware update without any action from the user, protected by a super-secret password changed every day which is just "growattYYYYMMDD", fortunately the inverter works fine without any Internet connection, it was just temporary for remote diagnostics to get warranty repair). About routing performance differences VyOS vs Debian - I think it could be because VyOS has a newer kernel and (depending on hardware) some out-of-tree NIC drivers, I don't see anything else that could result in so much difference. |
Beta Was this translation helpful? Give feedback.
-
The hidden build script I have no idea how this works for the VyOS team unless the hidden build script has numerous conditions with hardcoded build commands for specific package - if the package needs it's own build command why this isn't a part of the repository of the given package? Why there needs to be some master hidden build script that has hardcoded build commands for specific packages? I would hoped the new build system would fix the old issues so we don't need custom patches/build scripts, oh well. Of course the libpam-tacplus/libnss-tacplus/libtacplus-map combo doesn't have functional build script either but the https://github.com/dd010101/vyos-missing/blob/sagitta/packages/libnss-tacplus/build.sh works great. With some workarounds I can build the following: List of debs
Is that all? Seems like half? I scanned the vyos org and added every repository with the new GitHub Workflow to the build queue based on the workflow file. I would like to find some way how assemble the list of packages and their build script automaticaly. Maybe scanning every repository with branch is better tactic than scanning the workflows? |
Beta Was this translation helpful? Give feedback.
-
It shouldn't be any more complicating than fleshing out package.toml files for the external repos and using build.py to build those packages. The wrapper script would just detect if the package build directory already exists, and if not, source the appropriate package.toml, or even simply just calling the build routine via bash rather than build.py. I maintain a separate cloud init routine and then a general common routine which has an exception for libvyosconfig and vyos-utils, which each call for opam to be set up with this line: cloud-init is simply Otherwise, there's nothing particularly difficult and the answers are already public knowledge via the Jenkinsfiles. Mmm, don't forget to check if
build.py is an absolutely incredible evolution to this process in my opinion. For what little it's worth from me, hats off to Mr. sever sever. |
Beta Was this translation helpful? Give feedback.
-
There is one snag with the
The Here is Packages of the given circinus repository
Seems like the generic flavor is the culprit https://github.com/vyos/vyos-build/blob/circinus/data/build-flavors/generic.toml#L9 |
Beta Was this translation helpful? Give feedback.
-
Time to move on, I suppose. I'll be pulling their images out of my infrastructure imminently. |
Beta Was this translation helpful? Give feedback.
-
Hi, Do I understand correctly that if I continue to build equuleus and saggita using this vyos-jenkins repo I will still be able to get an iso with the up-to date base Debian, and only vyos-specific packages will stay frozen and outdated? In other word, we can basically continue to build and use both equuleus and saggita with more or less reliable manner, when at least the core OS components are updated? Or, this system will cease functioning, say because some specific packages can never be built or something? |
Beta Was this translation helpful? Give feedback.
-
circinus branch is what probably going to become Stream. However, no more trust in their words what Stream will be, how it's source code will remain open, how it can be built in the long run, and what functionality will be removed from Stream to be reserved for LTS exclusively. |
Beta Was this translation helpful? Give feedback.
-
New VyOS package build system begins to form - vyos/vyos-build#745
Looks like they want to migrate away from Jenkins Pipeline Syntax + bash to Python scripts. The first version did have insane duplication of build.py but they are improving this to use unified script.
That's awesome! Such unified build system in Python would simplify the build environment by a lot, it would improve the robustness of error handling and also it would reduce the overhead of maintaining and testing 👍.
Beta Was this translation helpful? Give feedback.
All reactions