Improve debian version normalization

[blshkv]

I have noticed that some distributions create a random snapshot and call it a next release.
Here is an example:
https://packages.debian.org/jessie/faifa (0.2~svn82-1 instead of the official v0.1)
Eventually, repology would pick it up and mark other distributions as "outdated":
https://repology.org/metapackages/outdated-in-repo/gentoo_ovl_pentoo/

An another example is the following:
https://repology.org/metapackages/outdated-in-repo/gentoo_ovl_pentoo/?search=wfuzz
The official release is 2.1.5 (see https://github.com/xmendez/wfuzz/releases), however, 2.1beta is marked as the latest and any other versions are marked as outdated.

I suggest extracting version from the official homepage as the current behaviour can be manipulated.

[blshkv]

FYI, Gentoo has a clearer name scheme:
https://devmanual.gentoo.org/ebuild-writing/file-format/

and it would be called either 0.1_p82, or 0.1_p82 or 0.2_pre82 (or YYYYMMDD instead of 82)

[blshkv]

An another bad example is this package:
https://repology.org/metapackage/sslstrip/versions
Alpine Linux Edge came up with 0.9.2 release name, however the official version (at github) is 0.9-2 which supposed to be a post release.

[lots0logs]

There are A LOT of cases like that. Here are just a few more examples:

https://repology.org/metapackage/b43-firmware/versions
https://repology.org/metapackage/cinnamon/versions
https://repology.org/metapackage/cinnamon-control-center/versions
https://repology.org/metapackage/cinnamon-session/versions
https://repology.org/metapackage/cinnamon-settings-daemon/versions
https://repology.org/metapackage/vivaldi/versions
https://repology.org/metapackage/xed/versions

[AMDmi3]

There are actually multiple unrelated problems here:

Incorrect version specified in the package

This is not repology fault, it should be fixed in upstream repos. For instance, cinnamon clearly states 3.4.0 as version. We can't do much with it automatically, but it can be ignored with rules (see rules.d/20.verignores.yaml)

I suggest extracting version from the official homepage as the current behaviour can be manipulated.

This is planned, but this is hard. For starters, I've planned to extract versions from GitHub. We can also fetch data from Anitya.

However, upstream version will be treated the same way as any other package version, so it won't really solve this problem. This is because

• upstream may change, and repology will think that outdated version is actually upstream latest. This should not happen, as the purpose of repology is to find latest version regardless of upstream problems. Because of that, I was able to update a lot of FreeBSD ports I maintain which changed upstreams and for which newer versions could not be reported by portscout, our upstream release monitoring tool.
• there's no way to reliably parse upstream for all packages

So, I suggest to fix these with rules for now. And of course maintainers should not use fake versions.

Incorrect version normalization

We normalize versions for most repos, e.g. 1.2.3~5dfsg6 is converted to 1.2.3, as 5dfsg6 is not related to upstream version in any way. However there sometimes are cases like 1.2.3~beta4 in the same repo, and we cut of beta part erroneously. These normalization rules may need to be revised. But it may be an upstream problem too, bacause IMO a sane version schema should never mix version parts which are related to upstream and parts which are related to the package revisions.

Incorrect version comparison

2.1.5 is clearly greater than 2.1beta. This was already fixed in libversion, but it was not updated in the production. Will fix itself with next update.

[ghost]

Due to upstream versioning sometimes not following semantic version numbers, downstream (operating system in this case) would then change to a scheme of versions they agreed on.

As an example, I used to released gnurl until 3 version ago as "gnurl-7_53_0" and in Gentoo, and later Guix I had to rewrite my release version to "gnurl-7.53.0".

In Guix as far as I am aware we have clear guidelines on version numbers and nameformats. I think it would be easier to:

1. ask operating systems (one kind of repository as far as I perceive this project) what their guidelines are on these two subjects.
2. gathered data about the other kinds of repositories and their naming schemes
3. come up with a comparison of common shared schemes.

There will be cases where no scheme applies or locations change (like: on new release move old current to subfolder archive), which is what every system dislikes.

[AMDmi3]

Well since upstream version is not our problem and 2.1beta handling was fixed by switching site to https://github.com/repology/libversion 1.1.0, this issue will track issues with version normalization.

[blshkv]

Here is few others different (hopefully) cases:
https://repology.org/metapackage/dsniff/versions must be 1.2.0 beta1
https://repology.org/metapackage/libimobiledevice/versions must be 1.2.0
https://repology.org/metapackage/libusbmuxd/versions
https://repology.org/metapackage/mphidflash/versions (weird, but 1.8 is in /dist fodler)
https://repology.org/metapackage/sqlninja/versions (0.2.999-alpha1)

[AMDmi3]

Please add detail which repos have problematic versions.

[blshkv]

Could you just click on the link and compare output? The list is way too long. For example, dsniff is marked as "1.2.0" (no beta) in some distros which is not correct. Others use several variations _beta1|b1|.b1 etc.

An another observation I made is that debian use "unreleased version"-<svn|git>number . This is the same with Gentoo's _pre (instead of svn|git)
https://packages.debian.org/stable/whatweb 0.4.8~git20141014-1

[l2dy]

In https://repology.org/metapackage/keybase/versions, nix_stable used a different version scheme and made all others shown as outdated.

[AMDmi3]

In https://repology.org/metapackage/keybase/versions, nix_stable used a different version scheme and made all others shown as outdated.

Fixed in the ruleset, but will only be deployed in some days.

[l2dy]

As a counterexample of normalization, onioncat makes official snapshots, but repology.org removed snapshot details from some repos and marked others as outdated.

[AMDmi3]

but repology.org removed snapshot details from some repos and marked others as outdated.

It didn't. Debian uses incorrect version schema to start with. It should be 0.2.2.svn559, while it is 0.2.2+svn559, when + suffix is used for debian-specific stuff and debian-generated snapshots (as in 1.2.10+dfsg1-1, 2.4+cvs20120801-1, 2.2.0+dfsg-6 (picked random from other debian packages), so we're in right to cut that away). And there's also Rosa, which just uses nonexisting version 0.2.2. Fixing this with rules.

[blshkv]

Could you please dedicate more time on this issue?
A fresh example is here: https://repology.org/metapackage/gnome-keyring/versions
Alt linux came up with un-existing 3.20.1 version.

[AMDmi3]

It is existing version. I believe other samples were fixed.

[blshkv]

I can't find the official release, http://ftp.gnome.org/pub/GNOME/sources/gnome-keyring/ the latest is 3.20.0. usbmuxd is not fixed. I'll give more examples in a bit

[AMDmi3]

It is there http://ftp.acc.umu.se/pub/gnome/sources/gnome-keyring/3.20 since July 4th.

[blshkv]

usbmuxd there is no 1.1.1 (OpenWrt abused few other packages)
faifa - there is no 0.2 (ffainelli/faifa#15)
whatweb there is no 0.4.8, pentoo provide 0.4.8_pre

[blshkv]

Perfect, thanks!

There are few more:
hostapd (there is no 2016 release)
iaxclient (beta only)
libmpsse (the last commit was on 20160602 not 20161223)

[blshkv]

livecd-tools is an another tool with the same name but a different upstream:

https://admin.fedoraproject.org/pkgdb/package/rpms/livecd-tools/
https://gitweb.gentoo.org/proj/livecd-tools.git

repology mixes up these two and report the one with a lower version numbering as outdated

[blshkv]

allinux come up with the new 2.3 version of pixiewps but it was not released by the upstream
http://www.sisyphus.ru/en/srpm/Sisyphus/pixiewps/get

It is listed as outdated:
https://repology.org/metapackages/outdated-in-repo/gentoo_ovl_pentoo/

[AMDmi3]

Both fixed. Next time, please create report for a corresponding port on the website. These cases have nothing to do with version normalization really.

[AMDmi3]

To fix this, it seems that we'll have to preserve a predefined set of suffixes (something like [+~-](a|alpha|b|beta|rc|pre|patch|git|hg|svn|cvs)[0-9]+) while doing version normalization for debian repos (currently they are all removed). There's room for expansion ([+~-][0-9]+), but we should act safe to not let any garbage get into version numbers.