Migration from PCRE to PCRE2

[BKPepe]

Due to recent findings described in #21800, we should track somewhere (= here) packages that are still using PCRE instead of PCRE2 to propose their developers to look into PCRE2 and prepare the support (Assignee: Here maintainers of relevant packages).

---

Reasons to switch to PCRE - Perl Compatible Regular Expressions 2:

• PCRE2 was first released in 2015 to replace the API in the original PCRE library, which is now obsolete and no longer maintained. ^{1 2}
• Since OpenWrt targets to be minimalistic, there is no reason to have installed these two libraries side by side on the device, which results in being bigger and it may not fit devices with small storage.  ³

---

Packages, which needs to be taken care of:

• aircrack-ng
  Maintainer: @ZeroChaos- even though I thought this package is rather not maintained
  Steps to be done:

1. Ask aircrack-ng developer to release a new version with PCRE2 as those changes were merged to the master branch (See PR: add PCRE2 support (follow-up) aircrack-ng/aircrack-ng#2391 )
2. Seek a new maintainer for this package in OpenWrt or if not found, we should decide if we want to keep this package here or not.
3. Backporting patch is simple enough, while backporting an additional patch proposed upstream is now needed. autotools: reset PCRE CFLAGS/LIBS with both PCRE and PCRE2 present aircrack-ng/aircrack-ng#2580
4. Pending aircrack-ng: bump to release 1.7 + PCRE2 #22256

• atftp
  Maintainer: @dddaniel
  Steps to be done:

1. Backport patch from https://sourceforge.net/p/atftp/support-requests/11/
2. Someone should step it and take maintainership of this package here.
3. Pending: atftp: bump to release 0.8.0 + PCRE2 #22228

• net-snmp
  Maintainer: @stintel
  Steps to be done:

1. Waiting for Does net-snmp plan to switch to pcre2? net-snmp/net-snmp#420 Support merged but still not in latest release net-snmp/net-snmp@d3e95c8
2. Pending net-snmp: move to PCRE2 library #22254

• shadowsocks-libev
  Maintainer: @yousong
  Steps to be done:

1. Remove this package because there were efforts to port it to PCRE2 and it is not merged and as well, this package is getting to be removed from Debian. See: Upgrade to pcre2 shadowsocks/shadowsocks-libev#1788
2. Pending and under consideration: shadowsocks-libev: convert to PCRE2 #22356

• uwsgi
  Maintainer: @Ansuel
  Steps to be done:
  Oh, this time, its different! There is a pending pull request:
  add libpcre2 support unbit/uwsgi#2543
  Pending uwsgi: add experimental pcre2 patch and drop pcre #22184
• nmap
  Maintainer: @nunojpg
  Steps to be done:
  There is support since the commit nmap/nmap@828ab48 , no release was tagged since that time
  Pending: nmap: bump to 7.95 release + PCRE2 #22226
• apache
  Maintainer: @heil
  Steps to be done:

1. Is anyone interested to take maintainership of this package?
2. Support is already there https://svn.apache.org/viewvc?view=revision&revision=1898399 , so we need to
   bump this package and switch the library in the Makefile
3. Pending: apache: bump to release 2.4.57 + PCRE2 #22225

• kismet
  Maintainer: @padre-lacroix and @sourceindex
  Steps to be done:
  This package is not maintained in our repository since it was introduced here even though upstream are releasing new versions
  This package is going to be removed in kismet: drop the package #22236
• mg
  Maintainer: @nxhack
  Steps to be done:

1. Needs to be investigated further

• bandwidthd
  Maintainer: @padre-lacroix
  Steps to be done:
  Upstream repository has been archived and read-only since February 2022. See: https://github.com/NethServer/bandwidthd
  Removed this package. (Remove package: Bandwidthd #22008)
• zsh
  Maintainer: @msva
  Steps to be done:
  There is support in the upstream repository based on https://sourceforge.net/p/zsh/code/ci/10bdbd8b5b0b43445aff23dcd412f25cf6aa328a/
  Ansuel inspection: Backporting the patch is doable but results in the package not correctly generating the libs...
  Thank god someone manage to find the libs problem. Pending: zsh: use libpcre2 instead of libpcre #22197
• slang2
  Maintainer: @jefferyto
  Steps to be done:

1. Needs to be investigated further

• czmq (czmq #22007)
  Maintainer: @ja-pa
  Steps to be done:

1. Needs to be investigated further

• micropython-lib
  Maintainer: @jefferyto
  Steps to be done:

1. Needs to be investigated further
2. Pending micropython-lib: move to PCRE2 #22243

• nginx
  Maintainer: @Ansuel and @heil
  Steps to be done:

1. Since version 1.12.5 is using pcre2 by default according to https://nginx.org/en/CHANGES , but still in the Makefile, there is legacy pcre being used. Pending nginx: openresty lua module + PCRE2 switch #21018

• nginx-utils
  Maintainer: @peter-stadler
  Steps to be done:

1. Adapt source to pcre2. Pending nginx-util: move to pcre2 #22185

• postfix
  Maintainer: @Shulyaka
  Steps to be done:

1. Pending postfix: bump to 3.8.2 release + PCRE2 #22533

• zabbix
  Maintainer: @champtar
  Steps to be done:

1. Pending zabbix: move to PCRE2 library #22534

• fdm
  Maintainer:
  Steps to be done:

1. Pending fdm: update to 2.2 release and switch to PCRE2 #22536

• freeradius3
  Maintainer: @odmdas / @neheb
  Steps to be done:

1. Pending freeradius3: switch to pcre2 #22532

• haproxy
  Maintainer:
  Steps to be done:

1. Pending haproxy: move to PCRE2 #22537

• libndpi
  Maintainer:
  Steps to be done:

1. Need further investigation.
2. Pending: libndpi: bump to release 4.8 + PCRE2 #22570

• privoxy
  Maintainer:
  Steps to be done:

1. Pending privoxy: update to release 3.0.34 + PCRE2 #22539

• tvheadend
  Maintainer:
  Steps to be done:

1. Pending tvheadend: drop support for PCRE #22540

• snort3
  Maintainer:
  Steps to be done:

1. Planned support for last year... Still not done... Took care of manually converting it
2. Pending snort3: add patch and move to PCRE2 #22610

• snort
  Maintainer:
  Steps to be done:

1. Pending snort: bump to release 2.9.20 + experimental PCRE2 #22600

Packages in other feeds, which needs to be taken care of:

• freeswitch
  Maintainer:
  Steps to be done:

1. Pending freeswitch: add patch moving package to PCRE2 telephony#841

• sipgrep
  Maintainer:
  Steps to be done:

1. Pending sipgrep: Move package to PCRE2 telephony#838

• rtpengine
  Maintainer:
  Steps to be done:

1. Pending rtpengine: bump to 11.5.1.12 release and set PCRE2 telephony#839

• kamailio
  Maintainer:
  Steps to be done:

1. Pending kamailio: bump to release 5.7.2 + PCRE2 telephony#840

I ask anyone to be responsible and look into it, sooner than later. Otherwise, if there is no response to the package, which you are maintaining, the package will be removed at the beginning of the next month - 1st October 2023, so we have a soft deadline. I am fully aware that if there is no deadline, no further actions will be taken. You are not doing it just for me but also for others, including OpenWrt users.

Thank you for your cooperation, guys and make sure that it is run-tested on the router! ;)

Footnotes

1. https://github.com/PCRE2Project/pcre2#pcre2---perl-compatible-regular-expressions ↩

2. https://www.pcre.org ↩

3. https://openwrt.org/toh/recommended_routers ↩

[1715173329]

shadowsocks-libev

I guess it's time to switch to shadowsocks-rust, which has much better performance (10x faster on aarch64 platform).

[stintel]

net-snmp/net-snmp#420

[BKPepe]

Thanks, @stintel, I included your referenced issue to OP.

[padre-lacroix]

For the package kismet, I originally become the maintainer to prevent this package to be abandoned.
But, I do not wish to maintain it anymore, mostly because this is a package that I do not use and also because I have no way to test it.
If somebody wants to become the maintainer, then I am OK with that, otherwise it can be dropped.

[jefferyto]

slang2 is done in #22020.

[neheb]

I looked into kismet before. seems to now be a convoluted project with protocol buffers and a bunch of other dependencies.

edit: also if I had to guess, kismet being C++ would mean pcre is not used.

[Ansuel]

@BKPepe is it ok to include patch while we wait for a release? (for pcre2 support)

[nxhack]

@BKPepe mg editor done.

[BKPepe]

I looked into kismet before.

@neheb The question is do we care about quality over quantity?
Many packages here are not maintained for various reasons. Almost everyone can add a new package to this repository, no matter what. No votes, if it has purpose or if it makes any sense. I know it will sound bad, but I would rather remove those packages, which could be updated by upstream but not here. This should hopefully result in end users or someone else complaining that the package is missing within OpenWrt and that they can help us. I'm not saying that there are not enough maintainers. We (existing ones) are doing it for some purpose. We care about it. We are trying to update it, we want to have the latest versions on our routers, etc.

I might be strict, but I would simply remove those packages. You know, it is very pleasant of you that you will take into kismet, but how many users are using it?

I fail to see such unmaintained packages here: https://downloads.openwrt.org/stats/awstats.downloads.openwrt.org.allextra2.html

Is anyone really using them? If we are doing that for a minority <5-10 users then it is probably not worth all the efforts we are putting into it. They can compile themselves. They can use GitHub Actions (simply by forking this repository), if they are not familiar with the OpenWrt build system and from the CI, download the package and install it or even prepare their own feed.

With the removing these packages, we will as someone from OpenWrt core team is saying: "Less using compute resources, faster build time, happy users"

[BKPepe]

@BKPepe is it ok to include patch while we wait for a release? (for pcre2 support)

@Ansuel: You have there two packages, where you are listed as maintainer. We can probably backport uwsgi pending pull request as it seems that for 3 weeks, there has been no activity. Regarding nginx, it can be done immediately because PCRE2 support is there for many weeks.

[Ansuel]

Was talking about other packages since the deadline is 1 October. For nginx I will check today what are the required thing to do. The blocker was lua support Il Ven 22 Set 2023, 09:45 Josef Schlehofer ***@***.***> ha scritto:

…

@BKPepe <https://github.com/BKPepe> is it ok to include patch while we wait for a release? (for pcre2 support) You have there two packages, where you are listed as maintainer. We can probably backport uwsgi pending pull request as it seems that for 3 weeks, there has been no activity. Regarding nginx, it can be done immediately because PCRE2 support is there for many weeks. — Reply to this email directly, view it on GitHub <#22006 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE2ZMQQ24PDSTHEQ6SCQD43X3U623ANCNFSM6AAAAAA4JHKF2M> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[BKPepe]

Having a deadline is great, but I think we need to put these packages to some categories:

1. OpenWrt packages maintainer is not active
2. The package is not maintained anymore.
3. Many people do not use the package, whereas 1st point applies.
4. PCRE2 support is on its way and it would be too pity to remove it
5. PCRE2 is already there, but 1st point apply.

So... it is not easy as it seems. :-)

I don't currently have the answer what are you looking for. It's open discussion, so any comment is appreciated. Time will tell us.

[Ansuel]

@BKPepe I like deadlines... force things to move and honestly we should add them all around our project. For me it's ok to apply patch and then drop them on release bump... (assuming they are mature enough and just stalled)

[BKPepe]

Yeah, I like deadlines as well that's why I gave it in the first place. Exactly for the reasons what are you saying. It moves things to move.

[neheb]

kismet can go away. I don’t use it.

[padre-lacroix]

I am the maintainer of kismet and I do not use it myself: as I mentioned in a previous message, I took it to prevent at the time (2015-2016) that it will disappear.

So, kismet can disappear as far as I am concerned: I cannot and will not maintain it as I cannot even test it or use it myself.

[oskarirauta]

@msva

Proposal for zsh libpcre2 usage: #22197

[BKPepe]

It seems that we do have a lot of progress here! Glad to see it, and thanks anyone, who looked into it.

• Unfortunately, I am not sure what to do with Aircrack-ng.
  • While looking at the link, which I already put here (https://downloads.openwrt.org/stats/awstats.downloads.openwrt.org.allextra2.html), it seems that Aircrack-ng was downloaded 146 times. It wouldn't be great to remove it. I asked the developers if they could release a new version because it would help us instead of backporting patches, using the master branch, and giving it to users, etc.
    • We still need to seek for a new maintainer here.

[trippleflux]

@BKPepe

Trying my best about Aircrack-ng, hmm not to be sound rude but the bad news is I don't yet feel the like or knowledgeable enough in aircrack-ng to maintain or use it.

[Ansuel]

@BKPepe micropython/micropython-lib#737 wasted way too much time on this but eheheheh

[Ansuel]

Well tomorrow is the deadline. I would drop pcre library and mark the remaining package as @BROKEN while we wait for the remaining package to be update. For the one where we don't have any info I would move them to package-abbandoned
https://github.com/openwrt/packages-abandoned

[stintel]

Well tomorrow is the deadline. I would drop pcre library and mark the remaining package as @BROKEN while we wait for the remaining package to be update. For the one where we don't have any info I would move them to package-abbandoned https://github.com/openwrt/packages-abandoned

Please don't. You can't seriously consider dropping net-snmp from a distribution that mainly targets network devices.

[Ansuel]

@stintel if you want I can waste also today on migrating another package to pcre2.

[crza]

@stintel according to https://www.zabbix.com/documentation/current/en/manual/installation/requirements#:~:text=libpcre3%27%20or%20%27libpcre1%27.-,PCRE%20v8.x%20and%20PCRE2%20v10.x%20(from%20Zabbix%206.0.0)%20are%20supported.,-libevent zabbix 6.0+ already has pcre2 support, so maybe no patches are needed?

[Ansuel]

Yep I will check each missing package. I just need the list

[BKPepe]

Hmm, I think what I did wrong, uhh! Yep, it's just a few more packages. Guys, I am so sorry for this! I am feeling bad, because of that. I will add those packages in OP in a short manner. Dne ne 29. 10. 2023 17:14 uživatel Christian Marangi < ***@***.***> napsal:

…

Yep I will check each missing package. I just need the list — Reply to this email directly, view it on GitHub <#22006 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7IDVF7HL52LCQS7N4J2XLYBZXFNAVCNFSM6AAAAAA4JHKF2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUGE2TKOJSGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Ansuel]

@BKPepe taking care of checking each suggested package and moving them... for now only 2 require special care (aka me having to convert the package I think...)

[BKPepe]

Can someone backport to openwrt-23.05 after PR is pushed / merged?

Hmm, I think this question is answered here in this thread already. ;) You might want to check also the stable branches. Of course, it will be backported over the time, no need to worry about it.

[Ansuel]

@BKPepe did you find other package that still use pcre?

[systemcrash]

freeswitch
kamailio
rtpengine
sipgrep
snort (not v3)

http://lists.openwrt.org/pipermail/openwrt-devel/2023-October/041689.html

[BKPepe]

Dont confuse it with other feeds. They are on their own. This is only for packages feed, where are the most packages. For other feeds something like this needs to be created or they will notice that once libpcre is gone. Dne po 30. 10. 2023 23:35 uživatel Paul Dee ***@***.***> napsal:

…

freeswitch kamailio rtpengine sipgrep snort (not v3) http://lists.openwrt.org/pipermail/openwrt-devel/2023-October/041689.html — Reply to this email directly, view it on GitHub <#22006 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7IDVCQJJ5VVYKT6T4FWD3YCATURAVCNFSM6AAAAAA4JHKF2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBWGE2TEOJSGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Ansuel]

@BKPepe @systemcrash Is the list now complete? only 3 package to do... freeswitch is ""easy"" and I will manually convert snort and snort3 is HELL ON EARTH.

[systemcrash]

Those are the ones that pop up when I run make. Nothing further to add.

[Ansuel]

Welp only snort3 is missing... might take a while as I won't be at pc in the next few days :(

[Ansuel]

And with also snort3 manually converted, there isn't any more package that still require pcre.... Now it's really a matter of testing and merge the pending pr...

[systemcrash]

How is this migration coming along? Any objections to merging these?

[Ansuel]

@systemcrash nmap are backport for the other 3 package.. pr is open but no feedback from maintainers

[0xFelix]

PR #22533 broke the PCRE support in postfix, PR #22765 fixes it.

[Ansuel]

One left :(

We should really spam or find a good way to make freeswitch guy aware of the deprecation.

[BKPepe]

Yeah... only left.

Hmm, maybe we can convience telephony maintainers to merge your pending patch. :)

• Most of it, most of your work 👏 , was backported to OpenWrt 23.05 and as well OpenWrt 22.03, if it was possible.
  • I am thinking... we can force it by removing libpcre library. :P :D

[Ansuel]

Honestly I totally get why he is a bit hesitant... they looks like big changes but reality is that it's the very basic handling of pcre2 (nothing fancy) I would wait a bit, I don't like forcing stuff especially when we are in the same org :D

[systemcrash]

  * I am thinking... we can force it by removing libpcre library. :P :D

Evil genius 😈

[BKPepe]

Naaah, I am not. 😇 We have been sitting on PCRE for a longer period than I thought. It looked like an easy job in the beginning, but several open-source projects were not prepared, and it took some time for developers to look into it, review @Ansuel's work, and then, if possible, release a new version, so we don't need to carry patches, which were quite huge.

Keeping this only for one remaining software... I am kinda convinced to drop it; it's the master branch after all, issues are expected, and I am quite sure that some hidden issues will be revealed once PCRE is removed.