unix-ffi: re: convert to PCRE2

[Ansuel]

PCRE is marked as EOL and won't receive any new security update.

Convert the re module to PCRE2 API to enforce security. Additional dependency is now needed with uctypes due to changes in how PCRE2 return the match_data in a pointer and require special handling.

The converted module is tested with the test_re.py with no regression.

[Ansuel]

(this is needed as we are deprecating pcre in openwrt and we are converting any user to pcre2)

[dpgeorge]

Thanks for the contribution.

What's the difference between libpcre2-posix, libpcre2-8, libpcre2-16 and libpcre2-32?

It would be possible to support both the old PCRE and PCRE2, but I don't think it's worth the effort.

[Ansuel]

The POSIX variant just provide wrapper to have POSIX interface for pcre2 API.
Pcre had the same variant and wasn't used here.

The -16 and -32 variant are used to support 16bit and 32bit string (UTF-16 and UTF-32)
UTF-8 should be enough and what is commonly used when migrating from pcre to pcre2

Supporting both versions would bloat the code with lots of ifdef and IMHO it's not worth it and pcre won't receive any update and would pose security risks.

[dpgeorge]

The -16 and -32 variant are used to support 16bit and 32bit string (UTF-16 and UTF-32)
UTF-8 should be enough and what is commonly used when migrating from pcre to pcre2

Aah, I see, thanks for the info. Indeed we should use the -8 variant.

I see that the test_re.py provided in this re directory doesn't pass anymore. It needs to pass.

[Ansuel]

@dpgeorge i remember it did pass correctly. Maybe i'm running it wrong? my test was compiling the lib and run the test using the compiled library. Something wrong in this?

[dpgeorge]

I'm simply running the following in the unix-ffi/re directory with this PR checked out:

$ micropython
MicroPython v1.20.0-557-g5aec051f9 on 2023-10-05; linux [GCC 13.2.1] version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import re
>>> re.search(r"a+", "caaab").group(0)
''

That's not correct output.

[Ansuel]

@dpgeorge mhh i'm confused

root@OpenWrt:~# micropython
MicroPython v1.20.0 on 2023-10-04; linux [GCC 12.3.0] version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import re
>>> re.search(r"a+", "caaab").group(0)
'aaa'

[Ansuel]

Also by putting the uncompiled re.py in /usr/lib/micropython/unix produce the same output

[Ansuel]

@dpgeorge I finally manage to repro the problem... It seems to be a problem with PCRE2_SIZE that is size_t and this change between 32bit and 64bit. Using ULONG handled both systems. (my test were done on a 32bit system)

[dpgeorge]

Can this call to sizeof to get the size be done at global module level, so it only has to be done once?

[Ansuel]

Better now?

[dpgeorge]

Thanks, I can confirm that it now works on my (64-bit) machine.

[BKPepe]

So, @dpgeorge it can be merged, right? :)

[projectgus]

Changes LGTM, thanks @Ansuel!

[dpgeorge]

Thanks for updating, this looks good now.

Merged.