RSA-OAEP Encrypt and decrypt issues with specifying the correct HASH type

[clauderobi]

Hi,

so background

I have a server that uses openSSL (in fact libcrypto) to perform various RSA operations. The very first thing is to encrypt something using a public key provided by a client. For the last 2 years, I have a Webapp, using the Webcrypto API, that fully interoperates with the server. Both the server and Webapp are using RSA-OAEP with SHA-1 (not sure what is the MGF is used by presumably it is MGF1 for both).

I now have the need to create a mobile app and I decided to use react-native.

Main problem
My react-native app fails at decrypting the reply from the server... :-(

So I decided to create a test harness that I could run in both the Webapp and in the react-native app. The data set provides a key pair (in PEM format), a cipher text and the expected clear text per set. I have multiple sets, some generated with react-native-fast-rsa and other with the openSSL cipher text (and corresponding keys).

On a per data set, the test does this:

1. validates the key pair by encrypting 'Hello' and decrypting it.
2. try to decrypt the provided cipher text and, on success, validate the result with the expected clear text

Over time I came to realize that the issue was probably related to the HASH selection.

HASH related problem / observation
The following tests / observations are made using a data set that provides a cipher text generated with react-native-fast-rsa (encryptOAEP using SHA1 and no label)

1. During the key pair validation phase, the result is always successful regardless of the hash name provided to encrypt and decrypt, even if the hash name is invalid (I use the string value '1' and well as the integer 1)
2. When decrypting the provided cipher text, decryption fails if the hash name is '1' but succeeds for ALL valid hash name (actually I tested with SHA1, SHA256 and SHA512) as well as with the integer 1. In other words, if a valid hash name is provided (or the numeral 1), decryptOAEP decrypts the same cipher text using different hash, which is a non-sense.

I am using a debugger so RSA.useJSI = false;

At this stage, I do not what is going on. And more specifically, which HASH is effectively used. It appears that it is not SHA1since interoperation with OpenSSL and Webcrypto is broken.

Any suggestion?

[clauderobi]

I did some more tests.

I am simply encrypting an arbitrary payload using a 1024 bits key and testing with various hash (using RSA.encryptOAEP) and various payload size. According the standard, using OAEP, the maximum payload size should be (for a 1024 bits key):
SHA-1: 86B
SHA-256: 62B
SHA-512: not possible

I was able to encrypt (and decrypt) upto 94B (using SHA-1). So, clearly, encryptOAEP and decryptOAEP are celarly not working as per the standard.

Using crypto-browserify works has expected. But it is so slow that is almost useless. But at least I know that my code is doing what it should.

Am I missing something?