ssl/tls support on http request node

[mshioji]

Hello,
I’m trying to send sensor data to business system no-code tool named “Kintone” from Cybozu.
https://kintone.cybozu.co.jp/

I made a flow to add record then it works fine on Ubuntu. However, Node-RED MCU on M5 Stack, xsbug said:
“node-red-mcu/nodes/network/httprequest/httprequest.js (107) # Break: parse: 1: invalid character!”
then actually it was redirect to https://

I understand current httprequest node only supports http, not for https but I am happy if there is a way to solve this situation.
It’s very nice if the data could transfer directly from device.

Attached is a flow for debug but it excludes some information to access:

1. API access token on function node named "token".
2. domain on http request node .
3. Application ID on function node named "record".

I'm running my kintone app for debug for a while. I can provide the above information for debugging purposes, so please send me a message if someone willing to see.

Thank you very much,
flows_kintone_api_debug.zip

[phoddie]

@mshioji - thanks for the report.

The HTTP Request node is implemented using fetch() which does support https. But, that is currently disabled. I did a quick hack and was able to connect to the site you mention above. The main issue will be managing the certificates, since the embedded device does not have any built-in certificates.

There is a secondary issue here which is that your project uses "http" instead of "https". That is OK. But, the site only wants to use "https" so it redirects from the "http" to "https" URL. The fetch() implementation doesn't implement redirects yet. It can and should. It is the redirect that is generating the parse error you see, since the http response body is HTML not JSON. I did a quick hack for that which worked, though the implementation is too ugly to commit.

Anyway... the good news: this can definitely work. And it should. HTTPS support is clearly very important. I'll work through the issues.

[phoddie]

Just a little more detail, from the Node-RED Editor perspective. In the HTTP Request dialog the first step will be to enable TLS for the request:

Then you can create a TLS configuration and "Upload" the certificate for the site you are using (if you get this wrong, the device will report an error that may help correct it).

Note: Don't try this yet. It won't work until the Node-rED MCU Edition is updated. I'm just sharing how it should eventually work.

[mshioji]

@phoddie - Thank you for your quick investigation. It seems it is hard for Microcontroller to support https but it's great it is work on your hacked environment. Thank you for your development effort.

There is a secondary issue here which is that your project uses "http" instead of "https".

regarding this secondary issue, originally kintone requires https but since I understand current httprequest doesn't support https, then I just tried http. therefore I will make it back to https access.

Thank you for providing details about node-settings. I will try it after the Node-RED MCU is updated.

Thank you again,

[phoddie]

Thank you for your patience!

The challenge has been finding a good way to convert the certificate data from Node-RED into a form that can be used efficiently on the microcontroller. Node-RED MCU Edition is set up to translate the flow and its code, but for TLS it is also necessary to transfer and transform data (the certificates). That's completely new to the architecture. After three failed attempts, I think the fourth attempt should work. I hope to have something for you to try in the coming days.

[mshioji]

wow, thank you for your hard work. I don't think many people thought it would be as hard as it is on microcontroller, but I learned this is the reality. anyway you are going to make it possible. Looking forward to it and happy to try it out when it will be ready.

[phoddie]

OK. There's a first step committed that is ready to try. It requires some explanation.

The good news is that it is actually pretty easy to use. But, there are a bunch of details that may be unfamiliar the first time. That's what most of those explanation is about.

To try this out, make sure to update the Moddable SDK to at least the July 28, 2023 commit (rebuild tools after updating) and get the latest Node-RED MCU Edition.

Most of the challenges are around the TLS certificates. When you make an HTTPS request in the browser, Node, or your computer has a big collection of TLS certificates that it uses to validate the connection. This allows the majority of public web sites to connect. Unfortunately, this certificate is too big to include in a microcontroller. Instead, we just install the certificates needed.

How do you know which certificate to use?

The easiest way is to make an https request and have it fail. ;) The error message will tell you the name of the certificate in the Moddable SDK that is needed. You see a message like this in the debugger:

$MODDABLE/modules/files/resource/Resource.c (44) # Break: Resource: Resource not found: ca109.der!

The name of the certificate needed is ca109. All the Moddable SDK certificates are in the repository. The Moddable SDK uses certificates in the DER format from $MODDABLE/modules/crypt/data. Node-RED requires the certificates to be in PEM format, so we use $MODDABLE/modules/crypt/data-pem instead.

How do we tell Node-RED to use the certificate?

In the HTTP Request settings in the Node-RED Editor, you'll need to check "Enable secure (SSL/TLS) connection". Normally this is only necessary for private certificates, but for Node-RED MCU Edition it is always required when using TLS.

Click the pencil icon next to "TLS Configuration" to create a TLS Configuration. This is where you will tell Node-RED about the certificate. There are lots of settings. The only one you should set is the "CA Certificate". Use that to install the certificate in PEM format you noted above.

You may also check "Use key and certificates from local files" to reference the certificate in a separate file. Either is fine.

At this point, you can confirm that your certificate is correct by running the HTTP Request node in Node-RED on your computer.

Note: the Moddable SDK supports the Certificate, Private Key, ALPN, and disabling verification options but they are not yet connected to Node-RED MCU Edition. That's future work.

About the URL

The URL Property in the HTTP Request editor requires a little explanation. If the URL begins with "http://" it is always an unsecure request (even if "Enable secure (SSL/TLS) connection" is checked). If the URL begins with "https://", it is always a secure TLS request (even if "Enable secure (SSL/TLS) connection" is not checked). If there is no scheme (the URL does not begin with http:// or https://), such as "www.moddable.com", then "http://" or "https://" is used depending on whether "Enable secure (SSL/TLS) connection" is checked.

Note: these behaviors are defined by Node-RED. Node-RED MCU Edition just tries to do the same.

About redirection

Many web sites automatically redirect nonsecure "http://" requests to "https://". In that case, what appears to be an unsecure request becomes a secure TLS request. This redirection takes some time and increases the resource load on the microcontroller. In that case, it is preferred to use an "https://" request rather than relying on the server to redirect.

Implementation notes

The nodered2mcu tool has been updated to extract certificate data so that it can be built into Moddable SDK resources in the firmware. This happens automatically. As part of extracting, the certificates are converted from PEM to DER format. When this happens, you'll see output like this in the build log:

# nodered2mcu flows
# extract from Node-RED 38c597bbed72c216-ca.der

The resource name is based on the ID of the tls-config node, to minimize the chance of name conflicts.

To support redirection, the fetch() implementation has been updated to automatically handle redirects. It updates the redirected and url properties accordingly, so a client can detect the redirect.

[mshioji]

@phoddie - Thank you for the update and detailed description, I tried TLS Configuration according to your guide and it works.
however, in my environment with ralph-san's node-red-mcu-plugin, it seems it doesn't work properly with CA Certificate 'upload'.
in this case xsbug keep showing 'Resource not found: ca176.der!' when accessing to the server.
instead of 'upload' , using 'Use key and certificates from local files' according to your another advice, it works perfectly.
in this case, https and http(redirect) access are both fine.

[flow for debug]

[function node kintone api]

anyway, it's great now we can securely post the sensor data from small mcu directly to the server via https.
nowadays many services on the web using https therefore I think this is a great step to keep node-red-mcu accessible to these services directly and securely. thank you again for your development effort.

[phoddie]

Glad that we're making some progress here. I opened an issue on the plug-in repository to try to sort out the missing embedded certificate problem.

I think it would be good to try MQTT next with TLS. From an implementation perspective, it shouldn't be too difficult now that the basic interaction work is in place from this work on HTTPS.

[mshioji]

Thank you for opened an issue on the node-red-mcu plugin repository. I'll take a look there.

It's great if we can use MQTT with TLS on small MCU. I found "Use TLS" checkbox on mqtt-broker node settings too.
I think I will try when it will be ready. ^^

[phoddie]

I found "Use TLS" checkbox on mqtt-broker node settings too.

Yes, exactly. The TLS-configuration node seems to be used consistently in the Node-RED editor to configure TLS. It is also in the TCP and WebSocket nodes. That's nice for users of Node-RED since they only need to learn how to configure TLS once.