Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Espressif ESP-IDF v5 #5909

Closed
AchimPieters opened this issue Dec 19, 2022 · 46 comments
Closed

Support for Espressif ESP-IDF v5 #5909

AchimPieters opened this issue Dec 19, 2022 · 46 comments

Comments

@AchimPieters
Copy link

Contact Details

[email protected]

Version

5.5.3

Description

Got the message: Please report this bug, all information can be found here: https://github.com/AchimPieters/esp32-homekit-demo.git

/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S: Assembler messages:
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:104: Error: unknown opcode or format name 'pushl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:105: Error: unknown opcode or format name 'pushl'
.....

.....
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1352: Error: bad register name: (%esp
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1352: Error: junk at end of line, first unrecognized character is `('
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1352: Internal error in demand_empty_rest_of_line at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/binutils/gas/read.c:3788.
Please report this bug.

Reproduction steps

  1. Install docker
  2. Install IDF V5.0 - docker pull espressif/idf:v5.0
  3. Clone repository - git clone --recursive https://github.com/AchimPieters/esp32-homekit-demo.git
  4. start IDF - docker run -it -v ~/ESP32-HOMEKIT-DEMO:/project -w /project espressif/idf:v5.0
  5. cd examples - cd led
  6. idf.py build

Relevant log output

No response

@embhorn
Copy link
Member

embhorn commented Dec 19, 2022

Hello @AchimPieters

Thanks for sharing this issue. I have requested a review by the team.

Kind regards,
@embhorn - wolfSSL Support

@gojimmypi gojimmypi changed the title Please report this bug Support for Espressif ESP-IDF v5 Dec 19, 2022
@gojimmypi
Copy link
Contributor

Hi @AchimPieters and thank you for your interest in using wolfSSL on the Espressif ESP32.

I've tried your steps to reproduce this error and confirmed there's a problem. Thank you for reporting this.

I noticed from the error message that you are trying to use ESP-IDF v5.0. At this time the wolfSSL Espressif examples are only for ESP-IDF v4.x. We have an open PR #5902 that may fix some of the breaking changes of the recent Espressif upgrade to v5. Do you have the option to use ESP-IDF 4.x?

More work is needed to have full support for wolfSSL on the Espressif ESP-IDF v5. I've renamed this issue to reflect that objective.

There's also a reference in the error message regarding x86 assembly. This is typically used for code performance optimization, but is not available on the Espressif Xtensa architecture at this time. The x86 asm should be disabled automatically when using #define WOLFSSL_ESPIDF. There are several key ESP32 settings missing from your user_settings.h; see the example user_settings.h

It looks like you may have cloned (a rather old) wolfSSL into your project components/common/wolfssl/ as a submodule? Having just a repo clone work as an Espressif component is definitely something I'd like to have supported, but I don't yet quite have that finished. For now the best way to use wolfSSL is to use the setup.sh as described in the README and install a more recent version of wolfSSL to either the ESP-IDF or your local project ./components/ directory. Not both.

Let me know if these suggestions are helpful and what else we can do to help you get wolfSSL working in your project.

@AchimPieters
Copy link
Author

@gojimmypi thank you for your reply! Tried to use ESP-IDF 4.x, unfortunately this gives a whole new scale of problems. Looking into the future, and the development of Home and Matter, the IDF 5 and up will become mainstream. Hope that you will support it soon.

Al your provided information is really helpful, thank you for that!

@gojimmypi
Copy link
Contributor

@AchimPieters let's see what we can do with the ESP-IDF v5 using a wolfSSL component install via the script and a known-good user_settings.h file mentioned above.

Please give that a try and we'll address any incompatibilities that may be encountered.

@AchimPieters
Copy link
Author

@gojimmypi that would really be appreciated! I updated my repository with all your instructions and got this result:

/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S: Assembler messages:

/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:104: Error: unknown opcode or format name 'pushl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:105: Error: unknown opcode or format name 'pushl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:106: Error: unknown opcode or format name 'pushl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:107: Error: unknown opcode or format name 'pushl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:108: Error: unknown opcode or format name 'subl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:109: Error: unknown opcode or format name 'movl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:110: Error: unknown opcode or format name 'movl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:111: Error: unknown opcode or format name 'movl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:112: Error: unknown opcode or format name 'pxor'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:113: Error: unknown opcode or format name 'pxor'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:114: Error: unknown opcode or format name 'cmpl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:115: Error: unknown opcode or format name 'jne'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:118: Error: unknown opcode or format name 'movl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:119: Error: unknown opcode or format name 'pinsrd'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:120: Error: unknown opcode or format name 'pinsrd'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:121: Error: unknown opcode or format name 'pinsrd'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:122: Error: unknown opcode or format name 'pinsrd'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:124: Error: unknown opcode or format name 'movdqa'
......

......
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1342: Error: unknown opcode or format name 'aesenc'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1343: Error: unknown opcode or format name 'aesenc'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1344: Error: unknown opcode or format name 'movdqa'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1346: Error: unknown opcode or format name 'aesenclast'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1347: Error: unknown opcode or format name 'subl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1348: Error: unknown opcode or format name 'xorl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1349: Error: unknown opcode or format name 'movdqu'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1351: Error: unknown opcode or format name 'movzbl'
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1352: Error: bad register name: (%esp
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1352: Error: junk at end of line, first unrecognized character is `('
/project/components/common/wolfssl/wolfssl-5.5.3/wolfcrypt/src/aes_gcm_x86_asm.S:1352: Internal error in demand_empty_rest_of_line at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/binutils/gas/read.c:3788.
Please report this bug.

The strange thing is that you said: The x86 asm should be disabled automatically when using #define WOLFSSL_ESPIDF. But it still gives this error.

Reproduction steps

Install docker
Install IDF V5.0 - docker pull espressif/idf:v5.0
Clone repository - git clone --recursive https://github.com/AchimPieters/esp32-homekit-demo.git
start IDF - docker run -it -v ~/ESP32-HOMEKIT-DEMO:/project -w /project espressif/idf:v5.0
cd examples - cd led
idf.py build

@gojimmypi
Copy link
Contributor

It looks like your project is still trying to use wolfSSL in common/wolfssl/wolfssl-5.5.3 and it is not seeing the user_settings.h file. Can that wolfSSL directory be removed?

Upon install with the setup script, wolfSSL should end up in a directory called components/wolfssl which by default should be in the ESP-IDF directory. There should be no other instances of wolfSSL source code anywhere else in the build path.

Also, I noticed that you included the same reproduction text steps that actually didn't work for me as-is, but might be problematic. Your ~/ESP32-HOMEKIT-DEMO:/project there is all upper case. The clone goes to a lower case directory. Perhaps you have a non-empty upper case directory that inadvertently gets used?

@AchimPieters
Copy link
Author

I changed the file structure as you recommended.

Reproduction steps

Install docker
Install IDF V5.0 - docker pull espressif/idf:v5.0
Clone repository - git clone --recursive https://github.com/AchimPieters/esp32-homekit-demo.git
start IDF - docker run -it -v ~/ESP32-HOMEKIT-DEMO:/project -w /project espressif/idf:v5.0
cd components - wolfssl-5.5.3 -ide -espressif - esp-idf - bash setup.sh
cd .. 5x
cd examples - cd led
idf.py menuconfig - component config - freertos- kernel and enable configENABLE_BACKWARD_COMPATIBILITY
idf.py build

@gojimmypi
Copy link
Contributor

gojimmypi commented Dec 20, 2022

Hi @AchimPieters

I have an interim update for you. There's an open PR by @bandi13 that fixes some of the ESP-IDF v5 problems, but may need just a bit of fine tuning for the default ESP32 user_settings.h.

First: remove the existing wolfSSL:

Remove ~/esp32-homekit-demo/components/common/wolfssl (and ~/ESP32-HOMEKIT-DEMO if present)

(edit: I removed the above ! that I intended as "important" but could otherwise be interpreted as "not")

Fetch the @bandi13 ESP-IDF_fixes branch into bandi13-wolfssl-install

cd ~/esp32-homekit-demo
git clone https://github.com/bandi13/wolfssl.git bandi13-wolfssl-install --branch ESP-IDF_fixes --depth 1

docker run -it -v ~/esp32-homekit-demo:/project -w /project espressif/idf:v5.0

from within docker, run the wolfSSL setup:

cd /project/bandi13-wolfssl-install/IDE/Espressif/ESP-IDF
./setup.sh

cd /project/examples/led
idf.py build

This gets us close to compiling. I saw a bunch of warnings about things being redefined (ok to ignore for now):

<command-line>: note: this is the location of the previous definition
[95/145] Building C object esp-idf/wolfssl/CMa.../wolfcrypt/src/port/Espressif/esp32_util.c.objIn file included from /opt/esp/idf/components/wolfssl/wolfssl/wolfcrypt/settings.h:269,
                 from /opt/esp/idf/components/wolfssl/wolfcrypt/src/port/Espressif/esp32_util.c:21:
/opt/esp/idf/components/wolfssl/include/user_settings.h:48: warning: "HAVE_HKDF" redefined
   48 | #define HAVE_HKDF

but then encountered some more serious errors in homekit/src/json.c and homekit/src/port.c:

/project/components/common/homekit/src/json.c: In function 'json_uint32':
/project/components/common/homekit/src/json.c:238:53: error: format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Werror=format=]
  238 |     size_t len = snprintf(buffer, sizeof(buffer), "%u", x);
      |                                                    ~^   ~
      |                                                     |   |
      |                                                     |   uint32_t {aka long unsigned int}
      |                                                     unsigned int
      |                                                    %lu

/project/components/common/homekit/src/port.c: In function 'homekit_random':
/project/components/common/homekit/src/port.c:162:12: error: implicit declaration of function 'esp_random'; did you mean 'srandom'? [-Werror=implicit-function-declaration]
  162 |     return esp_random();
      |            ^~~~~~~~~~
      |            srandom
/project/components/common/homekit/src/port.c: In function 'homekit_system_restart':
/project/components/common/homekit/src/port.c:174:5: error: implicit declaration of function 'esp_restart' [-Werror=implicit-function-declaration]
  174 |     esp_restart();
      |     ^~~~~~~~~~~

Let's see if we can get your build to this state and address the errors and I'll look more at the warnings.

@PaulMartinsen
Copy link

FWIW, I was able to build for ESP32 using v5 of ESP-IDF, VisualGDB and @bandi13 's changes in #5902 . Thanks for all your hard work on this!

@gojimmypi
Copy link
Contributor

@PaulMartinsen that's great news! I saw over on the VisualGDB forum that you also had some challenges with their update in VisualGDB. Glad to see you got everything working!

There are more wolfSSL examples for the ESP32 that I want to test with ESP-IDF v5 before claiming this issue is fully resolved, so I will go ahead and leave it open for now.

Let me know how things go with wolfSSL and if I can help in any way.

@gojimmypi
Copy link
Contributor

@AchimPieters just checking in to see if you've been able to get wolfSSL working with your project?

That's cool @PaulMartinsen was able to get it working, but I want to make your project is working as well.

@AchimPieters
Copy link
Author

@gojimmypi will come back to that, too busy at the moment.

@AchimPieters
Copy link
Author

@gojimmypi So that works!

There are only a lot of redefine warnings.

Reproduction steps

docker pull espressif/idf:v5.0
git clone --recursive https://github.com/AchimPieters/esp32-homekit-demo.git
docker run -it -v ~/esp32-homekit-demo:/project -w /project espressif/idf:v5.0
git clone https://github.com/bandi13/wolfssl.git bandi13-wolfssl-install --branch ESP-IDF_fixes --depth 1
cd bandi13-wolfssl-install/IDE/Espressif/ESP-IDF
./setup.sh
cd /project/examples/led
idf.py build

@gojimmypi
Copy link
Contributor

@AchimPieters woohoo!! that's great news!

Regarding the redfine warnings in ESP-IDF v5, I've had some recent challenges in my development environment, so I don't yet have a definitive answer for that. My first guess is that it is related to #5727 - with a change in the way this handled in v5.

Is this proof-of-concept for a larger project? I'm interested in learning more.

@gojimmypi
Copy link
Contributor

gojimmypi commented Dec 23, 2022

Oh, one more thing: looks like the @bandi13 #5902 has been merged, so we should be able to get this working from the wolfSSL master branch:

git clone https://github.com/wolfssl/wolfssl.git wolfssl-install --depth 1
cd wolfssl-install/IDE/Espressif/ESP-IDF

@AchimPieters
Copy link
Author

AchimPieters commented Dec 24, 2022

@gojimmypi Thank you for your swift reply! Regarding your question: Is this proof-of-concept for a larger project? I'm interested in learning more. the answer is yes, it's a port of @maximkulkin's esp-homekit-demo.

We also have a Life Cycle Manager for the ESP modules developed by @HomeACcessoryKid there is the HomeACcessoryKid life-cycle-manager and the newest one for the ESP32 modules HomeACcessoryKid LCM4ESP32

We are a small group that develop this Repro, you are welcome to join or to collaborate!

Btw. I added the wolfSSL master branch to my repro in the component's directory. Is there a way to run the idf.py build without having to run cd wolfssl-install/IDE/Espressif/ESP-IDF and ./setup.sh firts? Is there a way to slipstream this automatically?

@AchimPieters
Copy link
Author

AchimPieters commented Jan 8, 2023

@gojimmypi just checking in to see if you've been able to get the warnings out of the compiling part?

@gojimmypi
Copy link
Contributor

Hello @AchimPieters - yes, I've recently been able to get my Espressif toolchain issues worked out. I finally have a fully functional ESP-IDF v4x and v5 working in both WSL and VisualGDB. I've not yet had a chance to test with your project but I did update a couple of wolfSSL examples: wolfSSL/wolfssl-examples#360

The examples build without warnings. I'm planning to take a look at your project this week. Let me know what you see with the latest version of wolfSSL.

Regarding your question on not having to run the install for the Docker build: you could try moving the wolfSSL component from the shared directory in the ESP-IDF toolchain directory to instead the components directory in your project. (not both)

@AchimPieters
Copy link
Author

@gojimmypi Update: Everything compiles as it should, except wolfssl the warnings.

@gojimmypi
Copy link
Contributor

Hi @AchimPieters I'm still working on getting full support for ESP-IDF v5. Glad to hear that all is working, even with the warnings. I'll need to take a closer look as I don't see those warnings, but I'm also not using Docker.

In the meantime, you may be interested in #6018. The CMakelists.txt probably just needs a little bit of modification to work with a project not in the wolfSSL repo. Specifically the WOLFSSL_ROOT setting may be the only change needed. I haven't tested it outside of the local repo yet. I'm thinking this might be a nice alternative to submodules or installation scripts.

@AchimPieters
Copy link
Author

@gojimmypi thank you for your reply! I would like to invite you to fork my repro and apply your sugestion and them make a pull request, this way you recieve the credits you deserve!!

@gojimmypi
Copy link
Contributor

Hi @AchimPieters thank you! yes, I received an invitation but it didn't work when I clicked on it. At some point I could just fork the repo. I noticed your repo seems to be a copy and not a fork?

I do plan to revisit this soon. I've been focusing on other pressing matters, in particular a failing wolfSSL test during certain configuration settings.

In the meantime, how are things coming along? Given the size of audience using the @maximkulkin esp-homekit-demo have you all considered a commercial-grade license for support, etc?

@AchimPieters
Copy link
Author

AchimPieters commented Feb 10, 2023

@gojimmypi Update: made some steps (with the help of @maximkulkin) but now I've got this message:

 HomeKit: Starting server
>>> HomeKit: Using existing accessory ID: 93:7B:1B:75:3E:35
>>> HomeKit: Configuring mDNS
>>> homekit_setup_mdns: Accessory Setup ID = 1AB2
>>> homekit_run_server: Starting HTTP server
>>> homekit_server_accept_client: Free heap: 211948
>>> HomeKit: [Client 1] Got new client connection from 192.168.178.228
>>> homekit_client_process: [Client 1] Got 121 incoming data
>>> homekit_server_on_pair_setup: Pair Setup
>>> homekit_server_on_pair_setup: Free heap: 212324
>>> tlv_debug: Got following TLV values:
>>> tlv_debug: Type 0 value (1 bytes): \x00
>>> tlv_debug: Type 6 value (1 bytes): \x01
>>> HomeKit: [Client 1] Pair Setup Step 1/3
>>> homekit_server_on_pair_setup: Free heap: 212212
>>> crypto_srp_new: Initializing SRP
>>> homekit_server_on_pair_setup: [Client 1] Initializing crypto
>>> homekit_server_on_pair_setup: Free heap: 208500
>>> homekit_server_on_pair_setup: [Client 1] Using user-specified password: 202-02-021
>>> crypto_srp_init: Generating salt
>>> crypto_srp_init: Setting SRP username
>>> crypto_srp_init: Setting SRP params
>>> crypto_srp_init: Setting SRP password
>>> crypto_srp_init: Getting SRP verifier
>>> crypto_srp_init: Failed to get SRP verifier (code -1)
!!! HomeKit: [Client 1] Failed to initialize SRP
>>> client_sendv: [Client 1] Sending payload: HTTP/1.1 200 OK\x0D\x0AContent-Type: application/pairing+tlv8\x0D\x0ATransfer-Encoding: chunked\x0D\x0AConnection: keep-alive\x0D\x0A\x0D\x0A
>>> client_sendv: [Client 1] Sending payload: \x36\x0D\x0A\x06\x01\x02\x07\x01\x01\x0D\x0A
>>> client_sendv: [Client 1] Sending payload: \x30\x0D\x0A\x0D\x0A
>>> homekit_client_process: [Client 1] Finished processing
>>> HomeKit: [Client 1] Closing client connection from 192.168.178.228

Now trying to find out ow to solve the code -1 for wolfssl comes from?

PS. a commercial-grade licence is not a as its all non-profit

@gojimmypi
Copy link
Contributor

Hello @AchimPieters - I've not forgotten this issue. Apologies for my delay in following up.

The merge of #6287 does add full support for the ESP-IDF v5. At this time, I have only updated the wolfssl_benchmark and wolfssl_test apps. The other apps need attention for the version-specific WiFi networking.

Note those examples have the "no install" components/wolfssl/CMakeLists.txt. This release shares the user_settings.h for all wolfSSL projects. Coming soon is my updated, project-specific version.

@AchimPieters
Copy link
Author

@gojimmypi so that has been a long, time, had to do some other things first. But I have made progress!
But there is still a problem with Wolfssl (SRP). Most easy way is to go to my repro: https://github.com/AchimPieters/esp32-homekit-demo/tree/main and follow the REPRODUCTION STEPS then you will see what goes perfect with the changes made to Wolfssl i.c.w. IDF5.x, but there is still a problem :(

@gojimmypi
Copy link
Contributor

gojimmypi commented Oct 5, 2023

Hi @AchimPieters - good to hear from your & thanks for your timely comment! I've been recently tidying up examples.

I've been using the ESP-IDF 5.1 for quite some time. It's been working relatively well. I'd like to take your demo for a test drive. My development environment is not very conducive to docker containers. Would you happen to have the steps for the standard idf.py build? Also, I'm not sure what SRP is, if you could kindly clarify.

Note there's a new cmake for components. See the wolfssl_benchmark and the new template examples.

Additionally, since we were last in touch, I have wolfSSL as an Espressif ESP Registry Managed Component that you may be interested in. See wolfSSL blog and preview blog for wolfSSH and wolfMQTT.

I noticed your components directory is not in the respective example project directoriwa. I've found it works best when included.. With the new CMake the component source is not duplicated in each project. But each project has its own wolfSSL user_settings.h file.

Edit:

I tried building your LED Example and saw these errors that appear unrelated to wolfSSL, but are perhaps resolved in your docker container?

-- Component directory /mnt/c/test/esp32-homekit-demo/components/common does not contain a CMakeLists.txt file. No component will be added
-- Component directory /mnt/c/test/esp32-homekit-demo/components/esp-idf does not contain a CMakeLists.txt file. No component will be added
-- Component directory /mnt/c/test/esp32-homekit-demo/components/homekit does not contain a CMakeLists.txt file. No component will be added

[ .. snip .. ]
-- wolfssl component CMAKE_BUILD_EARLY_EXPANSION:
CMake Error at /mnt/c/SysGCC/esp32/esp-idf/v5.1/tools/cmake/build.cmake:266 (message):
  Failed to resolve component 'homekit'.
Call Stack (most recent call first):
  /mnt/c/SysGCC/esp32/esp-idf/v5.1/tools/cmake/build.cmake:302 (__build_resolve_and_add_req)
  /mnt/c/SysGCC/esp32/esp-idf/v5.1/tools/cmake/build.cmake:595 (__build_expand_requirements)
  /mnt/c/SysGCC/esp32/esp-idf/v5.1/tools/cmake/project.cmake:548 (idf_build_process)
  CMakeLists.txt:13 (project)


-- Configuring incomplete, errors occurred!
See also "/mnt/c/test/esp32-homekit-demo/examples/led/build/CMakeFiles/CMakeOutput.log".
HINT: The component homekit could not be found. This could be because: component name was misspelled, the component was not added to the build, the component has been moved to the IDF component manager or has been removed and refactored into some other component.
Please look out for component in 'https://components.espressif.com' and add using 'idf.py add-dependency' command.

I created the template example just for occasions like this. I hope it helps.

Let's first get your examples working from a standard command-line build, then work on containerizing it.

@AchimPieters
Copy link
Author

  1. SRP is a HomeKit notification - As far as I know, this is caused by the fact that WolfSSL is built with settings that made it compile wrong code that works with random number generator.
  2. I already use the Espressif ESP Registry Managed Component
  3. Did you follow each step in my LED Example ?
  4. The only container that is running in docker is that of Espressif IDF? Just try it https://www.docker.com and then eache step in my repro, you will be supprised ;)

@gojimmypi
Copy link
Contributor

compile wrong code that works with random number generator

Interesting. Do you have any additional details?

I already use the Espressif ESP Registry Managed Component

Ah yes, cool

Regarding the LED example, other than the skipped docker steps, (which if the only thing is ESP-IDF, should not matter, eh?), the idf.py menuconfig fails with this Failed to resolve component 'homekit' error. Do you believe the wolfssl component CMAKE_BUILD_EARLY_EXPANSION step is the cause?

-- Detecting CXX compile features - done
-- Building ESP-IDF components for target esp32
Processing 3 dependencies:
[1/3] espressif/mdns (1.2.1)
[2/3] idf (5.1.0)
[3/3] wolfssl/wolfssl (5.6.0-stable)
-- wolfssl component CMAKE_BUILD_EARLY_EXPANSION:
CMake Error at /mnt/c/SysGCC/esp32/esp-idf/v5.1/tools/cmake/build.cmake:266 (message):
  Failed to resolve component 'homekit'.
Call Stack (most recent call first):
  /mnt/c/SysGCC/esp32/esp-idf/v5.1/tools/cmake/build.cmake:302 (__build_resolve_and_add_req)
  /mnt/c/SysGCC/esp32/esp-idf/v5.1/tools/cmake/build.cmake:595 (__build_expand_requirements)
  /mnt/c/SysGCC/esp32/esp-idf/v5.1/tools/cmake/project.cmake:548 (idf_build_process)
  CMakeLists.txt:13 (project)


-- Configuring incomplete, errors occurred!
See also "/mnt/c/test/esp32-homekit-demo/examples/led/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1

@AchimPieters
Copy link
Author

I have to go to work, but will be back later and try to answer your questions, here is my compiling log maybe that clarifies some things?
Log.txt

@AchimPieters
Copy link
Author

Maybe @maximkulkin can explain this problem better, will ask him. He has pushed an update to wolfssl to fix it.

@gojimmypi
Copy link
Contributor

Would you please point me to the update?

I don't see any issues or pull requests by @maximkulkin here nor on your esp32-homekit-demo.

As for the log file, that was very helpful. It appears some of the settings are being assigned twice. This is likely in the user_settings.h and /or the options.h.

I tried the example, and it compiles successfully:

idf.py create-project-from-example "wolfssl/wolfssl^5.6.0-stable:wolfssl_test"

Clearly there's some difference in your environment that I'd like to identify and resolve. It would be helpful to me to have the steps to reproduce this without using docker.

@gojimmypi
Copy link
Contributor

Update: I am now able to reproduce this problem & currently investigating.

@gojimmypi
Copy link
Contributor

Hi @AchimPieters I have determined the cause of the duplicate wolfssl definitions that start like this at build time:

[683/866] Building C object esp-idf/wolfssl__wolfssl/CMakeFiles/__idf_wolfssl__wolfssl.dir/wolfcrypt/src/sha256.c.objIn file included from /mnt/c/test/esp32-homekit-demo/examples/led/managed_components/wolfssl__wolfssl/wolfssl/wolfcrypt/settings.h:272,
                 from /mnt/c/test/esp32-homekit-demo/examples/led/managed_components/wolfssl__wolfssl/wolfcrypt/src/sha256.c:45:
/mnt/c/test/esp32-homekit-demo/examples/led/managed_components/wolfssl__wolfssl/include/user_settings.h:51: warning: "HAVE_HKDF" redefined
   51 | #define HAVE_HKDF
      |
<command-line>: note: this is the location of the previous definition
/mnt/c/test/esp32-homekit-demo/examples/led/managed_components/wolfssl__wolfssl/include/user_settings.h:62: warning: "WOLFSSL_SHA512" redefined
   62 | #define WOLFSSL_SHA512

The resolve this problem, the compiler needs to have WOLFSSL_USER_SETTINGS defined. See the template example.

Suggested change

For your led example, please add this line to the beginning of your examples/led/main/CMakeLists.txt file:

set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DWOLFSSL_USER_SETTINGS")

Result of change

With that change, I still see a few warnings, but they appear to be specific to your project and unrelated to wolfSSL.:

                 from ../../../main/led.c:35:
../../../main/led.c:139:44: warning: initialized field overwritten [-Woverride-init]
  139 |         HOMEKIT_ACCESSORY(.id=1, .category=homekit_accessory_category_lighting, .services=(homekit_service_t*[]){
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:/test/esp32-homekit-demo/components/homekit/include/homekit/types.h:250:11: note: in definition of macro 'HOMEKIT_ACCESSORY'
  250 |         ##__VA_ARGS__ \
      |           ^~~~~~~~~~~
../../../main/led.c:139:44: note: (near initialization for '(anonymous).category')
  139 |         HOMEKIT_ACCESSORY(.id=1, .category=homekit_accessory_category_lighting, .services=(homekit_service_t*[]){
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:/test/esp32-homekit-demo/components/homekit/include/homekit/types.h:250:11: note: in definition of macro 'HOMEKIT_ACCESSORY'
  250 |         ##__VA_ARGS__ \
      |           ^~~~~~~~~~~
[3/7] Linking C static library esp-idf\main\libmain.a

Please give that a try and let me know how it goes.

Related problem

Please let me know if the CMakeLists.txt change also resolves your comment, from above:

SRP is a HomeKit notification - As far as I know, this is caused by the fact that WolfSSL is built with settings that made it compile wrong code that works with random number generator.


Thank you for your cooperation. Your example is very helpful and will provide guidance for future enhancements to make this process easier and more intuitive.

@AchimPieters
Copy link
Author

Hi @gojimmypi I have incorporated your change and here is the result: log.txt

It doesn't seem to work in my setup? Will do some further investigation.

@gojimmypi
Copy link
Contributor

Hi @AchimPieters - Ah yes, I believe I see what's going on here.

Commenting out the wolfSSL user_settings.h in the crypto.c file and instead using list(APPEND EXTRA_WOLFSSL...) in that submodules's CmakeLists.txt is likely the root cause of all those duplicate definition warnings.

I probably missed that when recompiling the LED example and the esp_homekit submodule was already built.

It is best to use only the wolfSSL user_settings.h file. See an example in the wolfssl_benchmark demo.

Otherwise, your app appears to be compiling properly. The warnings are as desired as they are definitions in two places: the cmake definitions and the user settings. You can see the dangers here of having conflicting configuration settings in the user_settings.h vs the CMakeLists.txt. Debugging could be quite challenging.

The other thing that I will likely change in the next release is the removal of the ./lib directory. That's only used for deployment and should not be in the final managed component source. I initially this this was problematic with a second user_settings.h file there, but the directory is unlikely to be included in the build process. Still, it doesn't belong there.

If you want to remove ./lib yourself, you'll need to move the component out of the managed_components directory and into the components directory. Do an idf.py fullclean and then rebuild after removing the directory & you'll see the specific warning about the component changing and how to make it unmanaged.

@AchimPieters
Copy link
Author

@gojimmypi uncommenting the wolfSSL user_settings.h in the crypto.c file, removing the lines list(APPEND EXTRA_WOLFSSL...) in CmakeLists.txt is not the root cause of all those duplicate definition warnings. When I do so, it gives whole new errors?

Debugging is quite challenging. But I will keep looking for the solution!

@gojimmypi
Copy link
Contributor

Hi @AchimPieters

The new errors you see are because the settings for wolfSSL (usually in the user_settings.h file) turn code segments on or off as needed. When features are disabled (or not yet enabled from a different settings file), entire code segments become invisible to the compiler:

#ifdef MYFEATURE
    /* not compiled unless MYFEATURE is defined */
    int my_feature()
    {
        return 1;
    }
#endif

The settings in the CMakeLists.txt are a brute-force solution to the wolfssl user_settings.h not being seen by the homekit code.

The homekit\src in the esp32-homekit-demo/components directory references wolfSSL, but the wolfSSL component libraries are in the esp32-homekit-demo\examples\led demo project, even though it appears the wolfSSL code is not needed in the led demo.

It would be best to put the wolfSSL component with the code that uses it (the homekit library).

You could, in theory, remedy the duplicate definitions by prefixing each one with an undef:

#undef  MYSETTING
#define MYSETTING

This is not at all a solution I recommend, as the warnings have a legitimate and valuable purpose. When the settings in the CMakeLists.txt and the user_settingsh later become out of sync, debugging will be considerably more difficult. I do recommend that you consider putting all wolfSSL settings in the user_settings.h file only.

The code in esp32-homekit-demo/components needs to see the wolfSSL source with the user settings already defined. The user_settings.h is nowhere in that directory tree. It needs to be in the local project components/wolfssl/include directory. May I suggest putting the wolfSSL component only in the HomeKit library that uses it directly. At some point in the future, we might make this more flexible, but for now that's how it works.

Each source file that uses wolfSSL needs first have an include for the user_settings.h before any other wolfSSL libraries are reference, as shown in the example.

Once #6844 is merged, I will be closing this issue as it relates to the original subject: Support for the Espressif ESP-IDF v5. You are welcome to open a new issue regarding the use wolfSSL components in shared libraries.

This is an interesting situation and I appreciate you bringing it up. I realize the code changes to remedy your compiler warnings are a bit complex. I won't have time to help you with that in the immediate future. But I hope my suggestions at least point you in the right direction. As you make progress, I'm happy to help along the way. A new issue is probably best for that.

Best Regards

@gojimmypi
Copy link
Contributor

Hi @AchimPieters -

I have a esp32-homekit-demo fork with some changes that demonstrates wolfSSL working with only user_settings.h configuration, removed from the HomeKit CMakeLists.txt.

I had quite a difficult time with this repo in Visual Studio, as the Icon%0D was not happy on a Windows/DOS file system, so I removed it. Oddly this one problem manifested itself as Visual Studio not being able to see any git changes for the entire repo.

Note I moved wolfSSL from the local led example project as a managed component to the shared /components directory. I'll need to do a little more testing to have a managed component outside of the example project subdirectory structure.

In particular, note there's this WOLFSSL_ROOT setting added:

set(WOLFSSL_ROOT "${CMAKE_CURRENT_SOURCE_DIR}")

I also added these lines to your crypto.c in the HomeKit submodule:

#include <wolfssl/wolfcrypt/settings.h>
#include <user_settings.h>

and the respective CMakeLists.txt now looks like this without the wolfSSL settings:

set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DWOLFSSL_USER_SETTINGS")

idf_component_register(
    SRC_DIRS src
    EXCLUDE_SRCS
        "src/homekit_mdns.c"
        "src/homekit_mdns_debug.c"
    INCLUDE_DIRS "include"
    PRIV_INCLUDE_DIRS "src"
    REQUIRES wolfssl esp_partition json http-parser mdns spi_flash
)

#if(CONFIG_HOMEKIT_SMALL)
#    list(APPEND EXTRA_WOLFSSL_COMPILE_OPTIONS -DCURVE25519_SMALL -DED25519_SMALL)
# endif()

idf_build_set_property(COMPILE_OPTIONS "${EXTRA_WOLFSSL_COMPILE_OPTIONS}" APPEND)

target_compile_options(${COMPONENT_LIB} PRIVATE
    -Wno-error=unused-variable
    -DHOMEKIT_MAX_CLIENTS=${CONFIG_HOMEKIT_MAX_CLIENTS}
    -DSPIFLASH_BASE_ADDR=${CONFIG_HOMEKIT_SPI_FLASH_BASE_ADDR}
    -DESP_IDF
)

if(CONFIG_HOMEKIT_DEBUG)
    target_compile_options(${COMPONENT_LIB} PRIVATE -DHOMEKIT_DEBUG)
endif()

Please review these suggested changes and let me know what you think.

@AchimPieters
Copy link
Author

AchimPieters commented Oct 12, 2023

@gojimmypi that looks promising, made the changes you provided, and the errors are gone...... as far as the previous ones ;) See the log. log.txt, But nevertheless it is worth of further investigation!

@gojimmypi
Copy link
Contributor

Hi @AchimPieters great to hear you are making progress!

and the errors are gone.

yay! (although technically they were warnings, and the code was still compiling)

Those new errors appear to indicate some features were not enabled from the user_settings.h file.

Can you please confirm you've updated the homekit component as described above? It appears the esp32-homekit-demo is still using an older submodule: homekit @ 3b5c3a4. For instance the crypto.c still has the #include "user_settings.h" commented out:
image

@AchimPieters
Copy link
Author

@gojimmypi updated! I would really encourage you to collaborate on the project, as you are so enthusiastic. I already send you an invitation.

@gojimmypi
Copy link
Contributor

updated!

excellent! and the result?

encourage you to collaborate on the project ... already sent you an invitation.

yes, thank you! I already accepted when I created my fork, but I think that only grants me permissions to push changes directly to your branch, eh? I can certainly create some PRs if you'd like. I'd prefer you to be the one driving applied updates so that you fully understand and control the changes being made.

@AchimPieters
Copy link
Author

log.txt

@gojimmypi
Copy link
Contributor

I'm away from my computer at the moment, but the implicit declaration of function 'Base64_Encode_NoNl' is very promising!

Please check that all the prior wolfSSL settings that were in the CMakeLists.txt are in the user_settings.h.

If everything is checked into GitHub, I'll take a look later. My fork should have all applicable changes with the exception of the submodule updates noted above.

@AchimPieters
Copy link
Author

AchimPieters commented Oct 14, 2023

Hi @gojimmypi,

I tried to look into the encodign errors like:

 4122 |             Base64_Encode_NoNl((const unsigned char *)shaHash, 4, encodedHash, &len);
      |             ^~~~~~~~~~~~~~~~~~
      |             base64_encode 

here: https://www.wolfssl.com/documentation/manuals/wolfssl/coding_8h.html#function-base64_encode_nonl

and

project/components/homekit/src/crypto.c:59:23: error: unknown type name 'Srp'
   59 | int wc_SrpSetUsername(Srp * srp, byte * secret, word32 size) {
      |                       ^~~
/project/components/homekit/src/crypto.c:80:1: error: unknown type name 'Srp'
   80 | Srp *crypto_srp_new() {
      | ^~~
/project/components/homekit/src/crypto.c: In function 'crypto_srp_new':
/project/components/homekit/src/crypto.c:81:5: error: unknown type name 'Srp'
   81 |     Srp *srp = malloc(sizeof(Srp));
      |     ^~~
/project/components/homekit/src/crypto.c:81:30: error: 'Srp' undeclared (first use in this function); did you mean 'srp'?
   81 |     Srp *srp = malloc(sizeof(Srp));
      |                              ^~~
      |                              srp
/project/components/homekit/src/crypto.c:81:30: note: each undeclared identifier is reported only once for each function it appears in
/project/components/homekit/src/crypto.c:84:13: error: implicit declaration of function 'wc_SrpInit' [-Werror=implicit-function-declaration]
   84 |     int r = wc_SrpInit(srp, SRP_TYPE_SHA512, SRP_CLIENT_SIDE);
      |             ^~~~~~~~~~
/project/components/homekit/src/crypto.c:84:29: error: 'SRP_TYPE_SHA512' undeclared (first use in this function); did you mean 'WC_HASH_TYPE_SHA512'?
   84 |     int r = wc_SrpInit(srp, SRP_TYPE_SHA512, SRP_CLIENT_SIDE);
      |                             ^~~~~~~~~~~~~~~
      |                             WC_HASH_TYPE_SHA512

for these here: https://www.wolfssl.com/documentation/manuals/wolfssl/srp_8h.html

but they dont give me the answere why these erros show up? would like to ask you if you could have a look.

Ps. Mabe get my latest git changes, first? https://github.com/AchimPieters/esp32-homekit-demo

@AchimPieters
Copy link
Author

@gojimmypi my last post is actually off-topic, and since the original issue "support for idf v5" has been resolved, I'm going to close this topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants