Update to OTNS version 2 #530
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi Esko, thanks for your great contribution! But this is a very huge PR of more than 30K lines change that none of the openthread team will be able to provide meaningful code review than accepting it as is. Is it possible to split this PRs into a few smaller PRs by features? |
Thanks! I've added some ideas on the review approach in the issue #528 . |
|
@EskoDijk Could you resolve the merge conflicts and make the CI tests green? |
…omodel that implements such feature)
…les more useful for analysis.
… go back in time); comment update.
…=false mode for useful quick and precise steps in simulation progress.
…nd to see detailed node logs at chosen log-level, and node-log to file storage; log format updates.
…o invoke global context for running command.
…l exit the node context.
…um bytes; extra asserts on radiomodel states.
…fault watch-state and watch loglevel of new nodes. This helps to see the log messages from the OT node, emitted directly at time t=0. Also 'watch all' command now supported.
…fault watch log level for newly created nodes.
… default watch LogLevel. Plus, added a TODO for future logger improvement.
…ats recalculation at JS side (#158) * [all] Update naming of gRPC ProtoBuf events; and allow multiple requests to re-use same VisualizeEvent stream for different purposes. NodeStats to be calculated only in Go code and to be sent via event to the JS/webpage side. * [dispatcher] moved nodeStats calculation into Dispatcher; exporting NopVisualizer as a base class to avoid the many empty methods for Visualizers. * [all] node stats calc now in dispatcher/stats.go; * [web] heartbeat display fix on JS nodestats * [radiomodel] bugfix in stats for case when transmitting node gets deleted. * [dispatcher][types] minor correction on MAC packet len for global loss calculation * [radiomodel] adding NumBytesTx stats to Radionode * [radiomodel][simulation] added phy.tx.bytes to node-counters KPI data. * [all] added PhyStats to RadioModel, and types in StatsLogVisualizer to write TxRate stats to a CSV file. * [otns_main] made Phy Tx stats visualizer optional with a cmdline flag. * [pylibs] using new -phy-tx-stats flag in SRP case study; added SRP reregistration case study (3) * [script] add warning msg for build_all * [ot-rfsim] git checkouts now moved to 'build_all' script - fixed some build config flags. Build script fixes - exclude features from 1.1/1.2 nodes that are not available for those. * [ot-rfsim] enable OT_BORDER_ROUTER to get 'prefix' CLI command in Routers * [ot-rfsim] comments added (removing the FIXME) * [pylibs][script] some Python venv magic for macos-14; update to Python >= 3.10 * [.github][script] using virtualenv for Python for all cases * GUIDE and README updated to virtual env use
This commit includes all PRs merged into OTNS (v1) main branch, since the OTNS (v2) fork was created in 2022. Some PRs are applied as is, others are adapted to the new OTNS2 codebase. Also this includes version increases for Go 1.x and some Go and Python libraries used to newer versions. It also introduces some new features that were needed to optimally merge the PRs according to their intent. * YAML based startup script configuration: some node types (ftd/mtd/br) can now have their own unique startup command sequence, encoded in the same script file. This also enables for the future a fully YAML based configuration of a simulation including all node positions, radio models, startup scripts, etc. * [github-actions] fix for pushes do not run CI (openthread#395) - this was not done. It appears it is currently not needed, leads to many cancelled CI tasks. CI tasks will run already on every push to the PR branch. Keeping this note as informative message. * update golangci-lint tool and fix pretty issue based on commit abf59e0 * [script] install python3-venv as an attempt to fix failing Docker build * Bump wheel from 0.34.2 to 0.38.1 in /pylibs/unittests (openthread#439) Bumps [wheel](https://github.com/pypa/wheel) from 0.34.2 to 0.38.1. - [Release notes](https://github.com/pypa/wheel/releases) - [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst) - [Commits](pypa/wheel@0.34.2...0.38.1) --- updated-dependencies: - dependency-name: wheel dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump golang.org/x/net from 0.0.0-20201021035429-f5854403a974 to 0.7.0 (openthread#478) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.0.0-20201021035429-f5854403a974 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](https://github.com/golang/net/commits/v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [.github][go.mod] update to go 1.22 * [.github] some version quoting issues / bump setup-go action * [.github][GUIDE][go.mod] require Go 1.18 minimum; build for 1.20 and 1.22 also; clarified GUIDE * docker file updates -> go 1.22 / removed unneeded items * [script] fix hanging golangci-lint for go 1.20, 1.22 - bump version to 1.59.0 * [scripts] add wheel to install_python_libs() (openthread#485) This should address this warning https://github.com/openthread/openthread/actions/runs/4321875033/jobs/7543625431#step:8:348 * [.github] quote go versions * [dependabot] reduce frequency to monthly (openthread#494) * [all] [tests] properly configure operational dataset (openthread#502) * [pylibs] fix raw test_commissioning by giving more time for nodes to scan channels. * [.github] change order of unit tests * [openthread] bump * Bump grpcio from 1.46.3 to 1.53.0 in /pylibs/unittests (openthread#506) Bumps [grpcio](https://github.com/grpc/grpc) from 1.46.3 to 1.53.0. - [Release notes](https://github.com/grpc/grpc/releases) - [Changelog](https://github.com/grpc/grpc/blob/master/doc/grpc_release_schedule.md) - [Commits](grpc/grpc@v1.46.3...v1.53.0) --- updated-dependencies: - dependency-name: grpcio dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [pylibs] fixes requirements.txt bump protobuf, wheel * [go.mod] lib versions update * [otnstester][cli] fix cli unit-test * [pylibs] in tests don't set trace watch level by default / remove unnecessary 'watch default debug' * [script] fix issue that historic commits not found in --depth 1 cloned repo. * [pylibs] force_key_rotation test extension * [pylibs] increase network-limits time limit for case BR is a Parent, to avoid spurious test fails. * [pylibs] otns-performance stress test runs with fixed random seed (to get a more predictable baseline performance for number of events), to avoid spurious test fails on number of events. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Simon Lin <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mason Tran <[email protected]> Co-authored-by: Jonathan Hui <[email protected]>
* [all] added separate 'log write' event type for OT nodes. This simplifies the parsing at OTNS side: not intertwined anymore with UART output. The otoutfilter package is removed (if in future needed for parsing UART-based OT logs, then it can always be added back in again)
…; updated BR script.
…count) (#160) * [all] adding more phy stats, including channel samples count. * [all] enabling saving of multiple Phy stats to CSV files * [all] fix in time window stats data: finalize the last line of CSV stats when simulation ends * [all] converted rate to Phy Bytes Tx. * [all] make time window stats conditional on configuration bool in Dispatcher config. If flag not active, no stats will be generated.
This feature adds support for simulated external IPv6 hosts/servers, that a simulated OT node can communicate with. Traffic to the external simulated host (or 'simhost') is routed to a Thread BR (if it publishes the correct routes on the Thread Network), then sent to the OTNS simulator. OTNS will determine a matching simhost (if any) and will use address/port translation to map the IP datagram to a localhost designated port. On this local port, a particular service/server can be started by the user. For example, a CoAP server or HTTP server. This feature also serves as preparation for future addition of CCM commissioning of simulated OT nodes. In such case, a Thread BR will "relay" commissioning traffic to an external/remote server like a cBRSKI Registrar. Includes the following commits: * [cli] introduction of 'host' CLI command * [pylibs] Add CCM-commissioning unit test (not actively tested for the moment - preparation for future functions) * [pylibs] extend Commissioning unit test to include multi-hop commissioning. * [dispatcher][pylibs] added HostEvents counter * [dispatcher][event] added event types for IP/UDP forwarding to from host via simulator * [pylibs] using aiocoap library for testing external CoAP server, in unit tests Includes commits for test/workflow/bug fixes and necessary improvements: * [.github] update workflow to separate out the version-unittests, to avoid this always failing on macos build. * [pylibs] fix unit test (TMF Coap messages may be sent in parallel of the test coap message) * [pcap] insert a dummy 802.15.4 frame at start of every PCAP file, to create a t=0 simulation time reference. This allows any PCAP entry to be associated to a corresponding radio (trace) log entry with exact (to the microsecond) precision. * [pcap] fix unit test, related to introduction of t=0 time-reference frame. * [pylibs] added further comments in partial_dataset case study. * [ot-rfsim] fix macos build error (due to warn) * [ot-rfsim] add C lang definitions for macOS build cpp in ot-rfsim project. * [ot-rfsim] adding 'a' in front of function call parameters; and minor format fixes in C code; adding comments; removing FIXME items. * [ot-rfsim] fix build for v1.1, v1.2 for using ot....() platform logging functions. * [pylibs] fixes to test_commissioning - test sometimes failed. * [pylibs] test_border_router: narrow test topology to avoid spurious test fails.
…isten on a different channel. (#163)
* [ot-rfsim] remove dataset auto-init from build config; also remove the default BORDER_ROUTER on. * [pylibs] adapted test_basic multi-channel test to use correct dataset init method. * [pylibs] updated test_topologies multi-channel test to use correct dataset init method.
#166) * [go.mod] [otnstester] reduce Go lib versions to avoid indirectly requiring Go 1.19+ language features. * [.github] ensure Go 1.18 is supported by including in CI tests. * [script][GUIDE] added 'bootstrap' script, GUIDE detailed, fixes in setup scripts, more packages added in setup for bare Docker ubuntu:jammy case. * [script] fixes in bootstrap sequence * [script] adding gcc/g++ to required deps * [pylibs] reduce aiocoap version to support macos w/ Python 3.9 * [.github][pylibs] lower to Python >= 3.9 for MacOS support.
* [submodules] added v11, v12, v13 shallow submodules. * [ot-rfsim][script] updated scripts to support version builds from submodules; build script got autodetect when to clean builddir. * [ot-rfsim] scripts for versions build - ignore compiler warnings to let build succeed on newer (e.g. Apple Clang) compiler. * [ot-rfsim] add separate ot version build dirs - avoiding needless rebuilds and also preventing Cmake config errors. * [script] bootstrap script now usable after a sw update (git pull) for rebuilding nodes. * [.github] enable py-ver-unittests for all platforms (incl macos); update deprecated cancel-previous-runs.
…flow-run-cleanup-action
…stem ; Make-pretty for all code types (#165) * [pylibs] Add CCM-commissioning unit test; with Commissioning unit test to include multi-hop commissioning. * [pylibs] test to include normal and CCM joiner in one test. * [ot-rfsim] fixed filter on outgoing (off mesh) packets * [ot-rfsim] fix build for FTD * [event] bugfix in event data len; tests added. * [pylibs][script] aiocoap install for unit-tests only * [dispatcher][pylibs] added HostEvents counter * [pcap] insert a dummy 802.15.4 frame at start of every PCAP file, to create a t=0 simulation time reference. This allows any PCAP entry to be associated to a corresponding radio (trace) log entry with exact (to the microsecond) precision. * [pylibs] added further comments in partial_dataset case study. * [pcap] fix unit test, related to introduction of t=0 time-reference frame. * [ot-rfsim] fix macos build error (due to warn) * [ot-rfsim] add C lang definitions for macOS build cpp in ot-rfsim project. * [ot-rfsim] adding 'a' in front of function call parameters; and minor format fixes in C code; adding comments; removing FIXME items. * [pylibs] added more documentation to test_sim_hosts and fixed testing-for-coaps in one test. * [ot-rfsim] fix build for v1.1, v1.2 for using ot....() platform logging functions. * [pylibs] fixes to test_commissioning - test sometimes failed. * [pylibs] narrow test topology to avoid spurious test fails. * [pylibs] Added CCM unit test for one CCM joiner * [ot-rfsim][cli] add version 'ccm' to place CCM nodes; adding branch pr-ccm repo of openthread for building CCM nodes; .gitignore cleanup. * [script] add make-pretty exclusion for new submodule * [ccm] added OT Registrar use in unit tests. * [ot-rfsim] change ccm build script style to match latest 'main' branch merged. * bumped openthread to latest code; build and API updates * [ot-rfsim] radio: compile-fix for otMacFrameGetSequence function signature change. * [ot-rfsim] script: build only needed targets; store all build results under ./build; introduced main() in all build scripts; separate build dirs for BR/non-BR builds. * [pylibs] 'joiner startccm' command added. * [.golangci.yml] new syntax to replace deprecated one * [.gitmodules] add commented OT CCM submodule (for future) * [.github][script] updated make-pretty script to support all languages and checks; removed check-pretty. * [ot-rfsim] improved build system, can now build/update all nodes quickly * [script] make-pretty script updated; added clang config files from OpenThread.
…les and make-pretty applied (#169) This updates the `make-pretty` script to support all used languages in the project. It's based on the same script in the `openthread` repo. Also make-pretty is applied on all sources. Additionally, Github yml files are updated to better comply with recommended steps order and to bump setup-go to v5. Some steps are renamed to provide more clarity what it does. And the 'build' test is now split in the Go build part and the OT nodes build part. These 2 parts each require a separate matrix (since the nodes build doesn't involve Go, and the Go build doesn't involve C/C++ ) * [ot-rfsim] make-pretty clang-tidy * [ot-rfsim] make-pretty clang-format * [all] make-pretty markdown .md * [ot-rfsim] make-pretty shell * [pylibs][script] make-pretty python (excluding generates files in proto/) * [.github] Updated Github yml workflow names; split OTNS and Nodes builds in build.yml; bring checkout before setup-go to avoid warnings. I.e. apply standard order as recommended. * [script] moved cmake and ninja to requires packages -> needed now with Lint requiring a node-build.
…m, Python-venv related fixes (Pydoc), adding 'test' script use, and link to WPAN-TAP format.
…ixes (#170) * [.github] added build.yml test for 'bootstrap' script which should install Go. * [.github] bump codecov-action@v4 - try to avoid CI error on upload * [script] remove explicit deletion of older node binaries - new build system will anyway 'copy over' the standard nodes after build. * [script] bootstrap with 'set -x' to log better during CI. * [script] realpathf - suppress error in case -s switch not supported by OS. * [pylibs] removed unneeded grpio module requirements; deleted deprecated test_real.py. * [pylibs] loosen 'wheel' version requirement to avoid module reinstallations every time
EskoDijk
force-pushed
the
pr-migrate-ot
branch
from
a312c84
to
0de3e3b
Compare
[otns_main] cmdline arguments help string rephrased for clarity.
…a custom OT node pointed to from OT_DIR. This is required for OpenThread CI.
…ed by Git during rebase to ot-ns/main.
EskoDijk
force-pushed
the
pr-migrate-ot
branch
from
d8ea962
to
32e3d0f
Compare
Yes, I've now done a rebase. Not 100% sure if Github now sees it as "merge-able" because I don't see a message about that. CI tests are now running and I can try to fix the remaining errors after the runs are done. |
@wgtdkp I've now updated one test on YAML save/load that was failing, with some more checks. All other CI tests of OTNS2 were green. I see that CI also tries to run, or schedule, the old CI tests of the The check "CLA" also failed due to the commit limit and advises to manually check contributors. For me as contributor it's okay, maybe you or an OT team member could check this manually for co-author Vinicius Guimaraes [email protected] ? Finally, there was an open issue raised by Jonathan if we could use submodules pointing to specific commits of OpenThread 1.1, 1.2, and 1.3, instead of branches on my private repo. This would be possible but only if we apply a patch file locally as for example explained here. What should we do? Applying a patch has the disadvantage that the submodule repo's get into modified state which could cause confusion and issues with later on updating the OTNS local repo to newer versions. |
|
Related to this, there's also a PR for |
|
The CLA failure is:
It looks like there are too many commits in this PR :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes all the features/commits that were added for OTNS version 2. A summary of key features is provided in the updated
README.mdfile.