Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REP-2017] Thread attributes configuration support in rcl #385

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

smorita-esol
Copy link

This REP proposes an infrastructure of ROS 2 receiving thread attributes configuration, which the executors utilize in the language binding layers (such as rclcpp, rclpy, and others).

For the discussion on some forums, please refer to the links below.
[Realtime WG]
ros-realtime/ros-realtime.github.io#18
[ROS Discourse]
https://discourse.ros.org/t/adding-thread-attributes-configuration-in-ros-2-framework/30701

And, for sample implementations, please refer to the PRs below.
ros2/rclcpp#2205
ros2/rcl#1075
ros2/rcutils#424
ros2/rcpputils#179

@ros-discourse
Copy link

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/adding-thread-attributes-configuration-in-ros-2-framework/30701/8

rep-2017.rst Outdated
Motivation
==========

The configuration and setting of thread attributes for ROS 2 executor threads are essential for controlling computational resources and meeting real-time requirements, which are typically mandatory for robot systems.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are several executors available. is your proposal applied to all of them?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@razr
Thanks for your feedbacks.

there are several executors available. is your proposal applied to all of them?

In the following pull request, I tried to demonstrate how existing executors can use this infrastructure(currently, only for single thread/multithread executors.)
ros2/rclcpp#2205
But, throughout the discussion on the pull request above, I'm reconsidering our approach now.
I will share the new approach, extending the existing executor to use this infrastructure afterward.

And, of course, you could apply this infrastructure for the newly developed executor that would come in the future.

Abstract
========

This REP proposed a feature of the ros client library (rcl) to accept the thread attributes, such as priorities, scheduling, and core affinity, which are to be passed to the language bindings like rclcpp.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure whether accept is the right word here. Maybe: This REP proposes an extension of the ROS Client Library that allows modification of the thread attributes, such as priority, scheduling policy, and CPU affinity to be used throughout the ROS 2 code base.

  • Do you extend ROS Utils or ROS Client Library?
  • Does it also allow to modify a thread name?
  • Is CPU Affinity POSIX compliant?
  • Is it a goal to stay POSIX compliant?
  • Do you support Windows as well?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@razr

Do you extend ROS Utils or ROS Client Library?
Yes. The prototype implementations have been shared as the pull requests below.
ros2/rcl#1075
ros2/rcutils#424

Does it also allow to modify a thread name?

If "it" refers to executors in the language bindings (e.g., rclcpp), yes, they can use parameters provided through command line arguments or environment variables to set their own thread names.

Is CPU Affinity POSIX compliant?

CPU core affinity is not POSIX compliant but is prevalent in most POSIX-like environments.

Is it a goal to stay POSIX compliant?

Not strictly, but we might as well refer to the specification to decide the practical parameter set.

Do you support Windows as well?

This proposal is not specific to any environment because the extension is just to receive the thread parameters at the rcl layer. But, if you want to use this infrastructure from above the language bindings (like rclcpp), you will need to implement some environment-specific code or modules to use those thread attributes, as I proposed for rcpputils in the pull request below.
ros2/rcpputils#179
In the above pull request, I proposed the rclcpp::thread class as an extension of C++ std::thread. Currently, the extension is only applicable to Linux environments. And, for other environments (like Windows or Mac environments), the rclcpp::thread acts like existing std::thread. But, you can extend the rclcpp::thread to use the thread attributes settings for Windows or Mac environments by adding some implementations to rclcpp::thread.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure whether accept is the right word here.

I've replaced it with "receive" in the commit below.
c06a453

rep-2017.rst Outdated

The configuration and setting of thread attributes for ROS 2 executor threads are essential for controlling computational resources and meeting real-time requirements, which are typically mandatory for robot systems.

These attributes are particularly critical for embedded robot systems that lack redundant resources, as they aid in optimizing CPU resource utilization.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to mention embedded here? I think it is applicable to any robotic system with deterministic behavior.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed the description referring to the embedded system.

c06a453

rep-2017.rst Outdated

These attributes are particularly critical for embedded robot systems that lack redundant resources, as they aid in optimizing CPU resource utilization.

However, currently, there is a lack of infrastructure to configure and set thread attributes for the threads used by the ROS 2 executors. As a result, specific code tailored to each target environment must be implemented. The implementation not only requires significant effort but poses some obstacles to continuous integration applied to the systems with multiple execution environments.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand tailored to each target environment

  • How is your proposal related to the CI/CD? I'm confused here. :)
  • Are your proposal applied during start-up, runtime, or both?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed the description referring to the CI/CD.

c06a453

rep-2017.rst Outdated

Having an infrastructure that configures thread attributes for ROS 2 executor through command line parameters, environment variables, and/or files eliminates the need for additional implementation in each environment and simplifies the deployment process across multiple environments without requiring code modifications.

The purpose of this REP is to propose a feature to the ros client library that allows it to accept thread attributes, store them internally, and provide interfaces to pass them to upper language bindings (e.g., rclcpp, rclpy, or future ones). By incorporating this feature and enabling language bindings to read the thread attributes, ROS 2 can effectively adopt them for real-time systems or environments that require strict CPU resource optimization, eliminating the need for code modifications in each specific environment.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does it mean to accept thread attributes and store them internally? What component will set the thread attributes? Maybe we could provide here a Hello, World example, e.g. ros2 launch helloworld <set some thread attributes>?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does it mean to accept thread attributes and store them internally?

You can take it literally.
With this proposal, the rcl receives a set of thread attribute parameters and stores it in it once during the process launch phase.
Then, after the process starts, the language binding layer retrieves the parameters via the newly added interface of rcl.

What component will set the thread attributes?

And then, the language layer (e.g. rclcpp or rcl, and so on) uses the parameters to set the thread attribute of the threads used by its executor(s).

Maybe we could provide here a Hello, World example, e.g. ros2 launch helloworld ?

As I mentioned briefly at the previous Realtime-WG meeting, I'm currently reproducing the benchmark results provided by the WG removing the Linux-specific thread control API.
And I'm also considering if I could share the reproducing procedure with you or not.

Please note that the list above is subject to modification during the review process for this REP.
For instance, it may be necessary to include an option specifically designed for extension purposes in a non-POSIX environment.

Using the provided keys, users should create an array of the thread attribute parameters like the one below.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why it is an array?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@razr

Why is it an array?

Is your question why we should handle multiple sets of thread attribute parameters for a process? If so, I assume that there are cases where we need to have multiple executors per process, as below.
https://github.com/ros-realtime/reference-system/blob/main/autoware_reference_system/src/ros2/executor/autoware_default_prioritized.cpp

In such cases, we need to provide a way to receive multiple thread attribute configurations. Among these configurations, the language binding layer can choose and apply the appropriate parameter set to each thread used by the executors.

RCUTILS_RET_ERROR
an unspecified error occured.

Rationale
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this chapter about ROS parameter?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@razr

Why do we need this chapter about ROS parameter?

Here, I'd like to explain the rationale of the reason why I decided to add a new interface to rcl for thread attributes setting, not using the existing ROS parameters infrastructure.
And I placed the next section, "Being implemented in the rcl, not in language bindings," for a similar reason.

rep-2017.rst Outdated

These attributes are particularly critical for embedded robot systems that lack redundant resources, as they aid in optimizing CPU resource utilization.

However, currently, there is a lack of infrastructure to configure and set thread attributes for the threads used by the ROS 2 executors. As a result, specific code tailored to each target environment must be implemented. The implementation not only requires significant effort but poses some obstacles to continuous integration applied to the systems with multiple execution environments.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please describe how it works now, e.g. what scheduler and what priority are used if there are no thread settings made?

Copy link
Author

@smorita-esol smorita-esol Sep 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@razr

could you please describe how it works now, e.g. what scheduler and what priority are used if there are no thread settings made?

It depends on how we implement the newly developed executor.
https://github.com/ros2/rclcpp/pull/2205/files#r1293464704

For example, the current proposal above for single/multithread executors acts like the existing ones when rcl receives no thread attribute sets. (e.g., No thread attributions are set.) But, if rcl receives any thread attribute set(s), each executor applies it(them) to each executor.

However, as I mentioned below, I'm reconsidering the behavior above of the existing executors.
https://github.com/ros-infrastructure/rep/pull/385/files#r1323789820


According to the document about the Client libraries [#REF-4]_, every language bindings have their own thread model. But, the thread attributes are OS-specific, not language. So, treating the thread attributes in the rcl commonly used by the language bindings is natural and effort-saving.

Backward Compatibility
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens with ROS 2 code where people already set thread attributes using available OS functions?

Copy link
Author

@smorita-esol smorita-esol Sep 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@razr

what happens with ROS 2 code where people already set thread attributes using available OS functions?

There is no effects to the existing ROS 2 applications because the rcl with the modification of this proposal does not affect the upper layers, including applications.
Of course, you can modify those applications to use the proposed infrastructure so that you would be able to change thread attribute parameters without modification of the code.

rep-2017.rst Outdated

::

+-----------+ +------------+
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is your proposal applicable for the rclpy as well?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@razr

is your proposal applicable for the rclpy as well?

In a strict sense, it can be applied to rclpy. However, some additional code will be required to make it compatible with the newly-added rcl interface in rclpy. Additionally, we may need to implement an additional Python thread control class, similar to what our team has done with rcpputils.

@ros-discourse
Copy link

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/threaded-callback-with-priority-affinity-and-overrun-handler/14977/4

@razr
Copy link

razr commented Sep 21, 2023

@MichaelOrlov maybe you could provide your feedback as well.

Copy link
Contributor

@fujitatomoya fujitatomoya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smorita-esol thanks for the enhancement proposal.

1st review, i have several comments to understand more details before design review.

rep-2017.rst Outdated
@@ -0,0 +1,359 @@
REP: 2017
Title: Thread attributes configuration support in rcl
Author: Masaaki Ueno, Koichi Oyama, Shoji Morita <[email protected]>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add email address for each author to identify?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add email address for each author to identify?

I've added all the email addresses in the commit below.
c06a453

rep-2017.rst Outdated

The configuration and setting of thread attributes for ROS 2 executor threads are essential for controlling computational resources and meeting real-time requirements, which are typically mandatory for robot systems.

These attributes are particularly critical for embedded robot systems that lack redundant resources, as they aid in optimizing CPU resource utilization.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, thread attributes are basically irrelevant to resource constrained environment, because if that is requirement, we need to set those configuration for high-spec computers as well.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed the description related to the resource-constrained environment in the commit below.
c06a453


However, currently, there is a lack of infrastructure to configure and set thread attributes for the threads used by the ROS 2 executors. As a result, specific code tailored to each target environment must be implemented. The implementation not only requires significant effort but poses some obstacles to continuous integration applied to the systems with multiple execution environments.

Having an infrastructure that configures thread attributes for ROS 2 executor through command line parameters, environment variables, and/or files eliminates the need for additional implementation in each environment and simplifies the deployment process across multiple environments without requiring code modifications.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in general, thread attributes are sometimes required for real-time system. these settings typically need to access the native-system.
this means that those are really bound to the native-system, so it is really hard to have the consistent APIs for platform agnostic ways.
(we obviously do not want to rely on some APIs that sometimes work but sometimes do not.)

and if we can have the consistent APIs to conceal the platform system dependency to set thread attributes, what is the main motivation to have those abstraction only for ROS 2?
if we have this abstraction library to set thread attributes, it can be used for any middleware and application framework, probably it would be better to keep that library for application-agnostic way for everyone?

it would make more sense to me,

  • we have open source platform-agnostic thread priority control library.
  • ROS 2 rcl and client libraries can rely on that library to control the thread attributes.
  • Introduce ROS 2 specific attributes based on robot and robotics use cases, these are meant to be applied to ROS 2 application.

i really do not see the justification that we need to have all of them in ROS 2... could you elaborate the motivation a bit?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in general, thread attributes are sometimes required for real-time system. these settings typically need to access the native-system.
this means that those are really bound to the native-system, so it is really hard to have the consistent APIs for platform agnostic ways.
(we obviously do not want to rely on some APIs that sometimes work but sometimes do not.)

The API (rclcpp::thread) proposed in the pull request below will maintain consistency in terms of interface for all the environments.
ros2/rcpputils#179
But, the behaviors of the APIs are not identical for each environment. For example, the API will result in an error in an environment that does not support a Round-Robin scheduling policy when the user tries to set the scheduling policy. But, such a behavior will be plausible for users.

Anyway, as you metioned before in other places, it should be a crucial point if the OSRF accepts to maintain the API that only returns an error for specific environments.

and if we can have the consistent APIs to conceal the platform system dependency to set thread attributes, what is the main motivation to have those abstraction only for ROS 2?
if we have this abstraction library to set thread attributes, it can be used for any middleware and application framework, probably it would be better to keep that library for application-agnostic way for everyone?

If we had such an almighty thread control library, we would be able to choose a construction like the one below, where the library is separated from the ROS 2 framework. And the library would be beneficial for a wide range of usage.

2023-10-05-11-25-05

But, if there were such a library, we should have an infrastructure to pass a set of thread attributes to the library. And this REP-2017 proposes such an infrastructure.

it would make more sense to me,

we have open source platform-agnostic thread priority control library.
ROS 2 rcl and client libraries can rely on that library to control the thread attributes.
Introduce ROS 2 specific attributes based on robot and robotics use cases, these are meant to be applied to ROS 2 application.
i really do not see the justification that we need to have all of them in ROS 2... could you elaborate the motivation a bit?

Regarding the first item, there is currently no thread control library for C++ that is able to control thread attributes and is environment agnostic. Even for future C++ standards, there are only discussions regarding thread names and stack sizes, as below.

cplusplus/papers#817

And, the thread-controlling feature of BOOST, whose dependency in ROS 2 has been removed, is not environment agnostic.
Therefore, we have to devise a new one, and it is plausible to start only with ROS 2 use cases to have minimum versatility for the first step, as we proposed the rclcpp::thread.

And, for the second item, in ROS 2 architecture, we have to place the thread-controlling feature in the language bindings. However, the thread attribute is not language-specific but environment-specific. So, it is plausible to treat thread attributes in the rcl layer, which is used by all the language bindings.

Regarding the last one, as I said above, my team wants to start with ROS2 use cases and generalize the thread attribute parameters enough for the usage. And then, if we find it beneficial for other purposes, we would like to separate it from the ROS 2 platform.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the behaviors of the APIs are not identical for each environment. For example, the API will result in an error in an environment that does not support a Round-Robin scheduling policy when the user tries to set the scheduling policy. But, such a behavior will be plausible for users.

this is an exact concern for me, it would be probably nice to keep those code in the application for the platform? so that user application have the full control to the platform. i was expecting that even with this APIs, the application ends up having the platform dependent code in the application because behavior is not consistent.

But, if there were such a library, we should have an infrastructure to pass a set of thread attributes to the library. And this REP-2017 proposes such an infrastructure.

i understand this REP is meant to the thread priority interfaces to ROS 2. but i think that it would be better to discuss on this detail after above consensus is met? (we can of course keep this open and discuss, but IMO i would like to have consensus with above approach.)

my team wants to start with ROS2 use cases and generalize the thread attribute parameters enough for the usage. And then, if we find it beneficial for other purposes, we would like to separate it from the ROS 2 platform.

as we talked offline, i would take opposite way for us(Sony). having general thread priority management library and abstraction would useful any edge IoT applications, we would use this ROS-agnostic way in the application, and create the ROS 2 Executors. instead, this feature only applied to ROS 2 application.

again, i think that this is really nice feature but i cannot convince me to keep this idea in the ROS 2 core. i would like to have more feedbacks from maintainers and developers.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is an exact concern for me, it would be probably nice to keep those code in the application for the platform? so that user application have the full control to the platform. i was expecting that even with this APIs, the application ends up having the platform dependent code in the application because behavior is not consistent.

This REP does not conflict with your idea that the application developers may place environment-specific code in their application. For such a developer, he or she can decide not to use this infrastructure.

On the other hand, there would probably be developers want

  • to delegate the thread attribute control to the ROS 2 infrastructure and/or
  • to change the thread attributes without code modification and/or
  • not to place such environment-specific code in their applications.

as we talked offline, i would take opposite way for us(Sony). having general thread priority management library and abstraction would useful any edge IoT applications, we would use this ROS-agnostic way in the application, and create the ROS 2 Executors. instead, this feature only applied to ROS 2 application.

I agree with you that there is an approach you mentioned.

However, it's worth noting that currently, there are no practical management libraries available, even though it seems beneficial, as you mentioned. Designing a thread abstraction scheme that is both system-agnostic and versatile for any purpose might be rather challenging.

In such a situation, it is one of the practical ways to start small and gradually expand our efforts from the real-time executor's use cases, as I mentioned above.

i would like to have more feedbacks from maintainers and developers.

I completely agree with you.

rep-2017.rst Outdated

These attributes are particularly critical for embedded robot systems that lack redundant resources, as they aid in optimizing CPU resource utilization.

However, currently, there is a lack of infrastructure to configure and set thread attributes for the threads used by the ROS 2 executors. As a result, specific code tailored to each target environment must be implemented. The implementation not only requires significant effort but poses some obstacles to continuous integration applied to the systems with multiple execution environments.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if there is the interface that application can register the threads to ROS 2 Executors? in that case, user application can manage full attributes in the application code, and the ROS 2 Executors use those thread to execute the events.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My team will propose the usage pattern below to give thread attribution to the ROS 2 executor from the user application, as follows:

  1. A user application retrieves the configured thread attribution from rcl via the newly added API proposed in this REP.
  2. The user application chooses a corresponding thread attribute parameter for the executor to be launched.
  3. The user application launches the executor, passing the thread attribute parameter through the executor's constructor.

My team will share the extension proposal for the existing single-thread/multi-thread executor within a few weeks in the following thread:

ros2/rclcpp#2205

I believe that it will be worth proposing because we currently cannot pass thread attribute parameters to ROS 2 executors.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

although this idea works to set the thread priorities to threads running inside the Executor, that application can register the threads to ROS 2 Executors can provide more straight-forward from user application perspective? probably we would need both to change the thread priority during runtime, i think.

rep-2017.rst Outdated
.. code-block:: YAML

- priority: 20
name: thread-1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How user can know this thread does exist in the process space that is going to run? (e.g 3rd party developed node.)

and if it does not exist, this command line fails to start the application since it cannot set the requested thread attributes? or print the warning and fall back to normal thread attributes set? just curious how to behave this command line interface.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, in this REP's proposal, rcl does not generate any threads. rcl just receives the thread attribute parameters and stores in it to be retrieved from the language binding layers.

rcl only checks whether the thread attribute parameters passed are correct or not in terms of YAML grammar. It is the upper language binding layer to check if the parameters are acceptable for the environment or not.

For the existing applications you are concerned, there is no effect on them because the extension of this REP does not affect spontaneously.
But, for those applications, my team will provide some extensions to enable them to use this infrastructure. In the proposal, the rclcpp's single/multithread executors define the reserved names for thread parameter names as follows:

  • RCLCPP_EXECUTOR_MULTI_THREAD
  • RCLCPP_EXECUTOR_SINGLE_THREAD

And, if they find the thread attribute having those names as string parameters, the executors use the parameters for the thread attributes used by the thread of each executor.
With this extension, the existing applications that do not use environment-specific thread control APIs will be able to use the newly added infrastructure.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI

But, for those applications, my team will provide some extensions to >enable them to use this infrastructure. In the proposal, the rclcpp's >single/multithread executors define the reserved names for thread >parameter names as follows:

  • RCLCPP_EXECUTOR_MULTI_THREADED
  • RCLCPP_EXECUTOR_SINGLE_THREADED

The extension above has been included in the commit below.

esol-community/rclcpp@35af6fa

And you can get started with the document below, using the benchmark applications (autoware_default_singlethreaded/autoware_default_multithreaded).

https://github.com/esol-community/reference-system/blob/rep2017_demo/misc/getting_started_thread_attribute_configuration_extention_rep2017.md

* Scheduling policy
* Priority

When using the parameter passing feature, users should create an array containing sets of these items as thread attribute parameters for the thread pool used by ROS 2 executors. The text format that users should comply with for describing parameters is YAML. Users can choose to pass the parameters either as a string or from a file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There might be multiple executors running in the process space. in that case, how does this behave?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if the process has multiple executors, a user application can be implemented to choose appropriate ones for each executor, as I mentioned earlier.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the process has multiple single threaded executors? Does your proposed solution described in #385 (comment) cover this case?

It would be helpful to see some usage examples with different scenarios. That would help to understand how the proposed solution works top to bottom.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@carlossvg

What if the process has multiple single threaded executors? Does your proposed solution described in #385 (comment) cover this case?

Yes. In such cases, as described below, the user application retrieves a thread attribute for each single thread executor and then passes it to the appropriate single thread executor one by one.

[Applications]

Applications retrieve thread attributes from rcl and pass one of them to the executor's constructor. The executor then configures the thread(s) accordingly.

It would be helpful to see some usage examples with different scenarios. That would help to understand how the proposed solution works top to bottom.

Our team is going to update the pull-request to demonstrate how applications pass the thread attributes to the existing executors (single thread/multit hread) and create a tutorial to show how we can use the infrastructure by using an existing sample application like below.
https://github.com/ros-realtime/reference-system

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@carlossvg

What if the process has multiple single threaded executors? Does your proposed solution described in #385 (comment) cover this case?

It would be helpful to see some usage examples with different scenarios. That would help to understand how the proposed solution works top to bottom.

Sorry for my late update.
The usage example explaining that our proposal is able to cover the case where a ROS 2 process has multiple single-threaded executors is prepared, as below.

[Getting started]
https://github.com/esol-community/reference-system/blob/rep2017_demo/misc/getting_started_thread_attribute_configuration_extention_rep2017.md

[code]
https://github.com/esol-community/reference-system/blob/rep2017_demo/autoware_reference_system/src/ros2/executor/autoware_default_prioritized_using_rep2017.cpp

Seeing the code above, you will find there are two methods to use the thread attributes.

The first one is to retrieve them by user application code (USE_USER_IMPLEMENTED_ATTR_FIND_ROUTINE). In this method, the user application code retrieves the thread attributes and passes them to the executor via a parameter for its constructor. Then, the executor sets the passed attribute to the thread it creates.

The second is to delegate the retrieving procedure to executors by passing some hints as the constructor's parameters. In this method, given a hint (like RCLCPP_EXECUTOR_HOTPATH), the executor searches the dedicated thread attribute matching it and sets the attribute to the thread created by the executors.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smorita-esol Thank you, now it's more clear to me.

I found a bit confusing how the default thread attributes are set for executors. This is because there is no one to one mapping between threads, executors and thread options.

name field in rcutils_thread_attr_s struct refers to a thread name. However, this seems to me more like the name of a thread attributes configuration. This configuration can be reused in different thread or executor.

In the reference-system example you shared, these names would refer to "RCLCPP_EXECUTOR_HOTPATH" and "RCLCPP_EXECUTOR_PLANNER". That is, this is not the name for a specific thread but the name of a configuration you want to reuse for multiple threads.

I would simply change the meaning of this name so it's clear there is not need to have a thread with that name.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@carlossvg
Thanks for your feedback.

I found a bit confusing how the default thread attributes are set for executors. This is because there is no one to one mapping between threads, executors and thread options.

Does your concern relate to the situation where there are no thread attributes being passed, either via a command line parameter or via an environment variable?
If so, the default behavior is the same as the existing one. In other words, the thread attributes for the caller's thread will be inherited by the called executor(s).

name field in rcutils_thread_attr_s struct refers to a thread name. However, this seems to me more like the name of a thread attributes configuration. This configuration can be reused in different thread or executor.

It is a good point indeed. My intention is that the field is used as a hint (or hints) for the retrieving process in applications or executors. Considering your comment, "tag" is more suitable than "name" for the purpose.

name:              String value representing the name of the thread
->
tag:               String value used as a hint (or hints) to retrieve an appropriate parameter set in application code or executors.

Does the above go along with your opinion?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does your concern relate to the situation where there are no thread attributes being passed, either via a command line parameter or via an environment variable?

Not really, the default behavior is fine for me. My comment was related to the hint/tag comment.

Does the above go along with your opinion?

Yes, I find this name more suitable. Thanks.

+-------------------------------+


Parameter configuration
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i believe that user calling these parameter configuration, must have the permission to change the thread scheduling polity and priority, unless these operation could fail?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course, the upper language binding layers or user applications will fail to set the thread attributes without appropriate privileges.

@ros-discourse
Copy link

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-2023-09-21/33733/1

- Added email addresses for all authors.
  ros-infrastructure#385 (comment)
- Deleted descriptions related to the CI/CD and embedded systems in the abstract.
  ros-infrastructure#385 (comment)
  ros-infrastructure#385 (comment)
- Added a concise explanation of related enhancements, including upcoming REPs, for the implementation of real-time extensions in the ROS 2 environment.

Additionally, updated to support the multiple core affinity, which previously supported only a single core affinity per thread.

Signed-off-by: Shoji Morita <[email protected]>
@fujitatomoya
Copy link
Contributor

@smorita-esol thanks for iterating, i will check the related PRs for more understanding.

smorita-esol added a commit to esol-community/rcl that referenced this pull request Oct 26, 2023
smorita-esol added a commit to esol-community/rcpputils that referenced this pull request Oct 26, 2023
smorita-esol added a commit to esol-community/rcutils that referenced this pull request Oct 26, 2023
@MichaelOrlov
Copy link

@MichaelOrlov maybe you could provide your feedback as well.

@razr I have a concern that Windows implementation is stubbed and the current API for the feature is tightly tailored to the POSIX implementation. If one would try to implement Windows version it will likely require API changes or will impose some limitations and will require changes in the REP itself.
Thread priorities on Windows are implemented differently than in the POSIX-compatible systems and I don't see that it was taken into account and provided enough abstraction.

cc: @clalancette

@smorita-esol
Copy link
Author

@MichaelOrlov maybe you could provide your feedback as well.

@razr I have a concern that Windows implementation is stubbed and the current API for the feature is tightly tailored to the POSIX implementation. If one would try to implement Windows version it will likely require API changes or will impose some limitations and will require changes in the REP itself. Thread priorities on Windows are implemented differently than in the POSIX-compatible systems and I don't see that it was taken into account and provided enough abstraction.

cc: @clalancette

Your concern is the same as Tomoya's and will be discussed either now or in the future when the need arises.
Our team is open to changing the proposed API if necessary and is curious to know if there are requirements for real-time use cases in the Windows environment. If there are no requirements, the limitation mentioned above does not matter because no one uses this infrastructure within that environment.
Otherwise, if there are any use cases for the Windows environment, we might have to project the Windows thread attribute features on the proposed infrastructure or change it to suit the features, as you mentioned. But the projection should not be completely covered against the POSIX's ones.

smorita-esol added a commit to esol-community/rclcpp that referenced this pull request Dec 14, 2023
… infrastructure

proposed in the REP-2017 below.

ros-infrastructure/rep#385

Signed-off-by: Shoji Morita <[email protected]>
smorita-esol added a commit to esol-community/reference-system that referenced this pull request Dec 15, 2023
the infrastructure proposed in REP-2017 below and some sample
YAML files of thread attribute settings for the existing
executors (Singlethreaded/Multithreaded).

ros-infrastructure/rep#385

Signed-off-by: Shoji Morita <[email protected]>
smorita-esol added a commit to esol-community/reference-system that referenced this pull request Dec 20, 2023
using the infrastructure proposed in the REP-2017 below.

ros-infrastructure/rep#385

The demo is to explain how to use the infrastructure in the process
that has multiple single-threaded executors as a reply to the question below.

ros-infrastructure/rep#385 (comment)

The YAML files for the existing executors are to demonstrate that
the existing executors can be enhanced with our proposed infrastructure,
supporting our mention below.

ros-infrastructure/rep#385 (comment)

Signed-off-by: Shoji Morita <[email protected]>
@ros-discourse
Copy link

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-for-2024-01-18/35779/1

smorita-esol added a commit to esol-community/rclcpp that referenced this pull request Jan 26, 2024
smorita-esol added a commit to esol-community/rcpputils that referenced this pull request Jan 26, 2024
smorita-esol added a commit to esol-community/rcutils that referenced this pull request Jan 26, 2024
smorita-esol added a commit to esol-community/rcl that referenced this pull request Jan 26, 2024
@ros-discourse
Copy link

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-for-2024-03-21/36814/1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants