Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of the control loop #7

Open
dogoepp opened this issue Mar 18, 2016 · 64 comments
Open

Improve performance of the control loop #7

dogoepp opened this issue Mar 18, 2016 · 64 comments

Comments

@dogoepp
Copy link
Member

dogoepp commented Mar 18, 2016

The default frequency for this control loop is 10 Hz for us. Let's set cycle_time_error_threshold to 0.1, i.e. the maximum allowed duration of an iteration to ensure 10 Hz iteration frequency. I observe on Positronic that the threshold is regularly exceeded (see bellow). Admittedly, the excess time is very little, but it shows that we are already close to the limit.

[ WARN] [1458293591.557913852]: Cycle time exceeded error threshold by: 3.7721e-05, cycle time: 0.100037721, threshold: 0.1
[ WARN] [1458293592.057749621]: Cycle time exceeded error threshold by: 2.0122e-05, cycle time: 0.100020122, threshold: 0.1
[ WARN] [1458293592.557678208]: Cycle time exceeded error threshold by: 4.261e-05, cycle time: 0.100042610, threshold: 0.1
[ WARN] [1458293593.158254929]: Cycle time exceeded error threshold by: 0.00104895, cycle time: 0.101048953, threshold: 0.1
[ WARN] [1458293593.358195718]: Cycle time exceeded error threshold by: 0.0010338, cycle time: 0.101033797, threshold: 0.1
[ WARN] [1458293595.658286463]: Cycle time exceeded error threshold by: 0.00106704, cycle time: 0.101067042, threshold: 0.1
[ WARN] [1458293595.858179759]: Cycle time exceeded error threshold by: 0.00102926, cycle time: 0.101029257, threshold: 0.1
[ WARN] [1458293596.857821194]: Cycle time exceeded error threshold by: 1.7817e-05, cycle time: 0.100017817, threshold: 0.1
[ WARN] [1458293597.057700941]: Cycle time exceeded error threshold by: 5.8185e-05, cycle time: 0.100058185, threshold: 0.1
[ WARN] [1458293598.257216257]: Cycle time exceeded error threshold by: 2.11e-05, cycle time: 0.100021100, threshold: 0.1
[ WARN] [1458293598.357235889]: Cycle time exceeded error threshold by: 2.033e-05, cycle time: 0.100020330, threshold: 0.1
[ WARN] [1458293598.458147884]: Cycle time exceeded error threshold by: 0.0009539, cycle time: 0.100953900, threshold: 0.1
[ WARN] [1458293600.757213708]: Cycle time exceeded error threshold by: 1.0134e-05, cycle time: 0.100010134, threshold: 0.1
[ WARN] [1458293600.858150496]: Cycle time exceeded error threshold by: 0.000973175, cycle time: 0.100973175, threshold: 0.1
[ WARN] [1458293601.457944227]: Cycle time exceeded error threshold by: 2.1029e-05, cycle time: 0.100021029, threshold: 0.1
[ WARN] [1458293603.157214574]: Cycle time exceeded error threshold by: 3.4369e-05, cycle time: 0.100034369, threshold: 0.1
[ WARN] [1458293603.258169032]: Cycle time exceeded error threshold by: 0.00100335, cycle time: 0.101003347, threshold: 0.1
[ WARN] [1458293603.957935811]: Cycle time exceeded error threshold by: 1.621e-05, cycle time: 0.100016210, threshold: 0.1

One possible axis of improvement is the find operations on _dynamixel_corrections in each read_joints() and write_joints(). Nonetheless, the C++ reference states that it's an operation with log complexity. The impact of these operations should be limited, because until know, we never used more than 18 actuators at the same time.

The alternative choice is to consider that the control loop cannot run at such frequency. I think it would be a sad limitation.

@naoki-mizuno
Copy link
Contributor

naoki-mizuno commented Apr 9, 2018

I'm also trying to figure out why I'm only getting 5 Hz for the joint_states with 7 servos daisy-chained at 1Mbps baud rate.

average rate: 5.207
        min: 0.189s max: 0.195s std dev: 0.00220s window: 5
average rate: 5.210
        min: 0.189s max: 0.195s std dev: 0.00141s window: 11
average rate: 5.208
        min: 0.189s max: 0.195s std dev: 0.00126s window: 16
average rate: 5.145
        min: 0.188s max: 0.239s std dev: 0.01047s window: 21
average rate: 5.158
        min: 0.188s max: 0.239s std dev: 0.00949s window: 26
average rate: 5.082
        min: 0.188s max: 0.241s std dev: 0.01444s window: 31
average rate: 5.100
        min: 0.188s max: 0.241s std dev: 0.01348s window: 36
average rate: 5.082
        min: 0.188s max: 0.241s std dev: 0.01443s window: 41
average rate: 5.096
        min: 0.185s max: 0.241s std dev: 0.01377s window: 46
average rate: 5.107
        min: 0.185s max: 0.241s std dev: 0.01313s window: 51

It seems like the recv is taking time (0.015 sec/call, twice per servo for position and speed, 7 servos total becomes 0.015 * 2 * 7 = 0.21 which is about 5 Hz). I changed the method to read more bytes at a time rather than byte-by-byte, but didn't see any improvement in the rate. Seems like the guys at dynamixel workbench are getting 30+ Hz so I would think my servos can do faster.

Reference:
http://forums.trossenrobotics.com/showthread.php?6690-Dynamixels-communication-speed

@jbmouret
Copy link
Member

The find should be really negligible here.

I think this comes more from how we broadcast commands and wait for replies.

Also, do the dynamixel_workbench code query the position of each servos at each time step? This takes a significant time (versus sending commands and that's it).

@naoki-mizuno
Copy link
Contributor

naoki-mizuno commented Apr 10, 2018

do the dynamixel_workbench code query the position of each servos at each time step?

I believe so. Although, they're also reading data for each servo (and not using "sync read" or "bulk read"), so I'm not sure where the difference comes from 😕 That being said, I haven't actually used dynamixel_workbench myself so I'll need to test it out on my servos to do a comparison.

@dogoepp
Copy link
Member Author

dogoepp commented Apr 17, 2018

One difference between DynamixelSDK and Libdynamixel is that they use a non-blocking call to read(). We were already considering this option which shall be tested (hopefully soon).

Also, They have slightly different settings for the serial interface, and there is a latency_timer thing.

For further reading, their reading method (for protocol 1) is here..

@gjjab
Copy link

gjjab commented May 17, 2018

Excuse me
Is this problem solved?
I also have the same problem!!
I use the latest dynamixel_workbench to controll 7 motors in ROS kinetic !!
How can I solve this problem about low frequency dynamixel state???
screenshot 2018-05-17 15_26_42

@dogoepp
Copy link
Member Author

dogoepp commented May 22, 2018

This problem is under investigation. I plan on going back to it from today on.

If you want to help on this investigation, it seems that most of the time is waisted in read(), in src/dynamixel/controllers/usb2dynamixel.hpp

@naoki-mizuno
Copy link
Contributor

@dogoepp

I plan on going back to it from today on

👍 Do you think it's not necessary to use bulk_read?

@dogoepp
Copy link
Member Author

dogoepp commented May 24, 2018

I think that it should help but I am not sure that it is the root cause, since it seems that robotis themselves do not use it.

@dogoepp
Copy link
Member Author

dogoepp commented Jun 1, 2018

I found a few, very interesting, lines in DynamixelSQK. They point to latency_timer which setting we can change.

It appears that by putting it to 1 (ms), we were able to run the control loop at 100 Hz with 5 actuators.

@naoki-mizuno could you confirm this observation ?

Next I'll see how we can use commands to read and write for all joints at once. This is expected to further improve the loop frequency.

@naoki-mizuno
Copy link
Contributor

@dogoepp I gotta say, you just made my day 😆 I set the latency timer to 1, and I was able to get the joint states at 50 Hz with 7 servos. I'm not sure how I overlooked that parameter when reading through their code, but I'm pretty happy with how smooth our manipulator works now!

@dogoepp
Copy link
Member Author

dogoepp commented Jun 8, 2018

Great ! I hope that the next step will make it even better.

@SammyRamone
Copy link

Hi sorry for digging this thread up. I just stumpled on it randomly
We actually achieved 400Hz at 4Mbaud reading (pos, vel, eff) and writing (pos) 20 servos. Setting the latency timer to 0 and using sync reads and writes can increase the frequency a lot.
You can find our implementation for a dynamixel ros_control hardware interface here:
https://github.com/bit-bots/bitbots_lowlevel/tree/master/bitbots_ros_control

@PedroDesRobots
Copy link
Contributor

Hi @SammyRamone ,
Thx you for your advice.
But which kind of Dynamixel motors did you use? (I guess Dynamixel pro series in your case). Actually, we are working with Dynamixel MX-28T which are limited to 1Mbaud.

@SammyRamone
Copy link

In my case MX-64 and MX-106. If you update the firmware you can use up to 4.5Mbaud. In reality 4.5Mbaud doesn't seem to work, but 4Mbaud does.
http://emanual.robotis.com/docs/en/dxl/mx/mx-28-2/#baud-rate

@dogoepp
Copy link
Member Author

dogoepp commented Feb 6, 2019

Sounds great. I think indeed that using sync read and writes could help a lot. If I recall correctly, either of these commands is only available in the version 2 of the protocol however. Good to know that implementing this would allow for much greater loop frequencies.

As for the firmware, it might be that some older versions of MX-28 cannot be upgraded.

Out of curiosity, could I ask why you have implemented your own hardware interface? Was the barrier of entry to use this one too high?

@SammyRamone
Copy link

Actually you can also use sync reads in the protocol version 1. Out of some reason its not documented, but it works. We used to do it before we switched to version 2. If you want to do it I can dig out our old code.

Yes thats possible. I remember that we had some problems with updating MX-28 servos but we didn't bother since we stopped using them.

Actually I implemented most of this over a year ago. I think at that point your software was not finished. I just had a look at it today because I thought about switching to yours since it doesn't make sense to have thousands of implementations for the same thing. But currently I'm missing the sync read and write functions.

Btw: Did you set the return delay time in the servos to 0? This is also a common source for low frequency. I don't really know why robotis did set the default value to something else than 0.

Second btw: we just developed a multi-bus controller board solution which enables us to read/write 20 servos with 1.3KHz, if you need even higher frequencies

@dogoepp
Copy link
Member Author

dogoepp commented Feb 7, 2019

I would like to help, but I do not have access to Dynamixels anymore and am rather busy. If @PedroDesRobots / @dalinel and/or you are willing to go this way, I would be happy to give a hand, at least to find the parts in the code that need modification.

I was not aware of the return delay time. It should be checked.

About your board, how is it connected to the computer? How do you handle the multiple buses?

@PedroDesRobots
Copy link
Contributor

Firstly, I have set the return delay time in the servos to 0. It was the default value 250 ms... as expected it increases the transmission speed. I was plotting time delay in my controller loop (update function) and that divided by two the time.

Concerning updating MX-28 servos I will try it soon. And I will take a look at sync reads and how to add it in our code.

@SammyRamone could you tell us more about your multi-bus controller? Because we are really interesting to increase our frequencies to be able to produce better controllers in velocity mode.

@SammyRamone
Copy link

I recommend to always set the return delay time to 0 automatically in all servos when starting up your software. Otherwise you will change a servo or update firmware at some point and wonder why your update rate dropped again. Trust me, I've been through this :D
Keep in mind that even when setting delay time to 0, it will still be around 50us (I don't know why, probably just because robotis writes bad firmware).

Regarding updates of the MX-28: I think the older versions had a chip with less memory and the new firmware doesn't fit on it. Therefore they don't work. But I'm not totally sure about this.

Regarding the multi-bus: We just took an multi channel FTDI chip (similar to the robotis U2D2 but with 4 channels). Currently we just let our software run 4 times on the different bus systems for tests. But it should be possible to have 4 hardware interfaces where the controllers connect to, but I didn't try it yet.
I'm currently writing a paper about it, hopefully it will be finished next week. I can send you the first version of it then, if you are interested.

@PedroDesRobots
Copy link
Contributor

I have upgraded the MX-28 firmware (v36 to v39) under the software RoboPlus. It was impossible to do it under R+ Manager... I don't know why but R+ never detect my servos but I suspect to use protocol 2 only.

Now I can increase the baudrate at 3 Mbauds (it didn't propose 4MBauds in the soft...) BUT when I'm trying to read address 5, which is the address number in control table for baudrate, it returns 1 (1Mbaud in protocol 1 or 57600 in protocol 2). In the end, the servo responds only at 3Mbauds expected result but only under protocol 1. I set the return delay time at 0 but it responds only for a --scan-timeout of 0,05. While with MX-28 v36 set at 1 Mbaud we are able to reach a scan-timeout of 0,001.

@SammyRamone could tell me which soft do you use to update servos (RoboPlus or R+) and how do you manage to communicate with protocol 2 ?
Do you know if we are able to recognize old MX-28 from new version ?

Btw, with our MX-28 servo v36 set at 1 Mbaud we are able to reach 1000Hz in our control loop. when I'm trying to communicate with 18 servos, the best result is around 20Hz.... and normally we should be able to reach 50Hz easily (~1000Hz / 18 servos ) but I guess it's not linear. Of course, our solution to use multi-channel FTDI chip is the best way to carry on to increase speed. Which kind of multi-channel FTDI do you use? If you can send me the paper it will be pleasant.

@jbmouret
Copy link
Member

jbmouret commented Feb 8, 2019

So, 1khz with a single Mx-28, but 20Hz with 18 Mx-28. Right? Which usb2dynamixel did you use? we have three options.

We could also simply use 4 USB adapters with a good USB hub.

Also, @SammyRamone might be using the 4-wires servos.

@dogoepp
Copy link
Member Author

dogoepp commented Feb 11, 2019

@jbmouret four wires or three wires are the same in term of communication : half-duplex.

@PedroDesRobots could you clarify your second paragraph? It is hard to figure what are your observations, under which circumstances.

In addition, It may be simpler to start by moving to sync read and write before going the way of multiple hardware controllers. This is no modification to existing hardware and has the potential of an immediate improvement, for free, for all users of this software.

@SammyRamone
Copy link

Hi, sorry for replying so late. There were just too many deadlines.
@PedroDesRobots We're using R+. RoboPlus does not support the newest firmwares if I'm not mistaking. We can communicate and set values without problems using R+. But we basically use this only for flashing the firmware. Otherwise we use our ros_control software that I posted earlier for communication. For debugging we also made a small ROS package with a few hacky scripts:
https://github.com/bit-bots/bitbots_lowlevel/tree/master/bitbots_dynamixel_debug

Regarding the MX-28: I'm not totally sure about these, since we stopped using them. You should be able to differentiate the versions by the chip that is installed in them. But again, I'm not totally sure about the 28s.

Regarding your 20Hz : I would guess one of them still has a return delay time. Best way to investigate this is using a logic analyser. I can recommend the one from Saleae. That way you can actually see what is happening on the bus.

@jbmouret I used RS-485 and TTL both (4 and 3 wires). There is no real difference in them, except that RS-485 works better on long distance transmissions and on bad cables.

The paper is unfortunately still in a state where I can't send it around. But I can sum up the most important points.
We used a FT4232H chip which is basically 4 times the FT232H (which is used in the Robotis U2D2) in one chip. It's basically the same as pluging in 4 U2D2s, just smaller and more convenient. Since the USB2 baudrate is much higher than the ones used on the dynamixel buses (480 Mbaud), it is no problem to merge 4 buses into one USB bus. Our results showed that it performs equally for a single bus as a single FTDI chip, so there seems no advantage on using multiple single chips.
We also compared this to other approaches which use a STM32 chip as a controller board (such as this one https://github.com/bit-bots/DXLBoard ) and found out that the FTDI approach is better, since it has lower latency in processing than STM32 chip (which was what we expected).
Overall we reached 1.3kHz with 20 servos reading 10 bytes and writing 4 in each cycle, when using 4 buses. On one bus we reached only 400Hz. So it scales well (but not linear since you have additional header bytes). The theoretical maximum would be much higher (700Hz for one bus) but it can't be reached since the servos need around 50us to respond, even when delay time is 0. This response delay is the same independent from your baudrate, therefore update rates do not scale linear to increasing baudrate.

When the paper is ready, I will be happy to share it with you.
Feel free to ask any further question, I will try to answer quicker than the last time ;)

@raess1
Copy link

raess1 commented Jun 29, 2020

I have some problem getting high loop speeds. I am running 12 x MX-28R in one setup with U2D2 and the servos are updated to protocol 2.0 and running at 4 MBPS. I get around 50hz. The servos are Daisy chained in Groups of 3.

My second setup will be 8 x MX-106R and 4 x MX-64R. Also Daisy chained in Groups of 3. But havent tried the loop speed on this setup yet….

I have set the return delay time in the servos to 0. (as discussed in improve loop speed thread)
and latency_timer to (1)ms

Any suggestions ? :)

@jbmouret
Copy link
Member

Hi

The specialists are @Aneoshun and @PedroDesRobots (but he is on vacation this week). We definitely get much higher frequencies with MX-28.

We have limited access to the lab right now (covid-19 restrictions), but we can have a look at the end of the week, maybe.

@raess1
Copy link

raess1 commented Jun 30, 2020

thanks @jbmouret I did some troubleshooting last night an it seems that I forgot to set return delay time to zero for one servo. With that fixed I can now hit around 70hz at the joint states topic for 12 servos. What rates could one expect?

I read that @SammyRamone suggest to set latency_timer to zero for improved freq. Will that have a big impact on the rate?

btw: First time on the legs
https://www.youtube.com/watch?v=Ix_kLGgavYM

@dogoepp
Copy link
Member Author

dogoepp commented Jun 30, 2020

Nice video and congratulations ! It's always nice to get to the point where the robot actually does something.

I guess that @jbmouret or @PedroDesRobots will know the exact loop frequency we can get for our robots (those with 18 servos). I think that it could run at 100 Hz.

If I recall correctly the driver would always make us wait the duration of latency timer before our software could access any new data from the servos. It thus had a serious impact on the loop frequency, as we are systematically requesting servo angular position, thus waiting for data. There is a little information on the matter in the readme of libdynamixel, admittedly too scarce.

I would then advise to put this setting to a low value. I think that we used as low as 0.

@SammyRamone
Copy link

SammyRamone commented Jun 30, 2020

The U2D2 should be fast enough to give you a decent update rate with only 12 servos.
There are following possible problems limiting your update rate:

  1. As it has been said, the latency timer has to be set to 0. There is some command which sets it to "low_latency", but this is still acutally 1ms. I use sudo sh -c "echo '0' > /sys/bus/usb-serial/devices/ttyUSB0/latency_timer" to set it to 0. You may need to change ttyUSB0 to something else.
  2. If I understand the code correctly, each value (position, velocity,...) of each servo is read with a single read. This is super slow. Not only do you need to transmit more bytes as in a sync read (due to additional headers), you also have to wait multiple times to for the servos to respond, which takes for the MX-64 and MX-106 around 50us. The MX-28 are maybe even slower as they have a different microcontroller. You should only send one sync read and one sync write per update cycle. You can find more details about this in my paper (https://www.researchgate.net/publication/337716753_High-Frequency_Multi_Bus_Servo_and_Sensor_Communication_Using_the_Dynamixel_Protocol).
  3. Check your main computer CPU. Is it fast enough to handle this update rate or is it at 100% workload? Also check if the library your using is not "busy waiting" during reads (which the robotis dxl library did in earlier versions), since your CPU core can do nothing else while waiting for the status package to be returned. This leads to having more other things to do between handling packages, thus a longer latency from your main computer.
  4. Check if you have any package losses. If one of your servos does not respond to a package, the master waits till the time out, which is quiet long. Some servos may also just sometimes not respond correctly.
  5. Do you have any other devices that you have connected to the bus, e.g an IMU? Speaking with those takes away additional bus time and depending on the model, they may also answer slowly. If you are looking for a fast IMU with dxl bus compatibility, you can have a look at ours https://github.com/bit-bots/bitbots_imu_dxl

Generally speaking, when having such problems, you should investigate what is actually running on your bus, to see what takes which amount of time. The easiest way to do this, is to use a logic analyser. I'm using one from Saleae and it works very well.
Also implement in your code, that the ROM registers (especially "return delay time") are set correctly on start up. It happens to easily, that one servo has wrong values and messes up everything.

@raess1
Copy link

raess1 commented Jun 30, 2020

Thanks @SammyRamone for the detail respons. Will try out some of your suggestions when I get home from work.

Some commments.
2. If I remember correctly I use the libdynamixel DEV Branch and dynamixel_control_hw devel Branch from my understanding have implemented buld_write and sync_read for Protocol 2.0.

  1. I am using the Jetson nano at MAXN . Will check the CPU usage later.

  2. I am running nothing else on the BUS. no IMU or nothing at the moment. I have adis 16465-a that I think I will use.

best regards
Robin

@SammyRamone
Copy link

I don't have experience with the libdynamixel, but generally a bulk_write is still slower as a sync_write due to having a larger package length. Still, it is much better than using single writes.
For the sync_read, it still makes a difference if you read all registers with one sync read or use multiple ones.

I'm using forks of the DynamixelSDK and Dynamixel Workbench from Robotis. I implemented reading multiple registers with a single sync read there. You can find it here:
https://github.com/bit-bots/DynamixelSDK/tree/master
https://github.com/bit-bots/dynamixel-workbench/tree/master

@raess1
Copy link

raess1 commented Jul 1, 2020

I just tried the ROBOTIS dynamixel_workbench and hitting around 330hz for 12 servos @2Mbaud over U2D2 for the MX-28R

And hitting 1khz for 3 MX-64R @4Mbaud

5B605E27-32F0-4023-9FAB-67A0C4008D0D

@SammyRamone
Copy link

Nice :)
Just out of curiosity to know how MX28 compare in speed: how many values were you reading? Only position or also velocity and effort? In a single or multiple sync reads?

@raess1
Copy link

raess1 commented Jul 2, 2020

Yeah didnt check :( just ran the dynamixel controller from here without modifying anything.

btw : if I run your suggestion for latency_timer I cant find any dynamixels. If I stick with latency_timer = 1 it works fine.
I ran your suggestion:
sudo sh -c "echo '0' > /sys/bus/usb-serial/devices/ttyUSB0/latency_timer"

I tried to brake out your bitbots_ros_control without all the extras. But still havent got it to work yet :(
Maybe I can run the orginal dynamixel_workbench_controllers if I can figuare out to set homing position or mounting offset.

I would rather use this lib since my robot is working already with it. But the hz is still a problem.

@SammyRamone
Copy link

The code looks like they are doing a single sync read for all 3 values. Seems like they improved their software over the last years. Then the 330Hz are also very close to what my formula would predict. Looks like the MX-28 are behaving similar to the MX-64/106.

Hm, thats strange. I already used it on multiple computers and it worked. But I didn't use the U2D2 but my own controller board, maybe there is a difference. Maybe it also has to do something with different kernel versions or something.

Yeah our code has a lot of specific extras. I would also think, that the standard dynamixel_workbench would be fine for you.

@raess1
Copy link

raess1 commented Jul 2, 2020

So i tried the mx-28 @4Mbaud and got around 450hz.
F7F7B40C-7BD0-4F7F-AD9A-58E9D61105A9

This is the error I get when using Latency timer = 0
559C7DE9-102E-4039-9ABB-E45C76525361

Also so that workbench supports multibus. So maybe for the big dog I get a few more U2D2 and run the loop at 1khz. :-)

Do you know how to set startup offset similar to what this lib uses on workbench? :)

🙏

@raess1
Copy link

raess1 commented Jul 2, 2020

@SammyRamone robot is up and running with workbench. I am able to hit 930hz on dynamixel/joint_states with a single U2D2. 😇
Still have latency timer = 1.

But now the robot is shaking a bit instead 🤔

I just tought you would like that info.

@SammyRamone
Copy link

So for the latency error, I have no idea.
I don't think that this library has a function for setting the offset. But you can set the offset via register directly in the servo, maybe this would be usable for your application.

Hm, maybe have a look on the data. It could happen that this library "claims" to have 930Hz while not actually giving the robot this frequency. I can recommend to move the servo and use PlotJuggler, with points activated in the plot, to have a look at the current position. If all values are different and you have a smooth curve its great. If you have some doubled values, the library is not really reading the servo at this speed. If you have longer periods of the same value, you have package loss and timeouts.

Thanks for the info, its interesting for me to know :)

@raess1
Copy link

raess1 commented Jul 3, 2020

Yeah I dont know why Rostopic hz gives me does valuta.

I am checking with rtq and get other values.
64336958-ED58-465C-9A95-BF24542B9A66
243642A6-E1DA-4CF1-812F-E492443FE906

Here is the jidder I am trying to fix.

https://youtu.be/x7afjRIYC0U

I will try plotjuggler tonight

@SammyRamone
Copy link

yeah 930Hz sounded higher than possible. The 320Hz the topic monitor shows is more realistic.

Your video looks a bit more like a problem with the PID controller (oscillation). But a plot might give more insight.

@raess1
Copy link

raess1 commented Jul 4, 2020

Yeah maybe. But If I change the loop frequency of the controller rqt give me high frequency again. Like 800-1000hz.

What plot would be helpful to check? Just check the joint position plot and see how smooth the curve is?

Yeah maybe it is something with the PID. But if I start the dynamixel_workbench without trajectory controller i don’t experience any oscillation. The joint are stiff. If I start the trajectory controller the oscillation starts and the joint are not as stiff as before. I tried to adjust the PID to original values which worked in the ResiBots hardware interface but without any major success.

Here is the latest test
https://youtu.be/Sp1nXspIky4

@raess1
Copy link

raess1 commented Jul 4, 2020

So here is the latest test with a 1khz loop. Adjusted a few things.
A49505F3-D3C1-4C1B-A4F0-B39C8E3C30AE

So around 940hz on joint states.
Checking the TF I see a downstream on robot_state_publisher at around 580hz.
65-70% CPU load on jetson nano maxN

@SammyRamone
Copy link

Just plot the current position of a joint (from joint_states message) while you move it. If you get multiple times the same value after each other, the servo is not actually read as often as the message is published.

Maybe it has something to do with the trajectory controller. I don't have much experience with it.

@PedroDesRobots
Copy link
Contributor

Hi @SammyRamone and @raess1,

Thank you for your interest in your library and its performance. It's clearly not an easy task to compare all these things together.
We gave a presentation in June 2019 at the ROSCon FR about the performance of our library : presentation slide

For a short summary, the performance of our library depends on :

  • Baudrate
  • Latency timer
  • Instruction (REG_WRITE et BULK_READ)
  • actuators number

Note 1: the performance it's clearly not linear which is a counter-intuitive result.
Note 2 : the average loop is an indicator but keeps an eye on the global data distribution (this is why the plot are using violin plot).

READ POSITION ONLY :
dynamixel_performance

READ AND WRITE :
dynamixel_performance_read_write

@jbmouret
Copy link
Member

jbmouret commented Jul 6, 2020

@PedroDesRobots This is with a 1Mbs baudrate?

@PedroDesRobots
Copy link
Contributor

Yes, I forgot to mention that.
MX-28 : 3 M Bauds
XL-320 : 1 M Bauds

So with MX-28 updated with protocol 2 and 4 M Bauds, the performance should be higher.

@raess1
Copy link

raess1 commented Jul 6, 2020

Thanks for the slide @PedroDesRobots.
I haven’t archived more then 70hz for the MX-28 with this lib. What branch did you use too achieve that?

As I said I am running 4MBaud with latest firmware on P2. The frequency seems a lot higher on the workbench controller.

@PedroDesRobots
Copy link
Contributor

I was using libdynamixel branch dev adn dynamixel_control_hw branch MX.

You should also set the latency timer to 0, but I guess you already did that.

Let me know if you got some trouble installing these branches.

@jbmouret
Copy link
Member

jbmouret commented Jul 6, 2020

@PedroDesRobots We should plan to merge these branches, right?

@PedroDesRobots
Copy link
Contributor

@jbmouret yes completely for the libdynamixel.

Concerning dynamixel_control_hw with the templated function is hard to be generic, we have to specify the actuator type.

@dogoepp
Copy link
Member Author

dogoepp commented Jul 6, 2020 via email

@raess1
Copy link

raess1 commented Jul 6, 2020

@PedroDesRobots Let me make a clean install with does and check frequency for a single Mx. I be back 😎

@raess1
Copy link

raess1 commented Jul 6, 2020

I am trying to catkin_make my workspace after install the libdynamixel. (I followed the guide for
libdynamixel)

-- +++ processing catkin package: 'dynamixel_control_hw'
-- ==> add_subdirectory(dynamixel_control_hw)
-- Using these message generators: gencpp;geneus;genlisp;gennodejs;genpy
-- Configuring done
-- Generating done
-- Build files have been written to: /home/rf/resibot_ws/build
####
#### Running command: "make -j4 -l4" in "/home/rf/resibot_ws/build"
####
Scanning dependencies of target dynamixel_control_p1
Scanning dependencies of target dynamixel_hw
Scanning dependencies of target dynamixel_control_p2
[ 16%] Building CXX object dynamixel_control_hw/CMakeFiles/dynamixel_hw.dir/src/hardware_interface.cpp.o
[ 33%] Building CXX object dynamixel_control_hw/CMakeFiles/dynamixel_control_p1.dir/src/dynamixel_control.cpp.o
[ 50%] Building CXX object dynamixel_control_hw/CMakeFiles/dynamixel_control_p2.dir/src/dynamixel_control.cpp.o
In file included from /home/rf/resibot_ws/src/dynamixel_control_hw/src/hardware_interface.cpp:2:0:
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp: In member function ‘virtual void dynamixel::DynamixelHardwareInterface<Protocol>::read(const ros::Time&, const ros::Duration&)’:
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp:282:71: error: ‘using element_type = class dynamixel::servos::Mx28 {aka class dynamixel::servos::Mx28}’ has no member named ‘get_current_positions_MX’; did you mean ‘get_present_position’?
         _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_positions_MX<id_t>(_ids_vector)); //servos::Mx28
                                                                       ^~~~~~~~~~~~~~~~~~~~~~~~
                                                                       get_present_position
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp:282:100: error: expected primary-expression before ‘>’ token
         _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_positions_MX<id_t>(_ids_vector)); //servos::Mx28
                                                                                                    ^
In file included from /home/rf/resibot_ws/src/dynamixel_control_hw/src/hardware_interface.cpp:2:0:
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp:308:71: error: ‘using element_type = class dynamixel::servos::Mx28 {aka class dynamixel::servos::Mx28}’ has no member named ‘get_current_speed_MX’; did you mean ‘get_present_speed’?
         _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_speed_MX<id_t>(_ids_vector)); //servos::Mx28
                                                                       ^~~~~~~~~~~~~~~~~~~~
                                                                       get_present_speed
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp:308:96: error: expected primary-expression before ‘>’ token
         _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_speed_MX<id_t>(_ids_vector)); //servos::Mx28
                                                                                                ^
In file included from /home/rf/resibot_ws/src/dynamixel_control_hw/src/dynamixel_control.cpp:38:0:
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp: In member function ‘virtual void dynamixel::DynamixelHardwareInterface<Protocol>::read(const ros::Time&, const ros::Duration&)’:
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp:282:71: error: ‘using element_type = class dynamixel::servos::Mx28 {aka class dynamixel::servos::Mx28}’ has no member named ‘get_current_positions_MX’; did you mean ‘get_present_position’?
         _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_positions_MX<id_t>(_ids_vector)); //servos::Mx28
                                                                       ^~~~~~~~~~~~~~~~~~~~~~~~
                                                                       get_present_position
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp:282:100: error: expected primary-expression before ‘>’ token
         _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_positions_MX<id_t>(_ids_vector)); //servos::Mx28
                                                                                                    ^
In file included from /home/rf/resibot_ws/src/dynamixel_control_hw/src/dynamixel_control.cpp:38:0:
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp:308:71: error: ‘using element_type = class dynamixel::servos::Mx28 {aka class dynamixel::servos::Mx28}’ has no member named ‘get_current_speed_MX’; did you mean ‘get_present_speed’?
         _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_speed_MX<id_t>(_ids_vector)); //servos::Mx28
                                                                       ^~~~~~~~~~~~~~~~~~~~
                                                                       get_present_speed
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp:308:96: error: expected primary-expression before ‘>’ token
         _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_speed_MX<id_t>(_ids_vector)); //servos::Mx28
                                                                                                ^
In file included from /home/rf/resibot_ws/src/dynamixel_control_hw/src/dynamixel_control.cpp:38:0:
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp: In member function ‘virtual void dynamixel::DynamixelHardwareInterface<Protocol>::read(const ros::Time&, const ros::Duration&)’:
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp:282:71: error: ‘using element_type = class dynamixel::servos::Mx28 {aka class dynamixel::servos::Mx28}’ has no member named ‘get_current_positions_MX’; did you mean ‘get_present_position’?
         _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_positions_MX<id_t>(_ids_vector)); //servos::Mx28
                                                                       ^~~~~~~~~~~~~~~~~~~~~~~~
                                                                       get_present_position
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp:282:100: error: expected primary-expression before ‘>’ token
         _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_positions_MX<id_t>(_ids_vector)); //servos::Mx28
                                                                                                    ^
In file included from /home/rf/resibot_ws/src/dynamixel_control_hw/src/dynamixel_control.cpp:38:0:
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp:308:71: error: ‘using element_type = class dynamixel::servos::Mx28 {aka class dynamixel::servos::Mx28}’ has no member named ‘get_current_speed_MX’; did you mean ‘get_present_speed’?
         _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_speed_MX<id_t>(_ids_vector)); //servos::Mx28
                                                                       ^~~~~~~~~~~~~~~~~~~~
                                                                       get_present_speed
/home/rf/resibot_ws/src/dynamixel_control_hw/include/dynamixel_control_hw/hardware_interface.hpp:308:96: error: expected primary-expression before ‘>’ token
         _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_speed_MX<id_t>(_ids_vector)); //servos::Mx28
                                                                                                ^
dynamixel_control_hw/CMakeFiles/dynamixel_hw.dir/build.make:62: recipe for target 'dynamixel_control_hw/CMakeFiles/dynamixel_hw.dir/src/hardware_interface.cpp.o' failed
make[2]: *** [dynamixel_control_hw/CMakeFiles/dynamixel_hw.dir/src/hardware_interface.cpp.o] Error 1
CMakeFiles/Makefile2:454: recipe for target 'dynamixel_control_hw/CMakeFiles/dynamixel_hw.dir/all' failed
make[1]: *** [dynamixel_control_hw/CMakeFiles/dynamixel_hw.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
dynamixel_control_hw/CMakeFiles/dynamixel_control_p1.dir/build.make:62: recipe for target 'dynamixel_control_hw/CMakeFiles/dynamixel_control_p1.dir/src/dynamixel_control.cpp.o' failed
make[2]: *** [dynamixel_control_hw/CMakeFiles/dynamixel_control_p1.dir/src/dynamixel_control.cpp.o] Error 1
CMakeFiles/Makefile2:528: recipe for target 'dynamixel_control_hw/CMakeFiles/dynamixel_control_p1.dir/all' failed
make[1]: *** [dynamixel_control_hw/CMakeFiles/dynamixel_control_p1.dir/all] Error 2
dynamixel_control_hw/CMakeFiles/dynamixel_control_p2.dir/build.make:62: recipe for target 'dynamixel_control_hw/CMakeFiles/dynamixel_control_p2.dir/src/dynamixel_control.cpp.o' failed
make[2]: *** [dynamixel_control_hw/CMakeFiles/dynamixel_control_p2.dir/src/dynamixel_control.cpp.o] Error 1
CMakeFiles/Makefile2:491: recipe for target 'dynamixel_control_hw/CMakeFiles/dynamixel_control_p2.dir/all' failed
make[1]: *** [dynamixel_control_hw/CMakeFiles/dynamixel_control_p2.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
Invoking "make -j4 -l4" failed
rf@rf-desktop:~/resibot_ws$ 

@dogoepp
Copy link
Member Author

dogoepp commented Jul 6, 2020

Just as a side-note, @raess1, could you please wrap code or raw text, like these errors, in triple backticks (this character : `) on blank lines ? It makes it more readable, as it uses a fixed-width font.

@raess1
Copy link

raess1 commented Jul 6, 2020

roget that @dogoepp fixed code above :)

@PedroDesRobots
Copy link
Contributor

Hi @raess1,

Did you install the right branch? libdynamixel branch dev and dynamixel_control_hw branch MX?

@raess1
Copy link

raess1 commented Jul 6, 2020

@PedroDesRobots yep I did git checkout on does branches!

@PedroDesRobots
Copy link
Contributor

And did you clean properly your old install on libdynamixel, then reinstall it?

This error : error: ‘using element_type = class dynamixel::servos::Mx28 {aka class dynamixel::servos::Mx28}’ has no member named ‘get_current_positions_MX’; did you mean ‘get_present_position’? _dynamixel_controller.send(std::make_shared<servos::Mx28>(0)->get_current_positions_MX<id_t>(_ids_vector)); //servos::Mx28 shows that it didn't find the function. This function is clearly define in the libdynamixel branch dev.

@raess1
Copy link

raess1 commented Jul 6, 2020

@PedroDesRobots I git cloned libdynamixel in resibot_ws/src

Did these steps
cd libdynamixel
git checkout dev
./waf configure
./waf build
sudo ./waf install

Then added path for new install in the CMakelist.txt

include_directories(
  ${catkin_INCLUDE_DIRS}
  "${libdynamixel_dir}/home/rf/resibot_ws/src/libdynamixel"
  include
)

And removed lines 24-38 CMakelist.txt Since they where not working for me.


# Searching for libdynamixel
if ("$ENV{RESIBOTS_DIR}" STREQUAL "" AND "${LIBDYNAMIXEL}" STREQUAL "")
  message("Using default location for libdynamixel : ~")
  set(libdynamixel_dir "~")
elseif (NOT ("${LIBDYNAMIXEL}" STREQUAL ""))
  message("Using user-defined location for libdynamixel : ${LIBDYNAMIXEL}")
  set(libdynamixel_dir "$ENV{LIBDYNAMIXEL}")
elseif (NOT ("$ENV{RESIBOTS_DIR}" STREQUAL ""))
  message("Using enviromnetal-defined location for libdynamixel : $ENV{RESIBOTS_DIR}")
  set(libdynamixel_dir "$ENV{RESIBOTS_DIR}")
endif()

if (NOT (EXISTS "${libdynamixel_dir}/include/dynamixel/"))
  message(FATAL_ERROR "libdynamixel was not found!")
endif()

# Searching for libdynamixel
if ("$ENV{RESIBOTS_DIR}" STREQUAL "" AND "${LIBDYNAMIXEL}" STREQUAL "")
  message("Using default location for libdynamixel : ~")
  set(libdynamixel_dir "~")
elseif (NOT ("${LIBDYNAMIXEL}" STREQUAL ""))
  message("Using user-defined location for libdynamixel : ${LIBDYNAMIXEL}")
  set(libdynamixel_dir "$ENV{LIBDYNAMIXEL}")
elseif (NOT ("$ENV{RESIBOTS_DIR}" STREQUAL ""))
  message("Using enviromnetal-defined location for libdynamixel : $ENV{RESIBOTS_DIR}")
  set(libdynamixel_dir "$ENV{RESIBOTS_DIR}")
endif()

if (NOT (EXISTS "${libdynamixel_dir}/include/dynamixel/"))
  message(FATAL_ERROR "libdynamixel was not found!")
endif()

@martindehmel94
Copy link

Hello guys,

I just read through your discussion. I am writing my masterthesis on the control a two-link manipulator using two Dynamixel XM340-W210 Servos. I was wondering how to further reduce the latency when reading out the position and velocity values.
I set the latency timer to 0 ms, as suggested above, but in this case no status packet is returned anymore, which results in an error of the Dynamixel SDK. I'm using group_sync_read with 4MBaud, and already set the return delay time to zero. So I have to leave the port latency at 1ms and am currently controlling the motors with 500 Hz (=2ms). Any suggestions to speed up the reading process?

Regards,
Martin

@martindehmel94
Copy link

Btw., this is my reading/writing time measurement at the moment:

test03

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants