Home | Tutorials | Wiki | Issues
Ask Your Question
0

Message drops between plugins

asked 2018-12-04 08:08:57 -0600

Julian Oes gravatar image

In PX4 Gazebo we send messages between plugins. While trying to speed up the simulation, trying to run it faster than realtime, we discovered that the higher rate messages (e.g. IMU updates at 2500 Hz) quite often get dropped from one plugin to the next one. The message skips seem to be depending on CPU load and also increase if gzclient is running.

The publication in one plugin looks like this:

transport::PublisherPtr imu_pub_;
...
imu_pub_ = node_handle_->Advertise<sensor_msgs::msgs::Imu>("~/" + model_->GetName() + imu_topic_, 1);

And the publication is:

imu_pub_->Publish(imu_message_);

The subscription in the other plugin is just:

transport::SubscriberPtr imu_sub_;

imu_sub_ = node_handle_->Subscribe("~/" + model_->GetName() + imu_sub_topic_, &GazeboMavlinkInterface::ImuCallback, this);

And using a sequence variable in the message I'm checking if I miss updates in:

void GazeboMavlinkInterface::ImuCallback(ImuPtr& imu_message) {
...

What I found is that most times the callback is triggered correctly, however, sometimes skips of 1 happen, and seldom up to 10-20 can skip in a row. This is with a publish rate of 2500 Hz.

I also tried to use a queue bigger than 1 but this did not fix the actual issue and drops would still occur. Looking at Wireshark, I don't see this communication between the plugins as loopback TCP communication, so I'm assuming this is "local" communication?

Is this expected? Is there any setting or workaround I should try? I understand that this is a high rate, however, that's not the point because it can also happen at a lower rate given CPU usage is high. Ultimately, we would like to implement the simulation in lockstep with the controller, so we really should never miss a sample but track one timestamp all the way through the system.

edit retag flag offensive close merge delete

Comments

I was able to work around it by adding a sequence to the message and then sleeping in `OnUpdate` until the sequence has changed and we can be sure that we have received the latest message.

Julian Oes gravatar imageJulian Oes ( 2018-12-06 10:52:10 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-12-06 10:35:41 -0600

nkoenig gravatar image

This is not expected behavior. I haven't looked into this issue, but you could try sending messages using a different transport mechanism such as Ignition Transport, or ROS.

edit flag offensive delete link more

Comments

Interesting. So there is a mechanism that makes sure that all callbacks are called, so all messages are delivered? Because from how I understand it, it's up to the Subscriber to "fetch" the messages before the queue is full and samples are overwritten.

Julian Oes gravatar imageJulian Oes ( 2018-12-06 10:54:25 -0600 )edit

There isn't a mechanism that guarantees all callbacks are called. For example, if your callback takes too long to complete then you might miss messages.

nkoenig gravatar imagenkoenig ( 2018-12-06 10:57:25 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-12-04 08:08:57 -0600

Seen: 24 times

Last updated: Dec 06