Best practices for programmatic start, unpause, and shutdown of Gazebo simulation

edit

I did something like you mentioned in my work as research assistant in the university. Probably this will become open source in the next month. So I will describe how i did it and give you some hints.

I have 4 packages.

• first generate's yaml files with the description of the simulation (models, poses ,plugins, parameter, etc)
• second loads these files and starts all launchfiles, spawn objects, etc.
• an observer to determine if a test has ended and if it was successful
• a logger to collect datas like rosbag, test time, etc
• (5.) a small visualization library

I also used python and the subprocess-API this worked fine for me. More problematic is to kill all the programs/nodes that was running (specially action_server like move_base). So at the end of the test i just reset the simulation and only kill nodes that i really need to be kill. Also over time the memory usage of my system getting really high. Couldn't figure out why is that. But for this reason i restart the framework every 100 tests.

Hope that will help you. If you have more questions please ask me.

I am also working on something similar to run repetitive experiments. Some things that helped my in addition to similar things in the last answer are the following:

• I have a very simple ROS node that I always start with my simulation to unpause the simulation at every iteration to make sure all nodes have time to start.
• To end every iteration after a fixed max. simulation time, I start also this node with a timer using the simulation time with the required="true" option to kill all nodes in the launch file once the timeout is reached.
• I wrote a simulation runner script that starts the launch file with all my nodes, including Gazebo and then makes sure to store log files and the ROS bag in a user-defined folder, being also possible to override some of the input arguments of the launch file using this YAML configuration in order to allow an optimizer, for example, to change the parameters of the simulation run. This simulation pool also orchestrates the different runs just in case you want to have multiple simulations running in parallel.

I have found out that the number of Gazebo instances running at the same time cannot be too high, otherwise they start having Seg Faults, showing some errors that are related to the rendering. I still haven't figured out why that happens. If you pack your simulation in a Docker container, you might also control that by setting, for example, the number of CPUs used, that helped my simulations a little bit so I could run multiple instances on the same machine.

I would just advise you not to use Gazebo 9 right away because of this issue. The gzserver is not being finished correctly after the gazebo_ros node dies, so it starts building up a bunch of processes with each iterations. Starting ROS_MASTER_URI and GAZEBO_MASTER_URI with different ports every time like done here in the simulation wrapper script is really important, especially if you want to still keep working on other things and running simulations manually while your simulations run in the background.