CPU/GPU/RAM utilization low but real time factor 0.1 with 100 actors
I have a pretty good simulation PC - gazebo and ROS are running in a chroot (I also tried outside the chroot). I am trying to simulate a swarm of robots. Problem is that the real time factor goes to the gutter exponentially. My actors arent that complicated - basically a diff controller, laser scan, wheels and a body on top. And even if they dont do anything (no publishing), the real time factor gets really low.
Funnily enough the CPU, GPU and RAM utilization do not seem to be so high (CPU: ~30%, GPU:~1%(headless), RAM: 40%).The gzserver process only uses about 12% of the CPU. Does anyone know where the bottle neck is? Is real time factor related to the number of actors/joints etc directly?
Manually changing max_step_size does bring up the real time update rate but this just means that gazebo does less computation instead of default 1000 times per second. But, why doesnt gazebo use all the CPU available? The only way I could get gazebo to use more CPU was to start mutliple instances of gazebo parallely.
I have seen several vague answers that claim that when the model is "complex", the real time factor goes low. But does anyone know why?
Some things that I have unsuccessfully tried (just to avoid answers suggesting things that I have already tried):
Collisions: I tried turning off collisions by commenting out collisions in urdf files and simplifying the collision models to use simple shapes.
Reduce CPU load: I tried removing laser scanner from the model and also use the gpu_ray plug in to improve CPU availability (although CPU seems to be under utilized anyway).
Simplify my model and use threads: I did a quick test with this very simple bot (basically a rectangle with wheels).
https://github.com/yangliu28/swarm_robot_ros_sim. Adding 100 of them gave a real time factor of 0.25 and CPU is still under utilized (total CPU load around 20% and gzserver using 12%). Using threads: According to http://gazebosim.org/tutorials?tut=pa..., I played around with island threading and finally got a 0.5 RT factor (for the simple robot model above) but with a total of 40% CPU load (gzserver was using only 20%).
