Gazebo backend (gzserver) CPU hungry during multi-robot simulations

edit

I was trying to simulate multi-robot slam in gazebo using turtlebot3 (4-5 bots) in a very simple world - a closed square of side 10 units. During the simulation the Real Time Factor value seems to reduce drastically at times and process gzserver demands a lot of cpu power. Is this behaviour expected? If so is there any configuration changes I can do to make gazebo less resource hungry in a multi-robot setting?

Setup : Ubuntu 20.04 LTS + ROS Noetic + Gazebo 11