Running Ignition Fortress headless in Docker for CI

asked 2021-12-26 00:56:19 -0500

Illuminatur
78 ●27 ●32 ●37

updated 2022-04-05 15:07:50 -0500

Dear Ignition Community,

i am trying to integrate Ignition Gazebo Fortress into a Continous Integration Pipeline in gitlab using Docker. Therefore, the installation instructions are followed inside Docker [1]. The linting steps, build steps, etc. are working so far.

For the test stage i have defined a *.test, where the ros_ign parameter bridge as well as ignition gazebo are to be started, since my test class is working on subscribed ROS topic, which are to be interpreted in terms of statistics (do lidars obtain the correct intensities, reflections, and so on ...).

The rostest file looks like:

<?xml version="1.0"?>
<launch>
    <include file="$(find ign_my_package)/test/test_ros_gpu_lidar_subscriber.launch"/>
    <test test-name="test_ros_gpu_lidar_subscriber" pkg="ign_my_package" type="test_ros_gpu_lidar_subscriber" time-limit="180.0" />
</launch>

The referenced test_ros_gpu_lidar_subscriber.launch file looks like:

<?xml version="1.0"?>
    <launch>
      <!-- Launch the bridge -->
      <node name="parameter_bridge_ros_subscriber" pkg="ros_ign_bridge" type="parameter_bridge" args="/lidar@sensor_msgs/LaserScan@ignition.msgs.LaserScan /lidar/points@sensor_msgs/PointCloud2@ignition.msgs.PointCloudPacked"/>
      <include file="$(find ros_ign_gazebo)/launch/ign_gazebo.launch">
        <arg name="ign_args" value="-r -s -v 4 gpu_lidar_retro_values_sensor.sdf --headless-rendering"/>
      </include>
</launch>

Since I have no rendering window available, i thought the recently released and mentioned EGL headless mode [2], would be the ideal application to enable a renderless runner via

 ign gazebo -s --headless-rendering

But when my CI pipeline comes the the test stage, where i am going to run some rostests based on the execution of launch files, i obtain the following error:

[Err] [Ogre2RenderEngine.cc:394] Unable to open display: terminate called after throwing an instance of 'Ogre::RenderingAPIException' what(): OGRE EXCEPTION(3:RenderingAPIException): No Interface could be loaded. Check previous error messages.Try disabling OpenGL plugin from plugins.cfg. in GlSwitchableSupport::GlSwitchableSupport at [...]

Are there any indications why this fails? Are there positive experiences integrating rostests with IGNITION GAZEBO (FORTRESS) within Docker headless?

Update:

Using the hints from the dolly example in the comments below

export DISPLAY=:1.0
export MESA_GL_VERSION_OVERRIDE=3.3

result to the following error:

[Err] [Ogre2RenderEngine.cc:394] Unable to open display: :1.0 terminate called after throwing an instance of 'Ogre::RenderingAPIException' what(): OGRE EXCEPTION(3:RenderingAPIException): No Interface could be loaded. Check previous error messages.Try disabling OpenGL plugin from plugins.cfg. in GlSwitchableSupport::GlSwitchableSupport [...]

I can think of two reasons for this:

Somehow, the combination of mesa/opengl/ogre-library combination does not fit well. But since i am installing the libraries as the default ignition tutorial proposes, this should work.
Second, maybe and somehow the way how my tests are build up using rostest and therefore starting ingition via launch-files (and the headless parameter) could also potentially bring in some problems. The dolly example is based on a dedicated test fixture class [3] starting up ignition.

Are there other examples using iginition headless using test suites, to see if

some packages/dependencies/whatever (preconditions for docker environment) are wrong or
the rostest integration is incorrect or not suitable for headless execution in CI/CD environments?

Best regards, Marc

References:

[1] https://ignitionrobotics.org/docs/for...
[2 ...

(more)

edit retag flag offensive close merge delete

add a comment

2 Answers

Sort by » oldest newest most voted

answered 2022-04-05 10:41:17 -0500

Illuminatur
78 ●27 ●32 ●37

updated 2022-04-05 15:10:00 -0500

Hi,

in the meantime i adopted my example (see above) to the solution, proposed by chapulina [0]. Therefore, the resulting test launch file only invokes a spawning mechanism for a robot model sdf, the IGN2ROS1 bridging stuff and a test fixture class - similar to the dolly one -, which loads a world description and defines the server to run headless via:

ignition::gazebo::ServerConfig config;
config.SetSdfFile(ignition::common::joinPaths(std::string(PROJECT_SOURCE_PATH), "worlds", "empty.sdf"));
config.SetHeadlessRendering(true);
ignition::gazebo::TestFixture fixture(config);

I tried to track the problem down to its roots, and found some deeper lying issues:

First, according to a recently merged tutorial [1] the following user to group assignment might be necessary (for ubuntu 20.04). Although the respective tutorial aims at the use within AWS, this could also apply to Docker. I have not yet been able to check this.

sudo usermod -a -G render user

Additionally, according to [2], it might be necessary to build Ogre2 with the compile flag

OGRE_GLSUPPORT_USE_EGL_HEADLESS

Since i installed ignition gazebo dolly's way by mimicing the shell script [3] and thereby the packages libignition-rendering6-ogre2 and libignition-rendering6-ogre2-dev are installed, I would assume that those libs had been built by this.

Analyzing the Ogre2 logs, which are stored inside ~/.ignition/rendering, the EGL devices seem to be missing:

17:39:48: *-*-* OGRE Initialising
17:39:48: *-*-* Version 2.2.6 (Cerberus)
[...]
17:39:48: Loading library /usr/lib/x86_64-linux-gnu/OGRE-2.2/OGRE/RenderSystem_GL3Plus.so
17:39:48: Installing plugin: GL 3+ RenderSystem
17:39:48: OpenGL 3+ Rendering Subsystem created.
17:39:48: OGRE EXCEPTION(3:RenderingAPIException): Couldn`t open X display :1.0 in     GLXGLSupport::getGLDisplay at [...]
17:39:48: Found Num EGL Devices: 0
17:39:48: OGRE EXCEPTION(3:RenderingAPIException): No EGL devices found! Update your GPU drivers and try again in EglPBufferSupport::EglPBufferSupport at [...]
17:39:48: EGL Headless raised an exception. Won't be available. Are drivers too old?

Delivering 0 EGL devices, it seems to be a problem with the EGL impl. of Ogre2 inside Docker.

Are there any other experiences, additional ideas or hints?

References:

[0] Run tests headless with EGL, https://github.com/chapulina/dolly/pu...

[1] https://github.com/ignitionrobotics/i...

[2] https://github.com/OGRECave/ogre-next...

[3] https://github.com/chapulina/dolly/bl...

edit flag offensive delete link

add a comment

answered 2021-12-28 17:14:39 -0500

chapulina
7474 ●2 ●23 ●54 https://louise.world/

Are there positive experiences integrating rostests with IGNITION GAZEBO (FORTRESS) within Docker headless?

Here's an example: https://github.com/chapulina/dolly/pu...

Maybe what you're missing are these environment variables:

export DISPLAY=:1.0
export MESA_GL_VERSION_OVERRIDE=3.3

edit flag offensive delete link

Comments

Hey chapulina,

thx lot. I just tried that solution, but that doesnt seem to work. Due to the limited amount of characters I am going to extend my problem statement above.

BR, Marc

Illuminatur ( 2022-01-05 04:21:12 -0500 )edit

Dear Louise,

in my docker environment, while copying almost the dolly's example shell script, i obtain the above mentioned error. Are there any special constraints how you setup the docker environment for testing dolly in the loop? Maybe any packages or constraints i am missing?

I am using the dolly approach using catkin instead of colcon, but it fails despite using the beforementioned environment variables this way:

[Err] [Ogre2RenderEngine.cc:394] Unable to open display: :1.0

BR, Marc

Illuminatur ( 2022-02-08 03:14:52 -0500 )edit

add a comment

Running Ignition Fortress headless in Docker for CI

2 Answers

Comments

Question Tools

Stats

Related questions

Running Ignition Fortress headless in Docker for CI edit

2 Answers

Comments

Question Tools

Stats

Related questions

Running Ignition Fortress headless in Docker for CI