CMocka: Enabling XML Output

I’ve been adding unit tests to my C-based projects with the CMocka testing framework. Now that I have tests defined, I want my build server to run the tests and track the results for each build. Jenkins requires test results to be reported in the JUnit XML format. Luckily, CMocka has built in XML support that’s quite easy to enable.

Enabling XML Output

The first step in enabling XML output is to set the CMocka output format. The easiest approach is to set the output format in your test program. This API must be called before any tests are run:

cmocka_set_message_output(CM_OUTPUT_XML);

You can also change CMocka’s output behavior using the CMOCKA_MESSAGE_OUTPUT environment variable. Note that CMOCKA_MESSAGE_OUTPUT has higher precedence than the cmocka_set_message_output API, so you can override a test program’s default formatting behavior.

Possible values for CMOCKA_MESSAGE_OUTPUT are stdout, subunit, tab, or xml. Options are case-insensitive.

Supporting Both Standard and XML Outputs

The reason I want to use XML is to support my build server, but this doesn’t work that well for local development. I’d prefer to read the standard output. So my chosen approach now is to NOT set the default to XML within the test program, but to instead use the environment variables. This way I have two test targets in my build system:

  1. One that runs the test suite and generates XML output for use by the CI system
  2. One that runs the test suite and prints to standard output for use during development

I have found this to be the best of both worlds.

Configuring XML Output Destination

By default, XML output will be printed to stderr. In order to write the results to a file, you must define the environment variable CMOCKA_XML_FILE. If the file already exists or cannot be created, CMocka will output the results to stderr instead.

CMOCKA_XML_FILE=testresults/libc.xml buildresults/test/libc.bin

XML Output With Multiple Test Groups

I like to organize my tests into multiple groups. This causes CMocka to generate invalid XML, as too many top level tags exist in the file. Jenkins will fail to parse the file, and you will also find that tools like xmllint will report errors.

To work around this, you can enable CMocka to generate a separate XML file for each test group. Simply include “%g” in the filename definition. The “%g” characters will be replaced by the group name.

CMOCKA_XML_FILE=testresults/%g.xml

A Note on CMocka API Usage With XML Results

You must use the cmocka_run_group_tests or cmocka_run_group_tests_name APIs to successfully generate XML test results. All other testing APIs will result in summary output instead of XML output.

Putting it All Together

Here’s a sample from the Embedded Artistry libc repository (prior to it being converted to Meson). I place my test results inside of my buildresults directory, which is untracked and gets cleaned by the build server. This sample assumes that you’ve changed the default output to XML within the test program.

.PHONY: test
test:
ifeq ("$(wildcard buildresults/testresults/)","")
    @mkdir -p buildresults/testresults
else
    @rm -f buildresults/testresults/*
endif
    @CMOCKA_XML_FILE=buildresults/testresults/%g.xml buildresults/x86_64_debug/test/libc.bin

Nowadays I primarily use Meson. I will register the test program with the built-in test runner, passing it the necessary environment variables so that the program generates XML test output. I then register a custom target that allows me to invoke the test program without these environment variables, giving me a much better experience when I’m developing and testing locally.

test('flex_tests',
  flex_tests,
  env: [
    'CMOCKA_MESSAGE_OUTPUT=XML',
    'CMOCKA_XML_FILE=' + meson.build_root() + '/test/%g.xml']
)

run_target('flex-tests',
  command: [flex_tests]
)

Further Reading

Share Your Thoughts

This site uses Akismet to reduce spam. Learn how your comment data is processed.