10 August 2020 by Klemens Morgenstern • Last updated 14 December 2021

Today we have a guest post from Klemens Morgenstern, an embedded C++ consultant. You can learn more about Klemens on his website.

Metal.Serial: Putting it All Together

I am finally somewhat done working on my metal.serial tool. This tool makes it easy to communicate with a device through a serial port (or any byte steam) with minimal overhead on the target. This series of articles goes through some issues that I found interesting when developing the project. You can find the current version of the tool on GitHub

In my previous two posts on metal.serial we discussed line information and debug symbols. Thanks to the awesome python pcpp preprocessor implementation by Niall Douglas, we can make this approach more powerful.

Here’s the basic idea: we use line-information combined with the debug data to figure out where in the code we are. Then we use pcpp to obtain more information about this location in the code.

PCPP

import pcpp
import sys
import os

proc = pcpp.Preprocessor()

proc.parse("""#define TEST_MACRO() "Hello world!"
const char * msg = TEST_MACRO();""", 'test.c')

# the preprocessor expects an output stream
res = io.StringIO()
proc.write(res)

assert res.getvalue() == """
const char * msg = "Hello world!";
"""

pcpp evaluates lazily, i.e. only once you call write or a similar function. This approach is useful if you want to write a CLI and generate preprocessed code, but it doesn’t really do the trick for my idea.

What we want to do is to capture the macro expansion itself, which we can do by overloading the macro_expand_args function:

class Preprocessor(pcpp.Preprocessor):
    def __init__(self):
        super().__init__()
    
    def macro_expand_args(self, macro: pcpp.preprocessor.Macro, args):
        if macro.name == "TEST_MACRO":
            #args are tokens, so we transform them to a list of strings
            args_str = ",".join([''.join(tk.value for tk in arg) for arg in args])
            print(self.source + '(' + str(self.linemacro) + '): ' + str(args_str))
        return super().macro_expand_args(macro, args) # call the actual function

This way we get the arguments printed when the preprocessor expands a TEST_MACRO.

proc = Preprocessor()    

proc.parse("""
#define TEST_MACRO(...)

TEST_MACRO(42)
TEST_MACRO(Hello, World!)

""", "test_thingy.c")

proc.write(open(os.devnull, 'w')) # for the evaluation

The snippet above will print:

test_thingy.c(4): 42
test_thingy.c(5): Hello, World!

Line Information, Debug Symbols and metal.serial

Now we can expand the macro and print out arguments that the preprocessors sees. We want to combine that with runtime information.

There is a lot of boilerplate code required (such as macro registries), which I would like to spare you from seeing. metal.serial, which is a part of metal.test, takes care that boilerplate for us.

You can define your on macros by implementing a MacroHook with the following interface:

class MacroHook:
    identifier: str

    def invoke(self, engine, macro_expansion):
		pass

In our case the identifier is TEST_MACRO. We would need to add it to calls of generate and interpret when using metal.serial. The invoke function will then be called once you write a code location where this macro is used.

Let’s turn this example into something useful. We want to write a test macro that contains a message and a condition. The message is known at compile time. so we’ll extract it through pcpp. This way we do not need to pass the information across the wire.

Here’s the macro:

#define TEST_MACRO(Condition, Message) METAL_SERIAL_WRITE_LOCATION(); METAL_SERIAL_WRITE_INT((int)Condition);

And the corresponding metal.serial hook:

import metal.serial

class TestMacroHook(metal.serial.MacroHook):
    identifier: str

    def invoke(self, engine: 'Engine', macro_expansion: 'MacroExpansion'):
        condition = engine.read_int() != 0 # get the condition
        res = 'Success' if condition else 'Failure'
        message = macro_expansion.args[1] # Whatever you wrote as message
        file = macro_expansion.file
        line = macro_expansion.line

        print(file + '(' + str(line) + ') ' + res + ': ' + message)

Combined with a bit of metal.serial boilerplate, we now can write a test:

int main(int argc, const char *argv[]) 
{
    METAL_SERIAL_INIT();
    
    TEST_MACRO(1, "This shall pass");
    TEST_MACRO(1 < 42, "This too shall pass");
    TEST_MACRO(0, "This shall fail");

    METAL_SERIAL_EXIT();
    return 0;
}

When interpreted, this program will produce the following output. Keep in mind that the source file, line number, and output strings are supplied by PCPP and not stored in the binary.

test_thingy.c(20) Success: "This shall pass"
test_thingy.c(21) Success: "This too shall pass"
test_thingy.c(22) Success: "This shall fail"

This is a rudimentary demonstration, but as you can see, we can do powerful things by combining line-data, debug information and the preprocessor.

metal.test comes with its own unit-testing library, but it is as easy as described above to write your own extensions to support another library.