Lessons from the Field

What to Do When You're Stuck

Projects don't always go well. We encounter many road bumps and errors along the way. There are always one or two major problems that suddenly appear and threaten to sink the ship. These problems can seem to drag on forever. We check our work, try every possible thing we can think of, and then resort to trying random things after that. No matter what, the problem won't yield. But you know a solution is out there, it always is, so you keep searching, even while a sense of dread builds up inside.

When we are in this kind of situation, it can feel hopeless. We can't provide a schedule to a client’s or manager’s looming or already past deadline. We feel the weight of the project resting upon our shoulders. Benevolent people want to ask how it's going and offer encouragement, which only emphasizes the importance of our work and the impact the delay is having.

I know this first-hand. At the time of writing this article, I have been stuck on the same problem for over a week, with probably 60 hours spent actively investigating it. We must resolve the problem somehow, or else the project will be a complete failure for us and for our clients. We also don't have unlimited time to solve the problem. Eventually, the clock will run out and the project will be terminated.

I started thinking about how I generally overcome this type of problem, and I noted these strategies:

  1. Detachment
  2. Take Care of Yourself
  3. Review Your Debugging Process
  4. Inject Novel Ideas
  5. Throw it Out and Start Over

Detachment

Before attempting anything else, we need to detach ourselves from the problem. Step away. We can grant ourselves a break and a reset. We want to get to a place where we can take a new perspective on our problem. This step is critical, and the longer you've been wrestling with the problem the harder it will be

It doesn't matter how you acquire a measure of attachment. There are many approaches, and you must find something that works for you:

  • Go for a walk
  • Exercise
  • Listen to music
  • Take a nap
  • Call someone you enjoy talking to
  • Cook something special
  • Slowly make a pot of tea
  • Play a game

This step seems small, but it is crucial. . I quoted the stoic philosopher Seneca in our article, How I Schedule My Day as a Consultant, and it is worth repeating here:

The mind must be given relaxation - it will rise improved and sharper after a good break. Just as rich fields must not be forced - for they will quickly lose their fertility if never given a break - so constant work on the anvil will fracture the force of the mind. But it regains its powers if it is set free and relaxed for a while. Constant work gives rise to a certain kind of dullness and feebleness in the rational soul.

We need to give our brain a rest. We've become emotionally invested in the problem, and our emotions (especially frustration) are clouding our thinking. We want to let go of our feelings about the problem. We need to let go of what we've already tried to do. Those attempts didn't work. We can't keep attacking the problem from the same point if we want to move past it. We're stuck.

Our goal in achieving detachment from the problem is to re-enable clear thinking and to make our minds receptive to new ideas.

Sometimes all we need for a breakthrough is fresh air and separation from the problem. Even if we don't find a solution during a break, a relaxed and detached perspective is necessary for clearing up our thinking and making progress on the problem. Former US Navy Seal Jocko Willink puts it succinctly:

Take Care of Yourself

With persistent and difficult problems, I've noticed a tendency to fall into a trap: I stop taking care of myself.

We can't forget to rest. Tricky problems lead to long working hours, poor eating habits, over-caffeination, and sitting too long. These actions will cloud our minds, slow our thoughts, and make us less effective.

Take frequent breaks. Don't work too late. Go for a walk every two hours. Don't skip meals. Make sure you are eating healthy food at your preferred mealtimes. Make sure you are staying well hydrated. Don't drink extra coffee. Get plenty of rest at night (extra, ideally, because your brain is working hard).

Taking care of yourself is crucial for maintaining detachment and clear thinking. It’s true that the problem is weighing on you, and that the team is waiting on you, which are both hard to deal with. But if you don't take the time to take care of yourself, you are reducing your effectiveness and prolonging the path to resolution, thus hurting the entire team.

Review Your Debugging Process

With tricky problems, we can easily fall into a trap of random debugging. We become frustrated and start running all sorts of random experiments, often bundling multiple changes into a single test run. Eventually, we end up feeling like we're repeating the same tests, but we're not exactly sure anymore.

When we find ourselves in this situation, we need to step back and revert to a proper debugging process. We should be working methodically, in small steps, testing hypotheses, and noting down the details and results of each experiment. Attempt one thing at a time, observe the outcome, and write both down in a debugging log. The goal is to find the critical variables and to rule out those areas of investigation that have no impact on the problem.

If you don't have a well-established debugging process you can revert to, I recommend reviewing Debugging: 9 Indispensable Rules. Here are the results outlined in that book:

  1. Understand the System
  2. Make it Fail
  3. Quit Thinking and Look
  4. Divide and Conquer
  5. Change One Thing at a Time
  6. Keep an Audit Trail
  7. Check the Plug
  8. Get a Fresh View
  9. If You Didn't Fix it, It Ain't Fixed

Inject Novel Ideas

We're working on a persistent problem, and we've already exhausted all of the ideas we had for solving it. We've probably even tried random things to see if anything would make the problem budge. If you're reading this article in the middle of such a problem, you might even think there are no ideas left for you to try.

This feeling is why it's crucial to detach, make sure we're taking proper care of ourselves, and re-establish a debugging process. Once we've restored our ability to think clearly and methodically, we need to infuse our mind with new ideas.

One place to turn for novel ideas is another human, ideally one who is detached from the situation. Talk through the problem with people who might be familiar with what you are working on, as well as people who are completely outside of your field. In the best case, we might receive extremely helpful advice. Otherwise, simply having to explain the problem will often provide us with new ideas or point out holes in our understanding. People notice when we're glossing over details, providing hand-wavy answers, or not able to answer questions. These are all areas we can explore further in our debugging efforts.

Sometimes another human isn't available. In such situations, the common advice is to explain your problem to a rubber duck (or other inanimate object). The act of translating our thoughts into words will provide many of the benefits talking to another human can provide.

Even after talking to our trusty companions, we may find that we're still out of ideas. In such cases, I like to let my subconscious work for me through a method I call "consulting the oracle". First, I state a clear question or problem statement either in writing or in my mind (not a yes/no question). One the question is asked, I apply a random input and receive an answer. The answer often prompts my subconscious mind into thinking up something related to both the problem and answer. My goal is to take the problem in a new direction: the solution is probably going to be in a place I haven't already looked.

My two favorite oracular works to use are the I Ching and Brian Eno's Oblique Strategies.

The I Ching is an ancient Chinese oracular text. You throw some coins and generate a hexagram, and the book provides an image, a judgment, and commentary on the symbol. The I Ching is often consulted when making decisions or trying to explore one's internal state, but it also can provide effective approaches and triggers for solving problems.

Oblique Strategies is a set of cards created by Brian Eno in the 70s. The cards offer guidelines and constraints to help artists (primarily musicians) work past creative blocks. Since the cards are designed for artists, not all of the strategies apply to a technical situation. Some, however, are powerful triggers that may cause you to reframe the problem completely. Using this deck, we can take two approaches. First is the strict approach: draw a card and follow it exactly. The second approach is to pull cards until we find one that resonates with us - our brains will give us a feeling of "hey, maybe that will work!" when we find a card that resonates. Follow that trail!

If you doubt the efficacy of consulting an oracle, I simply ask you to view this strategy from another perspective. The point is to provide a fresh take on our questions and to let our own minds provide the answers from our subconscious associations. We just need new input for the brain to generate new output. That input doesn't have to be reasonable, be related to the problem, or to mean anything in particular. We are simply trying to jiggle our brains so they can change state and generate a new perspective.

Throw it Out and Start Over

If you remain stuck, sometimes the best decision is to throw out our previous attempt(s) to solve the problem and start over. This can be a painful and counter-intuitive process, especially with a looming schedule deadline. But we know a lot more than we did during the initial implementation. We can implement a second-pass solution more quickly, or at least in a better fashion than the first attempt.

My main concern with a problem that drags on too long is that the problem isn't actually what (or where) we think it is. What if the problem is related to a small detail that is only tangentially related, or a detail that we have ignored completely in our investigations? We can scrap the idea and re-implement the same approach, or we can free ourselves to attempt an alternative approach. Either way we will overcome the problem or confirm that the problem is as real as we think it is.

Still need help?

Sometimes, we keep our head about us and follow all of these guidelines, but we can't get past the obstacle.

If you really can't figure it out, give us a call - we're detached from your problem and can bring in a new perspective. Plus, we love debugging and helping teams move past tricky and persistent problems.

Further Reading

Related Articles

Practical Decoupling Techniques Applied to a C-based Radio Driver

We advocate for spending time decoupling your firmware from the underlying hardware to enable portability and reusability. We often have developers tell us that they see the value in these outcomes, but they could use practical examples for how to effectively decouple firmware from the underlying hardware. Others have concerns about whether we can truly create portable APIs.

While preparing for a new client project, we discovered an open-source driver for the ON Semiconductor AX5043 radio IC. This driver serves as an excellent example of simple techniques you can use to achieve decoupling.

The example driver's author built the hardware interactions on top of a single interface abstraction (representing a SPI transfer), enabling you to quickly migrate the driver from one platform to another. The API design also enables you to support multiple radios on a single device. We'll take a look at these two techniques in greater detail.

Table of Contents:

  1. Framing the Problem
  2. Ambitious Abstraction
  3. Thinking About Responsibility
  4. Decoupling with Minimalistic APIs
    1. Handling Configuration
  5. Supporting Multiple Devices
  6. Putting it All Together
  7. Further Reading

Framing the Problem

The AX5043 supports two communication methods: SPI and "wire mode". This library focuses only on the SPI interface, which is the standard operating mode.

The typical implementation approach (taken on most projects we've encountered) would be to define a set of radio interfaces which talk directly to AX5043 device using existing SPI APIs. The SPI APIs may be provided by a processor vendor SDK, an SDK such as MyNewt or Zephyr, or a proprietary internal implementation. If the bus isn't shared with other devices, we would likely find that SPI information is hard-coded in the driver. We would also probably find that the driver is responsible for configuring its SPI peripheral with the settings it needs.

The approach described above would work just fine until one of these scenarios forces a change:

  1. Migrate to a new SDK (due to a processor change or underlying framework change)
  2. Support multiple hardware revisions which have different SPI/driver configurations
  3. Support multiple devices on a single SPI bus
  4. Support multiple driver instances on a single board

At this point the driver is ripped apart in order to support a new SDK or to enable flexibility through extra function parameters. The status quo often serves a team until the next major requirements change occurs, and another rewrite happens.

Ambitious Abstraction

More ambitious teams take a heavier approach and define generic APIs for each driver type. Modules which need to interact with a given driver type only know about and use the generic APIs. With C++, this type of abstraction is achieved with inheritance + virtual functions or templates + concepts. Here's an inheritance-based example:

// SPI Master Abstract Class
class SPIMaster
{
  public:
    virtual void configure(spi::baud_t baud) noexcept = 0;
    virtual spi::mode mode() const noexcept = 0;
    virtual spi::mode mode(spi::mode mode) noexcept = 0;
    virtual void start() noexcept = 0;
    virtual void stop() noexcept = 0;
    virtual comm::status transfer(void* data, size_t length) noexcept = 0;
};

// A derived class, which is a SPI peripheral driver that 
// implements the abstract class interfaces
class SomeSPIDriver : public SPIMaster
{
    // Implementation of each abstract function 
    // + possibly some extras that won't be in the generic interface
};

// Driver which talks over SPI
class AX5403 {
  public:
    // When we construct an AX5403 object, we take in a reference
    // to the ***generic*** SPI Interface
    AX5403(SPIMaster& spi) : spi_(spi) {}

    // Function implementations would talk using SPIMaster interfaces, 
    // and could work with any SPI driver derived from that base class
    void send(void* data, size_t size)
    {
        spi_.transfer(data, size);
    }

  private:
    SPIMaster& spi_;
};

For C, it is achieved using structures of function pointers:

struct spi_intf
{
    void (*configure)(spi_baud_t);
    spi_mode_t (*get_mode)();
    void (*set_mode)(spi_mode_t);
    void (*start)();
    void (*stop)();
    comm_status_t (*transfer)(void*, size_t);
};

struct spi_intf spi0 = {
    &configureSPI0,
    &getModeSPI0,
    &setModeSPI0,
    &startSPI0,
    &stopSPI0,
    &transferSPI0
};

void ax5403_send(struct spi_intf* spi, uint8_t* data, size_t size)
{
    assert(spi && spi->transfer);

    // Uses a generic function pointer, so different implementations 
    // can be supplied with no change to the driver
    spi->transfer(data, size);
}

This C-style approach operates on the same principal: consumers only know about the function pointers, not the details of the functions being pointed to.

Thinking About Responsibility

We should take a step back and think about responsibility. What are the core responsibilities of an AX5043 radio driver?

  • Communication with the radio
  • Initialization of the radio
  • Adjusting radio settings
  • Sending and receiving data over the radio

Notice that our driver doesn't need to be responsible for initializing the SPI bus, changing SPI operating modes, setting bit ordering, or starting/stopping the SPI driver. All our radio driver needs to know about SPI is how to transfer data over the bus it's connected to. It would be reasonable for the driver documentation to say: "We need the SPI bus to be initialized and operating in Mode 3 and transmitting LSB First", leaving it to the application programmer to ensure these requirements are met.

I bring this up because I'm commonly guilty of the "ambitious abstraction approach" shown above. The strategy works well, but it presents some problems.

For one, we've given the radio driver more control over the SPI driver than it actually needs. It's easy to misinterpret the availability of an API as it being appropriate to use in the current context. If our drivers are reaching beyond their responsibilities and configuring the SPI bus (or stopping/starting the SPI peripheral), we can easily introduce problems into our system - especially if drivers start changing configurations without the application developer knowing his settings are being overridden.

The second problem is that we haven't really decoupled anything. Our radio driver, for better or worse, is dependent upon the SPI driver abstract interface definition. We are not omniscient nor omnipotent, and eventually we will need to modify our abstract interfaces. The unfortunate impact is that every module which is dependent upon those abstractions must change. For something as fundamental as a SPI driver, the impact can be significant.

If we stay focused on the responsibilities of our driver, we can come up with a minimalistic decoupling strategy that avoids these problems.

Decoupling with Minimalistic APIs

Now that we've explored the problem space a bit more, let's get back to this wonderful example of a decoupled driver.

As we mentioned above, the AX5043 radio driver only needs to know how to send and receive data over the SPI bus it's connected to. Since SPI send and receives data at the same time, we can use a single function to handle a SPI transfer. In C, we can define our abstract interface using a function pointer:

void (*spi_transfer)(unsigned char*, uint8_t);

The driver devices this function pointer inside of the ax_config struct as a member variable. We'll look more at the configuration struct next.

The driver defines many hardware-interaction APIs for internal use, such as:

uint8_t ax_hw_read_register_8(ax_config* config, uint16_t reg);

uint16_t ax_hw_read_register_long_bytes(ax_config* config, uint16_t reg,
                                        uint8_t* ptr, uint8_t bytes);

uint16_t ax_hw_write_fifo(ax_config* config, uint8_t* buffer, 
                             uint16_t length);
uint16_t ax_hw_read_fifo(ax_config* config, uint8_t* buffer, 
                             uint16_t length);

Each of these functions references the abstract spi_transfer function pointer in the config struct to talk to hardware:

uint16_t ax_hw_read_fifo(ax_config* config, uint8_t* buffer, 
                             uint16_t length)
{
  /* read (short access) */
  buffer[0] = (AX_REG_FIFODATA & 0x7F);

  config->spi_transfer(buffer, length);

  status &= 0xFF;
  status |= ((uint16_t)buffer[0] << 8);

  return status;
}

Internally, the radio driver knows nothing about the details of the SPI driver; it only knows that this one function pointer exists.

The next detail to handle: how do we supply the instance of the spi_transfer function to the radio driver?

Handling Configuration

Somewhere else in the program, we need to initialize the SPI device:

if (wiringPiSPISetup(SPI_CHANNEL, SPI_SPEED) < 0) {
    fprintf(stderr, "Failed to open SPI port. "
                     "Try loading spi library with 'gpio load spi'");
}

We'll also declare and populate the radio configuration struct. The contents of this structure are referenced by the various radio APIs. One of the fields that must be populated is the spi_transfer function pointer.

ax_config config;
memset(&config, 0, sizeof(ax_config));

config.clock_source = AX_CLOCK_SOURCE_TCXO;
config.f_xtal = 16369000;
config.synthesiser.A.frequency = 434600000;
config.synthesiser.B.frequency = 434600000;

config.spi_transfer = wiringpi_spi_transfer;

config.pkt_store_flags = AX_PKT_STORE_RSSI | AX_PKT_STORE_RF_OFFSET;

Where does the wiringpi_spi_transfer function come from? It's actually a local function which matches the spi_transfer API and makes the proper call to the underlying SPI driver:

void wiringpi_spi_transfer(unsigned char* data, uint8_t length)
{
    wiringPiSPIDataRW(SPI_CHANNEL, data, length);
}

This approach, defining a local function which implements the necessary abstraction, will be the most common. It's doubtful that your SPI driver will actually have an interface that matches the radio driver's requirements. You need to create a mapping between the two. The actual coupling details of the two is kept outside of the drivers themselves and in a contained area.

For my designs, it is common for this kind of work to happen in a "board" module, which defines the various drivers, hooks them up to each other, and handles hardware initialization. This approach is an example of the Mediator software architecture pattern, since we've defined a single module which manages the coupling between other interacting modules.

Supporting Multiple Devices

The creation of a configuration struct and its use in all of the AX5043 APIs enables us to support multiple radios at once. This is common for C APIs, as we need some way to store information outside of the function itself.

In order to enable C APIs to be used with multiple objects, we need to:

  • Store all mutable configuration options and context-specific information in the struct
  • Store function pointer abstractions in the struct
  • Pass the configuration struct to the API (often done via pointer to avoid a copy onto the stack)

In this specific instance, we just need to declare two different ax_config instances, populate the settings for each radio, and supply a spi_transfer function pointer for each instance. We would expect the spi_transfer function implementation to differ for the two ax_config instances, as you need to differentiate between the two hardware devices.

#define SPI_CHANNEL_RADIO_0 0
#define SPI_CHANNEL_RADIO_1 3

void radio0_spi_transfer(unsigned char* data, uint8_t length)
{
    wiringPiSPIDataRW(SPI_CHANNEL_RADIO_0, data, length);
}

void radio1_spi_transfer(unsigned char* data, uint8_t length)
{
    wiringPiSPIDataRW(SPI_CHANNEL_RADIO_1, data, length);
}

int main(void)
{
    // Some stuff happens...

    ax_config radio0_config;
    ax_config radio1_config;

    // Configure some stuff...

    radio0_config.spi_transfer = radio0_spi_transfer;
    radio1_config.spi_transfer = radio1_spi_transfer;

    ax_init(&radio0_config);
    ax_init(&radio1_config);

    // Useful stuff happens next...
}

Because the APIs reference the spi_transfer function pointer in the configuration structure, and because each radio has a different configuration, our driver will talk to two different devices with no modifications.

We can see the utility of this approach by analyzing a function in the repository which does not accept an ax_config* input:

/**
 * Returns the status from the last transaction
 */
uint16_t ax_hw_status(void)
{
  return status;
}

This status value is set by various calls, such as:

uint16_t ax_hw_read_fifo(ax_config* config, uint8_t* buffer, uint16_t length)
{
  /* read (short access) */
  buffer[0] = (AX_REG_FIFODATA & 0x7F);

  config->spi_transfer(buffer, length);

  status &= 0xFF;
  status |= ((uint16_t)buffer[0] << 8);

  return status;
}

If we have two radios, we have no guarantee that the status value we're reading actually corresponds to our radio's last transfer. Instead, this API could be adjusted to take in an ax_config* with the status kept in the struct itself.

Putting it All Together

Both of these techniques can be easily applied to your C device drivers and libraries. Take a step back and think about the APIs your drivers actually require from other modules. Are you creating too many dependencies, or exposing too much information? Can you limit that exposure by requiring a minimal HAL, such as this example driver does?

Take a look through the AX project to see how much is implemented on top of one one abstract function. You're not limited to one abstraction, either. Simply define the minimal set that you need for your specific module.

Happy Hacking!

Further Reading

Programmers: Let's Study Source Code Classics

Updated: 20190729

Contemporary programmers are lucky: we live in a world where historical and influential program source code is available for us to review. However, most programmers only learn and study the programs they have worked on themselves. We rarely take the time to study historical works, and programming courses don’t typically spend any time on the subject.

We believe that software developers should review influential source code. This is similar to architects studying influential building designs (and critiques of those designs). Rather than repeating the same mistakes over-and-over, we should study the great works that preceded us and learn from and build upon their lessons.

Ideally, we would study great source code along with a commentary or critique which provides us information about the project’s context, successes, and failures. Such commentaries are rare, but here are a few excellent starting points:

You can also find a program you've used in the past and review the source code. It's important to start with a program you are familiar with, so you can anchor the functional behavior to the source code. Here are resources you can use to find and browse historical source code:

Change Log

  • 20190729:
    • Added additional Apollo 11 guidance computer links
  • 20190627:
    • Added links to Wolfenstein and DOOM source code + reviews