Featured

What to Do When You're Stuck

Projects don't always go well. We encounter many road bumps and errors along the way. There are always one or two major problems that suddenly appear and threaten to sink the ship. These problems can seem to drag on forever. We check our work, try every possible thing we can think of, and then resort to trying random things after that. No matter what, the problem won't yield. But you know a solution is out there, it always is, so you keep searching, even while a sense of dread builds up inside.

When we are in this kind of situation, it can feel hopeless. We can't provide a schedule to a client’s or manager’s looming or already past deadline. We feel the weight of the project resting upon our shoulders. Benevolent people want to ask how it's going and offer encouragement, which only emphasizes the importance of our work and the impact the delay is having.

I know this first-hand. At the time of writing this article, I have been stuck on the same problem for over a week, with probably 60 hours spent actively investigating it. We must resolve the problem somehow, or else the project will be a complete failure for us and for our clients. We also don't have unlimited time to solve the problem. Eventually, the clock will run out and the project will be terminated.

I started thinking about how I generally overcome this type of problem, and I noted these strategies:

  1. Detachment
  2. Take Care of Yourself
  3. Review Your Debugging Process
  4. Inject Novel Ideas
  5. Throw it Out and Start Over

Detachment

Before attempting anything else, we need to detach ourselves from the problem. Step away. We can grant ourselves a break and a reset. We want to get to a place where we can take a new perspective on our problem. This step is critical, and the longer you've been wrestling with the problem the harder it will be

It doesn't matter how you acquire a measure of attachment. There are many approaches, and you must find something that works for you:

  • Go for a walk
  • Exercise
  • Listen to music
  • Take a nap
  • Call someone you enjoy talking to
  • Cook something special
  • Slowly make a pot of tea
  • Play a game

This step seems small, but it is crucial. . I quoted the stoic philosopher Seneca in our article, How I Schedule My Day as a Consultant, and it is worth repeating here:

The mind must be given relaxation - it will rise improved and sharper after a good break. Just as rich fields must not be forced - for they will quickly lose their fertility if never given a break - so constant work on the anvil will fracture the force of the mind. But it regains its powers if it is set free and relaxed for a while. Constant work gives rise to a certain kind of dullness and feebleness in the rational soul.

We need to give our brain a rest. We've become emotionally invested in the problem, and our emotions (especially frustration) are clouding our thinking. We want to let go of our feelings about the problem. We need to let go of what we've already tried to do. Those attempts didn't work. We can't keep attacking the problem from the same point if we want to move past it. We're stuck.

Our goal in achieving detachment from the problem is to re-enable clear thinking and to make our minds receptive to new ideas.

Sometimes all we need for a breakthrough is fresh air and separation from the problem. Even if we don't find a solution during a break, a relaxed and detached perspective is necessary for clearing up our thinking and making progress on the problem. Former US Navy Seal Jocko Willink puts it succinctly:

Take Care of Yourself

With persistent and difficult problems, I've noticed a tendency to fall into a trap: I stop taking care of myself.

We can't forget to rest. Tricky problems lead to long working hours, poor eating habits, over-caffeination, and sitting too long. These actions will cloud our minds, slow our thoughts, and make us less effective.

Take frequent breaks. Don't work too late. Go for a walk every two hours. Don't skip meals. Make sure you are eating healthy food at your preferred mealtimes. Make sure you are staying well hydrated. Don't drink extra coffee. Get plenty of rest at night (extra, ideally, because your brain is working hard).

Taking care of yourself is crucial for maintaining detachment and clear thinking. It’s true that the problem is weighing on you, and that the team is waiting on you, which are both hard to deal with. But if you don't take the time to take care of yourself, you are reducing your effectiveness and prolonging the path to resolution, thus hurting the entire team.

Review Your Debugging Process

With tricky problems, we can easily fall into a trap of random debugging. We become frustrated and start running all sorts of random experiments, often bundling multiple changes into a single test run. Eventually, we end up feeling like we're repeating the same tests, but we're not exactly sure anymore.

When we find ourselves in this situation, we need to step back and revert to a proper debugging process. We should be working methodically, in small steps, testing hypotheses, and noting down the details and results of each experiment. Attempt one thing at a time, observe the outcome, and write both down in a debugging log. The goal is to find the critical variables and to rule out those areas of investigation that have no impact on the problem.

If you don't have a well-established debugging process you can revert to, I recommend reviewing Debugging: 9 Indispensable Rules. Here are the results outlined in that book:

  1. Understand the System
  2. Make it Fail
  3. Quit Thinking and Look
  4. Divide and Conquer
  5. Change One Thing at a Time
  6. Keep an Audit Trail
  7. Check the Plug
  8. Get a Fresh View
  9. If You Didn't Fix it, It Ain't Fixed

Inject Novel Ideas

We're working on a persistent problem, and we've already exhausted all of the ideas we had for solving it. We've probably even tried random things to see if anything would make the problem budge. If you're reading this article in the middle of such a problem, you might even think there are no ideas left for you to try.

This feeling is why it's crucial to detach, make sure we're taking proper care of ourselves, and re-establish a debugging process. Once we've restored our ability to think clearly and methodically, we need to infuse our mind with new ideas.

One place to turn for novel ideas is another human, ideally one who is detached from the situation. Talk through the problem with people who might be familiar with what you are working on, as well as people who are completely outside of your field. In the best case, we might receive extremely helpful advice. Otherwise, simply having to explain the problem will often provide us with new ideas or point out holes in our understanding. People notice when we're glossing over details, providing hand-wavy answers, or not able to answer questions. These are all areas we can explore further in our debugging efforts.

Sometimes another human isn't available. In such situations, the common advice is to explain your problem to a rubber duck (or other inanimate object). The act of translating our thoughts into words will provide many of the benefits talking to another human can provide.

Even after talking to our trusty companions, we may find that we're still out of ideas. In such cases, I like to let my subconscious work for me through a method I call "consulting the oracle". First, I state a clear question or problem statement either in writing or in my mind (not a yes/no question). One the question is asked, I apply a random input and receive an answer. The answer often prompts my subconscious mind into thinking up something related to both the problem and answer. My goal is to take the problem in a new direction: the solution is probably going to be in a place I haven't already looked.

My two favorite oracular works to use are the I Ching and Brian Eno's Oblique Strategies.

The I Ching is an ancient Chinese oracular text. You throw some coins and generate a hexagram, and the book provides an image, a judgment, and commentary on the symbol. The I Ching is often consulted when making decisions or trying to explore one's internal state, but it also can provide effective approaches and triggers for solving problems.

Oblique Strategies is a set of cards created by Brian Eno in the 70s. The cards offer guidelines and constraints to help artists (primarily musicians) work past creative blocks. Since the cards are designed for artists, not all of the strategies apply to a technical situation. Some, however, are powerful triggers that may cause you to reframe the problem completely. Using this deck, we can take two approaches. First is the strict approach: draw a card and follow it exactly. The second approach is to pull cards until we find one that resonates with us - our brains will give us a feeling of "hey, maybe that will work!" when we find a card that resonates. Follow that trail!

If you doubt the efficacy of consulting an oracle, I simply ask you to view this strategy from another perspective. The point is to provide a fresh take on our questions and to let our own minds provide the answers from our subconscious associations. We just need new input for the brain to generate new output. That input doesn't have to be reasonable, be related to the problem, or to mean anything in particular. We are simply trying to jiggle our brains so they can change state and generate a new perspective.

Throw it Out and Start Over

If you remain stuck, sometimes the best decision is to throw out our previous attempt(s) to solve the problem and start over. This can be a painful and counter-intuitive process, especially with a looming schedule deadline. But we know a lot more than we did during the initial implementation. We can implement a second-pass solution more quickly, or at least in a better fashion than the first attempt.

My main concern with a problem that drags on too long is that the problem isn't actually what (or where) we think it is. What if the problem is related to a small detail that is only tangentially related, or a detail that we have ignored completely in our investigations? We can scrap the idea and re-implement the same approach, or we can free ourselves to attempt an alternative approach. Either way we will overcome the problem or confirm that the problem is as real as we think it is.

Still need help?

Sometimes, we keep our head about us and follow all of these guidelines, but we can't get past the obstacle.

If you really can't figure it out, give us a call - we're detached from your problem and can bring in a new perspective. Plus, we love debugging and helping teams move past tricky and persistent problems.

Further Reading

Related Articles

Embedded Systems Architecture Resources

Updated: 20190717

After a decade spent building and shipping hardware products, I became convinced that many of the problems and schedule delays I experienced could have been avoided with a little bit of planning and thought. Repeatedly, we painted ourselves into corners with code that seemed to work well initially but caused problems months later when we finally started end-to-end system testing. Serious problems resulted in major software rewrites, changes in the technology stack, and delayed the ship date. Even worse, as I migrated from one product team to another, I noticed that we were repeating the same basic mistakes.

I started pondering this situation. Why were we dealing major design problems and risk areas at the end of the project instead of the beginning? How could we describe ourselves as "agile" if we weren't able to quickly adapt our programs to change? Why did none of the teams I was on use anything resembling design activity before starting to build a system?

These questions led me to a deep immersion in the topics of software architecture, systems thinking, and system design. I've applied countless lessons to our internal projects, client projects, and business development efforts. Value exploration, visual modeling, and minimalistic architecture efforts have significantly improved our work quality and derisked many projects.

"Architecture" and "design" seem to be words that send programming teams running for the hills. However, I've had multiple embedded developers share their frustrations with me - the same that started me on my journey - and expressed their interest in learning more about software architecture but not knowing where to start. So, here are all the resources I've collected on software architecture. I hope they help guide you in your own journey.

Table of Contents:

Where to Start?

There's a lot of material here! You don't need to read all of it to get started with architecture.

For general architecture exposure, I recommend picking 1-2 books from this list:

If you are focused on embedded systems, I highly recommend Real-Time Software Design for Embedded Systems. This book provides a blueprint for modeling and architecting embedded systems. You will be introduced to UML and a variety of modeling approaches that you can use when architecting an embedded system.

The next step is to actually practice! There is no need for a long, drawn-out architecture stage. Allocate 2-4 weeks for value exploration and architecture efforts before starting any new project. Perform stakeholder interviews and explore the value you expect the system to provide. Then focus on answering core questions, like:

  • What qualities and behaviors are most important?
  • What requirements do they place on the design?
  • What are the biggest risk areas?
  • How can we reduce risk?
  • What are we unsure about that might change?
  • How can we make sure to support those changes without requiring a system redesign?
  • What parts of the system will we buy, license, outsource, and build in house?

Those questions will inform the architecture effort. Model the system and begin prototyping the riskiest areas. As you develop the system, you will explore and refine the system architecture.

General Software Architecture

Before diving into embedded systems specifics, it is helpful to have a solid foundation in general software architecture techniques.

We've broken down our reading recommendations into the following categories:

What is Architecture?

Before diving into the how of architecture, it's helpful to know what it is.

Why Should We Architect?

Perhaps you're not convinced that architecture is valuable. Or perhaps you need to prepare yourself to advocate for architecture efforts on your projects. These articles will give you some insights into why we architect.

The Architect Role

These articles discuss the architect role itself, particularly the qualities and skillsets that are valuable to an architect.

Architecting

We recommend the following architecture books:

These articles from around the web provide countless insights into the practice of software architecture:

Phil Koopman has a selection of lectures which are generally applicable to architecture and design:

Additionally, the slides and course notes from Hassan Gomaa are a useful introduction:

Here are talks which relate to the subject of architecture:

Techniques

Here are some practical technique guides related to the architecture process, ideation, brainstorming, and value exploration.

Documentation

Architecture work and documentation go hand in hand. Here are valuable resources on the that discuss architecture documentation:

Visual Architecture Process

These guides relate to Bredemeyer Consulting's Visual Architecture Process. They provide a practical blueprint for architecting your systems.

C4 Process

Simon Brown created the C4 architecture model, which focuses on four areas of architecture: Context, containers, components, and code. This is another practical blueprint for architecting your system.

Embedded Systems Architecture

Even just a little exposure to software architecture will reveal how deep the rabbit hole goes. We're focused on embedded systems, so here are embedded-specific resources.

Our favorite books on the subject of embedded systems architecture are:

Hassan Gomaa, a professor at George Mason University, published course notes for two courses which discuss embedded systems architecture and modeling:

Phil Koopman published the following course notes which are useful for embedded systems architects:

Safety and Critical Systems

Here are lectures, coures notes, and essays related to architecting for safety and for critical systems:

Security

Here are lectures, coures notes, and essays related to architecting for security:

Systems Thinking

I would be remiss to talk about architecture without mentioning systems thinking. These two topics are intertwined: we must develop a habit of thinking about the system as a whole if we are to work at an architectural level.

Here are some of my favorite books and essays on systems thinking:

Design Patterns

Design patterns are extremely useful to learn and familiarize yourself with. These are non-obvious solutions to common scenarios and problems. For generally useful software architecture patterns, see:

Embedded systems often work well with event-driven architectures and/or state machines. For more information, see:

Embedded systems are often under tight memory constraints. A useful reference for embedded developers is:

Layered or Hexagonal architectures are common abstractions that work well for embedded systems. Here are some links on both types of design:

Here are design patterns related to safety and critical systems:

Here are anti-patterns to avoid:

Visual Modeling

UML

UML is frequently trashed by development teams (even those with no experience using it), but I find "UML-light" to be extremely useful for documenting and modeling my systems.

These books are wonderful resources for learning and applying UML:

Here are lectures related to UML:

As far as UML tools go, there are many options. We recommend three:

  • Visual Paradigm is our tool of choice due to its support of SysML and the ability to tweak the models to support our needs
  • StarUML is a UML modeling tool recommended to us by Grady Booch, who says he uses this tool on a regular basis
  • PlantUML is a great tool which generates UML diagrams from textual descriptions, enabling you to store UML diagrams under revision control and to include them in source-code comments

C4

If you prefer the C4 model, we recommend the following:

Who to Follow

You've already seen these names quite a bit throughout the article. I recommend keeping up with these folks:

Architecture on Embedded Artistry

We publish articles related to Architecture and Systems Thinking on on this website.

Architecture Articles

Systems Thinking Articles

Books Mentioned Above

Documenting Software Architectures: Views and Beyond (2nd Edition)
By Paul Clements, Felix Bachmann, Len Bass, David Garlan, James Ivers, Reed Little, Paulo Merson, Robert Nord, Judith Stafford
Design Patterns: Elements of Reusable Object-Oriented Software
By Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides
Pattern-Oriented Software Architecture Volume 1: A System of Patterns
By Frank Buschmann, Regine Meunier, Hans Rohnert, Peter Sommerlad, Michael Stal, Michael Stal

Change Log

Embedded Systems Security Resources

Security is an area that we all need to focus on when we build our embedded systems. Here's our collection of resources on embedded systems security (and security in general). If you have recommended books or links that discuss embedded systems security, let us know!

Table of Contents:

Books

Some of these books may be slanted toward/against vendors. We recommend getting the security concepts without putting too much stock into the opinion of vendors.

Lectures

Phil Koopman has a collection of lectures discussing embedded systems security:

Articles

Primers

Here are articles with general introductions to embedded systems security concepts:

Pitfalls

These articles focus on common pitfalls with embedded device security:

General Security Articles

The authors of Embedded Systems Security published a series of articles on EDN:

Twitter Accounts

Here are some Twitter accounts to follow. Some accounts have an electronics/hardware bent, but others are more general InfoSec.

Acknowledgements