In the previous article, I discussed the importance of spending time doing things outside of your professional field. By doing so, you expose yourself to new ideas, paradigms, patterns, and practices. Breakthroughs often come from taking an idea in one area and applying it to another. To help critical ideas spread across fields, we need to serve as cross pollinators. You never know what new insight will lead to a breakthrough moment in your work.
Today I want to explore an insight I gleaned from growing mushrooms at home: we don’t cut out contamination, we take a sample of the best material and start over.
Mushroom Cultivation
There is a lot to learn about when growing mushrooms: laboratory techniques, sterile culture, tissue transfers, creating the proper environmental conditions, and managing the mushroom lifecycle.
One aspect that I’ve enjoyed is cultivating mycelium on petri dishes with an agar growing medium.
I either take some mushroom spores and inoculate them on an agar plate, or I take a piece of tissue from a mushroom I liked so that I can clone it and propagate that strain. The spores or tissue will grow out, feeding on the nutrients present in the agar medium.
Once the plate is fully colonized by mycelium, I can divide up the contents and use them to inoculate jars of sterilized substrate, which will eventually be used to grow larger and larger batches until there is sufficient quantity for putting into fruiting conditions.
Of course, this doesn’t always go according to plan. When the mycelium grows out on the agar plate, there is the possibility that only a subset of the mycelial growth is ideal for reaching fruiting conditions.
There’s also a chance that contamination (e.g., bacteria or mold) was introduced to the agar plate. This could be from the spores/tissue itself, the environment, my clothes, or the tools I used. We do not want these contaminants added to the sterilized substrate jars, as they will potentially out-compete the mycelium or cause significant problems later in the growing process.
In both of these cases, we can clean up the growth by making a transfer from one agar plate to another. We take the best part of the plate (the tiniest core of the best part, at that) and propagate it forward.
I want to emphasize this once more: we take the best part and propagate that forward. We don’t cut out contamination, no matter how small it looks. Whenever we open the plate to cut out contamination, there is a chance that all we’ve done is introduce new contaminants. There are also potentially other contaminants present on the plate that we haven’t seen yet. Or, in the worst case, we are dealing with a mold that has already released its spores all over the plate. Instead, we will select the best looking part that is farthest away from contamination, cut that out, and put it on a new plate. We will do that as many times as necessary to get a clean sample.
Application to Engineering
This culturing technique is a direct analog to what Dr. Deming and Harold F. Dodge pointed out:
Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product. As Harold F. Dodge said, “You cannot inspect quality into a product.”
Dr. Ackoff echoes this same thought:
Finding and removing defects is not a way to improve the overall quality or performance of a system.
More pointedly, Dr. Ackoff notes that:
An improvement program must be directed at what you want, not at what you don’t want.
In the case of the agar example above, we are not focused on removing contamination or removing the parts of the mycelial growth that we don’t want. Instead, we’re focused on taking what we want and propagating that forward.
In the programming and engineering projects I’ve been involved with over the past decade, we often take the exact opposite approach. We try to squash the bugs and eliminate the errors, both of which are usually identified after the code has been integrated into the product and released to internal testers or to customers. This approach is simply finding and removing what we don’t want. If this approach has been shown to not work in other industries and fields, why do we continue to use it on software projects?
I propose that we instead take a lesson from mushroom cultivation: rather than continually debugging our software, it’s better to save the best 5% and start again from scratch using the core 5% as our starting point.
We can take a similar approach to prototypes and to refactoring/redesign efforts. Let’s say we have a piece of code – a function, driver, library, or module – that does something in a novel, unique, or important way. We can take that notable piece and use it as the starting point for the next revision (a new driver, a refactored module, the production version of the program). Sure, our old work gets tossed, but we are keeping the best parts and ideas for the next iteration. We’ve cut out the baggage that is holding us back and created the space to improve and expand.
Certainly we have a reluctance to throw away our work. After all, we already invested so much time into the initial design or implementation – isn’t it better to make improvements to that design rather than start again? This is just the sunk cost fallacy rearing its head. I know that I have always been surprised at how much better the succeeding iterations are when compared to the first one whenever I have started from scratch. Of course, we have to pay careful attention to our technique as we go along, else we risk introducing contamination (i.e., errors) into succeeding versions.
I must admit that this is not an idea I can take credit for. Once I made these connections, I was reminded of something I read on Jack Ganssle’s blog years ago. Jack (and Barry Boehm) agrees with the idea that it’s better to toss out bad code and rewrite it from scratch:
We all have a visceral feel for another rule of thumb: a little bit of the code causes most of the problems. We’ve all had that nasty bit of code that breaks every time someone changes nothing more than a comment. Fear of Editing is a symptom of this problem. We try to beat the beast into submission but it’s a never-tamed hydra.
5% of the functions consume 80% of debugging time. I’ve observed that most projects wallow in the debug cycle, which often accounts for half of the entire schedule. Clearly, if we can do something about those few functions that represent most of our troubles, the project will get out the door that much sooner.
Barry Boehm observed that these few functions that create so much trouble cost four times as much as any other function. That suggests it’s much cheaper to toss the junk and recode than to reactively remove the never-ending stream of bugs. Perhaps we really blew it when first writing the code, but if we can identify these crummy routines, toss them out, and start over, we’ll save big bucks.
My Rule of Thumb is: when the developers are afraid to change a function, it’s time to rewrite that code from scratch.
This idea is also reflected in the writing process. When I write, I typically start with a huge brain dump. I get everything out of my head and onto paper, with no requirements for coherency or excellence. Then, a day or so later, I review the initial brain dump and look for anything that grabs me. It might be just a single paragraph out of 10 pages of notes that jumps out, but this paragraph becomes the seed for a new article. The rest of that brain dump never sees the light of day.
Want to build quality into your product?
Stay on the lookout for kernels of excellence in your work. Make sure you are propagating them forward into better and better work over time. Move those kernels away from the contamination, the bugs, the crap. Save only the best, and start again if you need to.
Further Reading
- Why Professional Programmers Shouldn’t Have Programming Side Projects
- Inspection is Too late: The Quality, Good or Bad, is Already in the Product
- Beyond Continual Improvement
- Jack’s Rules of Thumb
- Wikipedia: Sunk Cost
That Jack Ganssle quote was exactly what this reminded me of! It also reminded me of your previous thoughts on complexity (https://embeddedartistry.com/blog/2019/02/11/hypotheses-on-systems-and-complexity/), @phillip. Namely, that systems tend towards increasing complexity naturally. A lot of open source projects tend that way, I think, since people see its initial utility and then try to continually add functionality or ports to other systems. I like the idea of a “total rewrite” to occasionally clear out the cruft.