Product Development

Hypotheses on Systems and Complexity

A famous John Gall quote from Systemantics became known as Gall's Law. The law states:

A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.

I've always felt the truth of this idea. Gall's Law inspired me to think about the evolution of complexity in systems from different perspectives. I've developed five hypotheses in this area:

  1. A simple system that works (and is maintained) will inevitably grow into a complex system.
  2. The tendency of the Universal System is a continual increase in complexity.
  3. A simple system must increase in complexity or it is doomed to obsoletion and/or failure.
  4. A system's complexity level starts at the complexity of the local system/environment in which it participates.
  5. A working system will eventually collapse due to unmanageable complexity.

I call these ideas "hypotheses" because they are born of late-night thoughts while watching my newborn child. They have not been put through sufficient research or testing for me to call them "axioms", "laws", or "rules of thumb". These ideas may already exist in the systems cannon, but I have not yet encountered them.

The Hypotheses in Detail

Let's look at each of these hypotheses in turn, then we can discuss their implications for our projects.

Hypothesis 1: Simple Systems Become Complex

My first hypothesis is fully stated as follows:

A simple system that works (and is maintained) will inevitably grow into a complex system.

This is a restatement of Gall's Law from a different perspective. I believe that a working simple system is destined to become more complex.

This hypothesis is opposed to another systems maxim (quoted from Of Men and Laws):

A working system (and by happy accident, systems sometimes work) should be left alone.

Unfortunately, this recommendation is untenable for two reasons:

  1. Human beings are not disciplined enough to leave a working system alone.
  2. If a working system is not maintained, it will inevitably become obsolete according to Hypothesis 3.

Humans are the ultimate tinkerers. We are never satisfied with the status quo. We have the tendency to expand or modify a system's features and behaviors once we consider it to be "working" (and even if it's not working). Our working systems are destined to increase in complexity thanks to our endless hunger.

Hypothesis 2: Universal complexity is always increasing

My second hypothesis is fully stated as follows:

The tendency of the Universal System is a continual increase in complexity.

At its core, I believe that Hypothesis 2 is simply a restatement of the Second Law of Thermodynamics, but I include it for use with other hypotheses below.

The Second Law of Thermodynamics states that the total entropy of an isolated system can never decrease over time. Thanks to the Second Law of Thermodynamics, all processes in the universe trigger an irreversible increase in the total entropy of a system and its surroundings.

Rudolf Clausius provides us with another perspective on the Second Law of Thermodynamics:

[...] we may express in the following manner the fundamental laws of the universe which correspond to the two fundamental theorems of the mechanical theory of heat.

  1. The energy of the universe is constant.
  2. The entropy of the universe tends to a maximum.

I have an inkling that complexity and entropy are closely related concepts, if not actually the same. As such, I assume that the complexity of the Universal System will increase over time.

The reason that I think complexity increases over time is that I can observe this hypothesis in other sciences and directly in the world around me:

  • After the big bang, simple hydrogen coalesced into stars (and planets and solar systems and galaxies), forming increasingly complex elements as time progressed
  • Life progressed from simple single-celled organisms to complex networked species consisting of hundreds of sub-systems
  • Giving birth progressed from a natural, body-driven affair to one of complex rituals that is carried out by a large team of experts at great cost in specialized locations (i.e., hospitals)
  • Finance has progressed from exchanging metal coins and shells to a complex, automated, digitized, international system of rules and cooperating systems

Corollary: Complexity must be preserved

The idea exists that complexity can be reduced:

An evolving system increases its complexity unless work is done to reduce it.
-- Meir Lehman

Or:

Ongoing development is the main source of program growth, but programs are also entropic. As they age, they tend to become more cluttered. They get larger and more complicated unless pressure is applied to make them simpler.
-- Jerry Fitzpatrick

Because of the Second Law of Thermodynamics, we cannot reverse complexity. We are stuck with the existing environment, requirements, behaviors, expectations, customers, resources, etc.

Energy must be invested to perform any "simplification" work, which means that there is a complexity-entropy increase in some part of the system. Perhaps you successfully "simplified" your product's hardware design so that it's easier to assemble in the factory. What other sub-systems saw increased complexity as a result: supply chain, tooling design, engineering effort, mechanical design, repairability?

Complexity must be preserved - we only move it around within the system.

Hypothesis 3: Simple Systems Must Evolve

Hypotheses 1 and 2 combine into a third hypothesis:

A simple system must increase in complexity or it is doomed to obsoletion and/or failure.

The systems we create are not isolated; they are always interconnected with other systems. And as one of John Gall's "Fundamental Postulates of General Systemantics" states, "Everything is part of a larger system."

The Universal System is always increasing in complexity-entropy, as are all subsystems by extension. Because of the ceaseless march toward increased complexity, systems are forced to adapt to changes in the complexity of the surrounding systems and environment. Any system which does not evolve will eventually be unable to cope with the new level of complexity and will implode.

The idea of "code rot" demonstrates this idea:

Software rot, also known as code rot, bit rot, software erosion, software decay or software entropy is either a slow deterioration of software performance over time or its diminishing responsiveness that will eventually lead to software becoming faulty, unusable, or otherwise called "legacy" and in need of upgrade. This is not a physical phenomenon: the software does not actually decay, but rather suffers from a lack of being responsive and updated with respect to the changing environment in which it resides.

I've seen it happen enough on my own personal projects. You can take a working software project without errors, put it into storage, pull it out years later, and it will no longer compile and run. This could be for any number of reasons: the language changed, the compiler is no longer available, libraries or tooling needed to build and use the software is no longer available, the underlying processor architectures have changed, etc.

Our "simple" systems will never truly remain so. They must be continually updated to remain relevant.

Hypothesis 4: "Simple" is Determined by Local Complexity

Hypothesis 2 drives the fourth hypothesis:

A system's complexity level starts at the complexity of the local system/environment in which it participates.

Stated in another way:

A system cannot have lower complexity than the local system in which it will participate.

Hypothesis 2 indicates that a local (and universal) lower bound for simplicity exists. Stated another way, your system has to play by the rules of other systems it interacts with. The more external systems your system must interact with, the more complex the starting point.

We can see this by looking at the world around us. Consider an example of payment processing. You can't start over with a "simple" payment application: the global system is to complex and has too many specific requirements. There are banking regulations, credit card regulations, security protocols, communication protocols, authentication protocols, etc. Your payment processor must work with the existing banking ecosystem.

Now, you could ignore these requirements and create a new payment system altogether (e.g., Bitcoin), but you are not actually participating in the same local system (international banking). Even still, the Universal System's complexity is higher than your system's local complexity, and players know the game. You can skip the authentication requirements or other onerous burdens, but external actors can still take advantage of your system (e.g., Bitcoin thefts, price manipulation, lost keys leading to un-claimable money).

Once complexity has developed, we are stuck with it. We can never return to simplicity. I can imagine a time when the Universal System's complexity level will be so high that humans will no longer have the capacity to create or manage any systems.

Hypothesis 5: Working Systems Eventually Collapse

Hypothesis 5 is fully stated as follows:

A working system will eventually collapse due to unmanageable complexity.

Complexity is always increasing, and there is nothing we can do to stop it. There are two complexity-related failure modes for our system:

  1. Our system becomes so complex that we can no longer maintain it (there are no humans who can understand and master the system)
  2. Our system cannot adapt fast enough to keep up with the local/universal system's increases in complexity

While we cannot forever prevent the collapse of our system, we can impact the timeframe through system design and complexity management efforts. We can strive to reduce the rate of complexity increase to a minimal amount. However, as the complexity of the system increases, the effort required to sustain the system also increases. As time goes on, our systems require more energy to be spent on documentation, hiring, training, refactoring, and maintenance.

We can see systems all around us which become too complex to truly understand (e.g., the stock market). Unfortunately, Western governments seem to be reaching a complexity breaking point, as they have become so complex they can't enact policy. To quote Matt Levine's Money Stuff newsletter:

What if your model is that democratic political governance has just stopped working—not because you disagree with the particular policies that particular elected governments are carrying out, but because you have started to notice that elected governments in large developed nations are increasingly unable to carry out any policies at all?

Perhaps unmanageable complexity doomed the collapsed civilizations that preceded us. Given that thought, what is the human race's limit on complexity management? We've certainly extended our ability to handle complexity through the development of computers and algorithms, but there will come a time when the complexity is too much for us to handle.

Harnessing these ideas

These five hypotheses are one master hypothesis broken into different facets which we can analyze. The overall hypothesis is:

The Second Law of Thermodynamics tells us that our systems are predestined to increase in complexity until they fail, become too complex to manage, or are made obsolete. We can manage the rate of increase of complexity, but never reverse it.

The hypotheses described herein do not contradict the idea that our systems should be kept as simple as possible. Simplicity is still an essential goal. However, we must realize that the increase in complexity is inevitable and irreversible. We must actively work to prevent complexity from increasing faster than we can manage it.

Here are some key implications of these ideas for system builders:

  • If your system isn’t continually evolving and increasing in complexity, it will collapse
  • You can extend the lifetime of your system by investing energy to manage system complexity
  • You can extend the lifetime of your system by continually introducing and developing new acolytes who understand and can maintain your system
    • This enables collective management of complexity and transfer of knowledge about the system
  • You can extend the lifetime of your system by giving others the keys to understanding your system (documentation, training)
    • This enables others to come to terms with the complexity of your system
  • You can never return to "simplicity" - don't consider a "total rewrite" effort unless you are prepared to scrap the entire system and begin again
  • These hypotheses speak to why documentation becomes such a large burden
    • Documentation becomes part of the overall system's complexity, requiring a continual increase in resources devoted to managing it

Developing a skillset in Complexity Management is essential for system designers and maintainers.

Further Reading

Related Articles

Related Books

A Look at Ten Hardware Startup Blunders, Part 4: Conclusion

At Embedded Artistry, we've spent countless hours meeting with founders, working at startups, and swapping war stories with fellow engineers. After enough iterations, we started to notice a repeated set of critical strategic missteps made by new hardware companies. These missteps result in compromised designs, increased development costs, and continual schedule delays.

We've selected ten mistakes regularly observed at new hardware companies which fall into three general categories: process, focus, and team. This article is the final article in our four-part series

  1. A Look at Ten Hardware Startup Blunders, Part 1: Process
  2. A Look at Ten Hardware Startup Blunders, Part 2: Schedule and Focus
  3. A Look at Ten Hardware Startup Blunders, Part 3: Team

A Review of Our Top Ten

Mistakes made at the beginning of a company's or product's life can have long-reaching effects.

The beginning is the most important part of the work. -- Plato

The impact of the mistakes we highlight are not immediately apparent. Like cancer, the damage slowly builds up over time until there is no going back. Your company will have to undertake painful exercises to get back on course, and if the damage is too great your team may not fully recover.

When the problems are finally big enough that your team must face them head on, they are often so far removed from the causes that it feels like an inevitable consequence of building a new product. Keep an eye out for signs of these common blunders and make a course correction before it becomes too painful:

  1. Failure to separate research and development phases
  2. Failure to consider data collection requirements
  3. Lack of defined internal processes
  4. Skipping software architecture
  5. Prioritizing mechanical design
  6. Setting the schedule based on hardware milestones
  7. Heading straight to China for manufacturing
  8. Committing to a schedule before having the execution team
  9. Hiring junior or mid-level engineers and expecting senior-level work
  10. Attempting to build a full-stack engineering team

None of these challenges and missteps are new. You can read about some of these observations in engineering classic written in the 1940s. So why are we repeatedly encountering the same basic problems?

Jack Ganssle, an embedded systems consultant, highlights our lack of learning while reflecting on "NASA's lost software engineering lessons":

And today, 49 years later, we still haven't learned those lessons. Large software projects are routinely plagued with the same sort of problems we "learned" to avoid half a century ago.

Jerry Fitzpatrick, a software consultant and author of The Timeless Laws of Software Development has this to say:

Collectively, we don’t seem to be learning from the past. I don’t know if this is due to an education shortfall, the influx of so many new developers, the acceleration of business expectations, or other factors. I suppose if the reasons were obvious, there would be fewer mistakes.

After all this time, the only thing that has changed is that we expect our engineers to produce faster and faster, even though we haven't fixed our fundamental attitudes and processes. We can change the trend and learn from companies who have gone before us.

Further Reading

Here are classic works related to software project management, scheduling, requirements, and estimation:

  1. The Mythical Man Month, by Fred Brooks
  2. Applied Software Project Management by Andrew Stellman and Jennifer Greene
  3. Software Requirements by Karl Wiegers and Joy Beatty
  4. Software Estimation: Demystifying the Black Art by Steve McConnell
  5. The Economics of Software Quality by Capers Jones

For a straightforward approach to software project estimation, try the Wideband Delphi technique described in the Karl Wiegers's paper "Stop Promising Miracles".

We frequently write about product development on Embedded Artistry:

The following websites provide excellent product development and manufacturing advice that can benefit new hardware teams:

Jack Ganssle, a veteran of the embedded systems industry, shared his own "Top 10 List for How Embedded Projects Run Into Trouble". He's also published a series on embedded.com which covers each point in further detail:

Thanks for Reading

Thanks for following this series. We hope that you will be able to identify and avoid these common blunders on your next project. We must learn from our mistakes to avoid repeating them endlessly. Wouldn't you rather solve new and interesting problems instead of rehashing the same old ones?

Would you tell me, please, which way I ought to go from here?”
“That depends a good deal on where you want to get to,” said the Cat.
“I don’t much care where–” said Alice.
“Then it doesn’t matter which way you go,” said the Cat.
“–so long as I get SOMEWHERE,” Alice added as an explanation.
“Oh, you’re sure to do that,” said the Cat, “if you only walk long enough.”

  • Alice’s Adventures in Wonderland, Chapter 6

A Look at Ten Hardware Startup Blunders, Part 3: Team

Updated: 20190205

At Embedded Artistry, we've spent countless hours meeting with founders, working at startups, and swapping war stories with fellow engineers. After enough iterations, we started to notice a repeated set of critical strategic missteps made by new hardware companies. These missteps result in compromised designs, increased development costs, and continual schedule delays.

We've selected ten mistakes regularly observed at new hardware companies which fall into three general categories: process, focus, and team. This article is the third in our four-part series and focuses on team-related mistakes.

Team-related Mistakes

Building a team which can execute on your company’s business goals is a challenging task. Team composition can make or break a new company due to the serious cash flow constraints they face. Companies must focus carefully on hiring to ensure they have a mix of senior and junior engineers while also building a balanced team focused around the core business value. Hiring recklessly or staffing up too quickly can become a serious drain on company resources and morale.

The following three mistakes related to team composition are among the most common we encounter at new hardware companies:

  1. Committing to a schedule before having the execution team
  2. Hiring junior or mid-level engineers and expecting senior-level work
  3. Attempting to build a full-stack engineering team

Committing to a Schedule Before Having the Execution Team

In the previous article, we discussed how teams can build an unworkable schedule by focusing exclusively on hardware development milestones. Another major scheduling mistake is committing to a schedule before hiring an engineering team to execute your plan.

Hardware startups often base their schedules around market timing or investor expectations. The schedules are entirely fantastical, since teams construct them without input from the engineers who will be designing and building the product. Fundamentally, the company has agreed to deliver a product before having the ability to produce that product.

Teams making this kind of overly-aggressive schedule also exhibit unrealistic expectations for the time it takes to hire engineers, especially firmware developers. Young teams often think they can fill an engineering position in 2-6 weeks. Reality is not so kind: in the San Francisco Bay area, we find that it takes between 3-6 months to hire for each firmware engineering position. If a team’s schedule is dependent on quick hiring, reality will expose the plan as impractical.

When the leadership team sets the schedule without input from the actual engineers doing the work, and the hiring takes longer than expected, the new engineers that join the team are immediately put under pressure. The firmware is almost always “late” by the time the first engineer joins the team, and everyone else on the team is anxious to see the system working. This situation leads to conflicts, cutting corners, unhappy engineers, and delayed releases.

Your schedule is ultimately dictated by the capabilities of your team and the staff you have on hand at the present moment. You need your team’s input when creating the plan. Keep these facts in mind and create a schedule that matches your team’s realistic capabilities.

Hiring Junior or Mid-Level Engineers and Expecting Senior-Level Work

The most common mistake we see young hardware companies make is hiring junior or mid-level engineers and expecting them to produce senior-level work. This can occur because the CEO is not aware of a senior engineer’s value, because the schedule “necessitates” beginning work without senior engineering guidance, or because the team gave up on finding a senior engineer after a long search.

Hiring a senior engineer is essential for the success of a project, especially in today’s world of highly interconnected systems. These companies simply must start with someone who has shipped products before and led the whole process first hand. Junior engineers lack the experience, perspective, and design skills necessary to architect and build new systems. When someone with underdeveloped engineering and design skills builds a system, the most common result is a hard-to-maintain system held together with duct tape and prayers. What’s under the hood often resembles a shanty town, not the 5-star hotel the CEO envisioned. New features become increasingly difficult (or impossible) to implement due to the fragility of the system. When you add multiple interacting systems into the mix, the resulting design problems can multiply and prevent you from shipping your product.

Senior engineers have the experience and foresight to set requirements and identify critical design problems before it’s too late. They can provide guidance to junior engineers and ensure that they aren’t overtaxed. Senior engineers are also crucial when working with contractors, design houses, and other external suppliers. Junior engineers may not have the experience or perspective to review the work products, judge whether the quality is sufficient, and determine if the design satisfies the requirements. Having a senior engineer to manage the work of external teams acts as an insurance policy against integrating low-quality work into your system.

If you want to avoid schedule delays and hard-to-maintain systems, you must have senior engineers on staff.

Attempting to Build a Full-stack Engineering Team

Many new hardware companies focus on building a full-stack team that can handle all aspects of product development. As my friend Saket Vora pointed out, companies should instead build a team that focuses on their core business value.

Most small companies outsource accounting, bookkeeping, and payroll activities to external firms, since they do not directly contribute value to the business. For most new hardware teams, they can likewise outsource the bulk of firmware and hardware development to consultants, contractors, and design firms. Given the difficulty in hiring senior engineers at early stage companies, utilizing external experts can also help alleviate the problem of expecting senior-level work from junior engineers.

Unless your company’s focus is innovative firmware or electrical development, writing firmware and designing circuit boards does not add direct value to your business. Your team will need one experienced engineer who can integrate and review the outsourced work. Your remaining full-time engineers should be fully focused on feature development, integrating with external services, and improving the customer experience.

Another aspect to consider is whether you can justify the employment of your full-stack team. We’ve repeatedly seen companies hire multiple full-time firmware and electrical engineers and successfully keep the team busy while shipping the first product. As efforts shift from new product development to feature improvements, marketing, and customer support, these engineers find there is no work for them to do. If you don’t immediately have the second product on deck, the under-utilized engineers get laid off, leave the company because they’re bored, or simply hang around and serve as a drain on your runway.

Headcount is a precious commodity, and it can quickly turn into an anchor on your company. Make sure you carefully build your team to deliver on your core business value propositions. You don’t get style points for doing it all in house, and the customer can’t tell the difference.

Next Up

That’s it for common team mistakes. Next week, we’ll summarize our list of ten common blunders and provide resources for teams that wish to learn more about product development and project management.

Change Log

  • 20190205
    • Added links to the other articles in the series