Engineering Lessons I’ve Learned from Working at the Japanese Tea Garden

20 April 2020 by Phillip Johnston • Last updated 21 June 2022

Since September 2017, I’ve been volunteering at the San Francisco Japanese Tea Garden in Golden Gate Park. I work with the head gardener, seasoned bonsai devotees, and a master gardener to prune trees, mend fences, beat back bamboo, pull weeds, catch koi fish, and clean ponds. These tasks may not seem relevant to engineering teams, but my time in the Japanese Tea Garden has allowed me to observe how the Garden is managed, maintained, and expanded. I am a business owner and engineer first, so I am continually on the hunt for competitive advantages and valuable lessons.

There is more momentum and legacy wrapped up in the Japanese Tea Garden than even our most storied and long-running software projects. The San Francisco Japanese Tea Garden was constructed in 1894 for the Midwinter Fair. It has been a continual fixture of Golden Gate Park, remaining in operation for 125 years and counting. You feel a visceral sense of history and time as you walk through the Garden, touch the trees, and hear the stories.

The Japanese Tea Garden is a continually operating business. It does not close for maintenance or holidays. It is not a trivial business either; as of 2019, the Japanese Tea Garden receives around 600,000 visitors yearly. Work must happen while the garden is open, which sets constraints on execution. It may not surprise you to learn that these constraints lead the garden to adopt a working and planning style similar to the Agile and Extreme Programming movements.

I believe that programmers and engineers benefit greatly from learning how other businesses and industries manage their problems. Our field is young, but inevitably we will maintain and improve software infrastructure that is a century old. There is no nobility in struggling and re-creating the lessons learned by other industries and by other people in other times.

What I’ve learned from the Japanese Tea Garden can help you manage your own soon-to-be-long-running projects.

Table of Contents:

1. Beyond the Standard Gardening Analogy for Software
2. My Mental Model of the Japanese Tea Garden Process
3. Meta-lessons Learned
   1. Know Your History
   2. Have a Clear Vision, but Make Changes Incrementally Over Time
   3. Maintenance Dominates
   4. You Can’t Make Everything a Feature
   5. The Best Improvements are Imperceptible to the End Users
4. Practical Lessons Learned
   1. Refactoring: Coarse-to-Fine
   2. When You Don’t Know What Else to Do, Start by Tidying
   3. The Job Isn’t Done Until You’ve Cleaned Up
   4. Many Important Jobs are Tedious
   5. End the Day with Something Usable
5. Putting it all Together
6. Further Reading

Beyond the Standard Gardening Analogy for Software

One of my favorite analogies for software development is that it is akin to gardening. I cannot deny that it is my favorite analogy because of my love for gardening, plants, and the time I’ve spent observing nature.

Software projects have always felt similar to my gardening experiences. If we do not continually prune, weed, and maintain a garden, complexity and entropy will take over and overwhelm us. Incremental and continual efforts to abate the chaos and complexity of the garden (or software system) is more efficient than the alternative: ignoring the garden (or software system) until it is weed-ridden and unusable and then finally deciding to clean it up in one singular, exhausting effort. Continual extraction of weeds while they are young is much more efficient than mass removal of well-established and deeply rooted plants. Likewise, it is more efficient to fix software issues as they arise rather than defer them until we are ready to pay for the corrections in a mass effort.

This is not to say gardening is the only viable analogy for software development. Software development can simultaneously be like gardening (I say as a gardener), like engineering (I say as an electrical engineer), and like construction (I say as the son of a home builder).

In this article, I am moving beyond the typical form of “software is like gardening” or “software is like construction” analogies. Here, I consider that “maintaining a software product after launch (for years and years!), is like maintaining and improving the San Francisco Japanese Tea Garden”.

My Mental Model of the Japanese Tea Garden Process

Before we dive into the lessons learned, I want to spend some time showing you how the operation of the Japanese Tea Garden is sufficiently complex and is useful for learning lessons relevant to engineering and software projects.

The Japanese Tea Garden is alive and continually evolving. This is true at a basic level, because it is composed of plants and fish, which have their own lives, behaviors, and tendencies. But it is also a system that is shaped, maintained, and enjoyed by humans, each with their own ideals, behaviors, and tendencies. It’s a business with an income, and accordingly the Golden Gate Park management and the customers have their own expectations for the Garden.

The Japanese Tea Garden has a legacy and a context that must be kept in mind. Japanese gardens in general have a certain aesthetic, and the San Francisco Japanese Tea Garden in particular has its own aesthetic and history. These aesthetics constrain the choices that gardeners can make.

The Garden is old. When you work in the Garden, you are never starting from a blank slate. Something with a history already exists in the space you are working. Every plant has a story attached to it. The Garden has evolved over time. Entire sections have been added, removed, or changed at different times by different individuals with different visions. Dozens of gardeners have taken part in its care and development, each with a different understanding of the Garden and how to care for it. These decisions continue to ripple through the Garden’s design.

Currently, the head gardener’s ideal reigns: we must improve the Garden while honoring and preserving the essence of what it is. In the future, we can expect the ideal to shift in a new direction. But even then, knowingly or unknowingly, the future maintainers will still be bound by the momentum of the Garden’s history. There will still be the inevitable complaints about the poor decisions of the people who came before them, just as there are today.

Because of its continual operation, customer expectations, and the existing structure, we cannot make dramatic changes to the Garden. Some work can be done “behind the scenes” (before people arrive), such as watering. Most work is performed while people are visiting the Garden. Complex tasks are split into smaller chunks to reduce the impact on the customer’s experience.

We must do some work, even if it will disrupt the customer experience. For example, cleaning the ponds disappoints people, as they are a significant part of the Garden’s makeup. However, not maintaining the pond is disastrous: algae builds up, the water turns green and nasty, and the beautiful koi fish die. Such work is structured to minimize the impact on the Garden experience, so even if a pond is being cleaned there is still 90% of the Garden to enjoy.

Most of the day-to-day work in the Garden is analogous to software development. New features (trees, plants, and rocks) are added. Old features (topiary-style pruning and cluttered scenes) are removed when they no longer serve a purpose. There are “bugs”, such as pipe leaks, pond leaks, rotting fence posts, dying plants, rodents, and the ceaseless bamboo takeover attempts.

Some changes happen quickly, such as pruning a tree. Other changes happen slowly, such as changing the overall shape and structure of a tree. Some changes in the Garden are made on such a long timeframe that you might not be alive to observe their final development.

The Japanese Tea Garden deals with standard project management issues. Necessary work can’t begin because there is no budget or leadership interest. We can see this in the current state of disrepair of the pagoda, which has been unmaintained for decades and is now visibly rotting. The head gardener has finally created political will and a budget to repair the pagoda. Even though a budget is approved and the contractors are ready to begin work, city bureaucracy is adding unanticipated delays.

As always, there are never enough people to support the amount of work that needs to be done, both within the Japanese Tea Garden and within Golden Gate Park. Last-minute fire drills happen when Golden Gate Park managers suddenly demand help on other Park projects. These fire drills cause schedule delays for previously planned Tea Garden work.

Because of these factors, work scheduling within the Garden resembles an Agile or Extreme Programming approach. The park has to stay open and not disappoint customers, and the solution is the continual improvement of the Garden in daily cycles. The team kicks off their day with a meeting that resembles a standup: the team discusses the work to be completed that day, including any unplanned changes to the schedule or scheduled help from apprentice gardeners. There is a focus on individual tasks that can be performed entirely within one working day, from start to finish, including tidying up. Regular chores have a checklist. Work is divided up across the team, with different gardeners focusing on different areas of the Garden.

Meta-Lessons Learned

The Japanese Tea Garden is a complex system with continual pressure to add new features, to maintain the existing feeling and structure, and to produce revenue for the City of San Francisco. The garden is continually being used by customers, and it is continually being maintained and improved by the staff.

We are in a similar situation when we are building and maintaining software and electronic products. Many of us have to deal with legacy systems that were created by people who came before us. Often, we dislike these designs or the code. We obsess over improving the system or throwing it out completely and starting over. We find ourselves constrained in the same way as the gardeners at the Japanese Tea Garden. We are understaffed, under-funded, and have paying customers to support. There are valuable lessons to draw from the art of improving and maintaining an actively used garden over such a long timeframe. The following project management meta-lessons I’ve learned provide valuable insights into maintaining legacy systems, how we should think about our projects, and the way we plan our work.

1. Know Your History
2. Have a Clear Vision, but Make Changes Incrementally Over Time
3. Maintenance Dominates
4. You Can’t Make Everything a Feature
5. The Best Improvements are Imperceptible to the End Users

For lessons that developers can apply directly, see the next section.

Know Your History

One of the most important lessons I’ve learned from working at the Japanese Tea Garden is the importance of knowing your history and the context for your work. You cannot make effective changes within a system if you don’t know where you’ve come from and where the ship was previously headed. You cannot make effective changes if you do not understand the goals and ideals of the framework you’re operating within.

The head gardener at the Japanese Tea Garden models this daily through his research into the Japanese Tea Garden’s history and by studying Japanese gardens around the world. After the internment of Japanese Americans during WWII, many of the gardeners responsible for maintaining the Japanese Tea Garden knew nothing about Japanese gardens or Japanese gardening techniques. British topiary forms crept into the garden, plants were roughly trimmed with power tools, scenes were crowded with new plants, and the style began to shift. By understanding this historical context and improving his knowledge of Japanese gardening, he has slowly course corrected and reversed many of these changes, once again regaining the former Japanese garden aesthetic.

How many of us as software developers and engineers spend any time understanding the history of our projects? We often just complain about how terrible the code is and how the previous developers hacked together a shantytown instead of taking the responsible path. Rarely is there an effort to understand the historical context behind the system and the decisions that lead to the current moment. We rarely consider how many previous developers were involved and the impact that circumstances had on their work. Rarely are we patient enough to work within the existing context to make improvements over a long timeframe.

If you don’t know where you came from, how can you decide where to head next? How do you know that you won’t recreate the same situations and pains that were already experienced in the past? How do you know that you aren’t contributing to the eventual downfall of your system?

By studying the context and history of your system, you are better armed with information that can improve your project.

Have a Clear Vision, but Make Changes Incrementally Over Time

One difficulty with maintaining and improving a garden is the different time scales you have to keep in mind. You might want to make drastic changes to a tree, but it will take 10–20 years of new growth until your vision can be fulfilled. You must learn to hold a clear vision in mind for the end goal. You must become comfortable working toward that vision with small, incremental changes made continually over a long timeframe. You are making massive transformations over months and years, although in the present moment it looks and feels like you are making no progress.

I have never been a part of a software project that took this slow, incremental approach to improvement, even on supposedly Agile teams. For most of the industry, speed is king. We focus on how quickly we can make changes and build a new system. Another industry tendency is to throw out a frustrating or poorly designed system and build a new one from scratch. This new system will fix all the problems that the old system exhibits, we say. Inevitably, this second system is no better than the first system and exhibits many of the same problems. These efforts also take much longer than the team originally envisioned, often dragging on years after their original release schedules.

Continual, incremental changes are less of an imposition on your customers and on your own internal teams. A complete teardown and rebuild not only causes a delay, but it requires everyone to re-learn these systems from the ground up. Small incremental changes can sometimes go unnoticed by your customers altogether.

Consider an alternate scenario the next time you are considering a massive rewrite: what if you just steadily worked to improve your existing code base over the same timeframe? Identify the problem areas or qualities that need improvement, define and document a vision for where you are headed, and create quantifiable goals that you can use to evaluate your progress. Then, figure out how you can slowly morph the code piece-by-piece, one day at a time. While the immediate payoff is low, your improvements will quickly compound over a timeframe of months and years.

Rarely do we find that a drastic sweeping change is useful. Sometimes they are called for, but it’s much rarer than you might think.

Maintenance Dominates

Over two-and-a-half years at the Japanese Tea Garden, I have seen two new trees and a few stones added to the garden. Most of the work we perform is maintenance, moving plants, or removing plants completely.

As engineers and software developers, we have rose-colored glasses when taking on a new project. We always think about the joy of creation and building something new. Rarely do we look to the future, where we must maintain and support our project for years and years. Engineers commonly look at this “maintenance” work with disdain. New employees handle the maintenance tasks, while the experienced engineers move on to a new project.

The reality of building things is that on a long enough timeframe, maintenance work dominates, not novel creation. We must actively maintain our projects, or the increasing complexity of the world will render our project obsolete. This maintenance obligation never ends. Inevitably, many projects die because their maintainers become tired of the work.

Remember, too, that adding more features (or plants to a garden) means even more work to maintain the program/garden. If you add too many things too quickly, you might overwhelm yourself with the sheer volume of unanticipated maintenance work.

Frequently, I see the gardeners make decisions because it will reduce maintenance requirements while still preserving the Garden aesthetic. I rarely see these decisions made on software projects; instead, the focus remains entirely on adding to the software with no regard to future maintenance requirements.

Here’s a thought exercise: what work or decisions would reduce the time you spend on maintenance efforts for your projects? These improvements pay dividends in time savings and free you to work on something new.

You Can’t Make Everything a Feature

When we are pruning a tree, our focus anchors us solely to the tree we are working on. We discuss changes within that focused context. Eventually we will step back and realize that the tree is just one component of a much larger scene. Often, we see that we spent too much time debating about detailed changes for a supporting character.

We commonly repeat the mantra of “not everything can be a feature” while we are working. This is an attempt to force our minds into maintaining the larger context. A scene that is exclusively composed of features lacks cohesiveness. Nothing stands out. A supporting cast of plants serves important roles, such as framing the scene, providing color, or drawing your eye toward the primary feature because of the “flow” of the different shapes. Ideally, we want to work rapidly on supporting plants and focus our detailed efforts on the features.

We see a tendency for feature creep on software projects too. Marketing teams emphasize the new features that have been added in the latest release. A product that starts out with a focused set of features eventually grows into an unwieldy mess. Users might wait years for fixes and improvements to the core functionality while the development team continues to churn out new features for those marketing bullet points. Rarely are the new features the reason that people wanted to use the product in the first place.

Step back and remember: not everything should be a feature. We must keep the whole scene in mind and focus our efforts on what really matters.

The Best Improvements are Imperceptible to the End Users

It is often the case within the Japanese Tea Garden that those improvements we consider the “best” are ultimately indiscernible to customers outside of producing a general good feeling. We might prune a tree or remove a plant, but we are the only ones who really notice the details of the changes. The composition of the scene remains largely the same, but it just “feels better” and “works better” than it did before.

This differs from many software products, where teams are constantly changing software designs, interfaces, and behaviors. Consider how you feel when Twitter and Facebook change their designs. Many users are angry and frustrated in these situations, not left with a “general good feeling.”

In our experience, internal team desires are the driving force behind new designs, not customer requests. As a customer, I’m looking for fixes to the existing problems. Instead, companies regularly deliver a new design or new features but continue to ignore major flaws and bugs for years on end. I feel a pervasive feeling of frustration in these situations. I bet that is not what the team thinks their customers are experiencing.

Small, incremental changes feel better, because these changes are less of an imposition of your customers. We are not requiring them to re-learn our system over-and-over-again. This is true for gardens, software, and many other aspects of human life. Tear-downs, rewrites, and redesigns might bring us enjoyment and internal glory. But the small, incremental, unnoticed changes bring your customers more joy than those massive changes.

Practical Lessons Learned

I would be remiss to leave out actionable lessons that can be used in our day-to-day software development and engineering work. Here are five lessons I’ve taken from the Japanese Tea Garden:

1. Refactoring: Coarse-to-Fine
2. When You Don’t Know What Else to Do, Start by Tidying
3. The Job Isn’t Done Until You’ve Cleaned Up
4. Many Important Jobs are Tedious
5. End the Day with Something Usable

Refactoring: Coarse-to-Fine

Whenever we are pruning a tree, the general principle is to “remove coarse-to-fine”. You want to look for large changes before you focus on smaller details such as thinning out the foliage.

This is the lesson that has been hardest for me to internalize. There have been countless occasions where I have spent 15–30 minutes pruning a branch, only to have the head gardener come by with a saw and completely remove the branch.

We do not want to waste our effort in this way. Throwing away work never feels good, especially when we suddenly see that we could have avoided it altogether.

Take a moment and elevate your perspective when beginning a new task. Make sure you see the entire scene and understand it. Identify the biggest changes to make or components to build. Make those before you work on any of the smaller tasks or refinements.

Don’t waste your time and energy by making small changes that could be eliminated with one a single stroke.

When You Don’t Know What Else to Do, Start by Tidying

Sometimes I am unsure of how to approach pruning a tree. I know, intellectually, that it needs to be “thinned out” so that light hits the interior, the trunk and hidden branches become visible, and the tree itself feels lighter. But when I look at the tree, or even zoom into individual parts, I don’t know where to start.

In these moments, I fall back on a simple activity: tidy up the tree. I remove dead leaves, dead or dying branches, and detritus that has fallen from taller trees. This act serves two purposes:

1. Even if I do nothing else to the tree, I’ve improved the scene by cleaning it up
2. As I work on cleaning up the tree, I learn its structure and see the interesting and unnecessary parts

The same approach will work whenever you feel stuck on a programming or engineering problem. If you find yourself unsure of where to begin, start by tidying the code, design, or documentation. Fix the formatting, add comments, remove unnecessary statements or components. As you work, you will notice other ways to make improvements.

The Job Isn’t Done Until You’ve Cleaned Up

When making estimates or planning our days or weeks, most of us budget our time based on how we execute the task itself. For example, when planning trees to work on, we are tempted to only budget time it takes to prune the tree.

Working in the garden emphasizes a stark fact: the job of pruning a tree is more than just the act of pruning. The job isn’t done until clippings are removed from the ground, the paths are swept to look neat, trampled grass has been raked and made to look untouched, the clippings are taken to the compost pile, the tools are cleaned, and everything is put away.

Sometimes cleaning up can take longer than the primary task.

Whenever you’re estimating or planning, are you accounting for the time spent cleaning up? Or just the development effort itself?

I observe that developers (including myself) often estimate the time to write the code or solve the problem. We forget about the time we’ll spend writing tests, debugging our design, cleaning up the code for publication, reorganizing our branches and commit messages, updating documentation, working through code reviews, addressing feedback, merging into master, and working with QA to confirm that the problem is fixed.

Keep this in mind: the job isn’t done until you’ve completed the primary work and cleaned up after yourself.

Many Important Jobs are Tedious

Some important jobs are fun, like pruning trees or catching koi fish. However, many important jobs are tedious. We still must do them.

In the garden, sometimes you have to weed, pick diseased pine needles off of a tree, or dig a hole for a new fence post. Picking off individual needles is tedious, careful work, but doing it makes the plant look beautiful and helps fight the infection. Without fence posts, the garden would be trampled with foot traffic in places that people should not be. Rotting posts will topple when someone leans against them, creating a liability risk.

Many engineers think they are above tedious and boring work, especially if it involves maintenance. Many push such work down the totem pole. Understand: the importance of the work is never directly related to your enjoyment. Important work must be done, even if it is not fun. The whole team should share in the burden of the tedious-yet-important jobs, not just those at the bottom of the totem pole.

We can make tedious jobs bearable by taking a moment to reflect on their value. We can gamify the situation, racing to see who can accomplish the task the quickest. We can use tedium as a stand-in for meditation, allowing ourselves to become completely wrapped up in the task at hand, performing it to the best of our ability. Such strategies are effective, whether you are gardening, washing dishes, filling out paperwork, refactoring a program, writing tests, or manually counting resistors for a hardware assembly kit.

End the Day with Something Usable

When working in the Garden, the goal is to end a work session with something the public (i.e., your customers) can use. We don’t want to leave plants in a state that is obviously half-complete. We remove equipment at the end of each working session, and evidence of our work is removed. The result is that the garden is in a continually viewable and pleasant state.

Sometimes this approach isn’t possible. Cleaning the ponds is one example. You need to drain the ponds half-way, catch the koi fish, complete the draining, clean out the organic matter from the pond floor, prune plants which overhang the pond, refill the ponds, and replace the fish. That work cannot be completed within a single day, especially because it takes an entire day just to drain or fill a pond.

The gardeners do their best to minimize the impact of pond cleaning on the customer experience. The Garden contains four separate ponds. In the past, all the ponds would be drained at once, and they would remain empty for 2–3 months while cleaning proceeded. These ponds are a major feature of the Garden. Keeping them empty for such a long period is unattractive and disappointing for customers. Now, the cleaning is scheduled so that only one pond is out of commission at a time. Each pond is out of commission for only one week. As a result, they shorten the overall timeline for pond cleaning. Some customers still express disappointment when there is pond work, but at least they can enjoy the rest of the garden.

A common tendency for software developers is to work on a feature or branch for a long time, making infrequent merges to master once everything is working. This is akin to keeping the ponds empty for 3 months at a time. When we work in this style, we often find that master received changes which force us to update our branch or that our changes conflict with another developer’s pending changes. Instead of working on your own branch for months at a time, larger efforts should be split up, reviewed, and integrated in pieces. We don’t want to wait until completing the entire overarching effort before we integrate our changes.

This is not a new idea within the software development world. Ending the day with something usable is akin to the Extreme Programming ideal of continuous integration, which is now taking the form of continuous delivery.

Putting it all Together

At some point, almost all of us work in maintenance mode on projects that already exist. To maximize our effectiveness, we must learn how to explore and improve these systems with the proper mindsets. Using the lessons learned from the San Francisco Japanese Tea Garden can help us deal with legacy software and systems.

I find it fascinating that two different operating environments can converge on similar difficulties, lessons, and practices. It is important to notice that solutions like incremental improvement and continuous integration/delivery are generally applicable to multiple fields.

According to the head gardener, processes similar to those proposed by Agile and Extreme Programming adherents are not only adopted in the maintenance of the garden, but also in the creation of a Japanese garden:

A difference between the process of how Japanese and western gardens are built is that while western gardens are generally built from a master plan down, Japanese gardens are constructed from the details up. A stone is placed, and then a tree is placed in relation to that stone, and then the path is constructed to take advantage of the best view of that scene, etc. It isn’t that a conceptual plan doesn’t precede the construction. It’s just that in the work of putting the pieces together, the materials dictate the flow in a much more improvisational nature. Rigidly adhering to a master plan would miss the opportunities found during the process.

We continue to re-discover the value of these ideas in different fields, and that is an indication of their importance.

Further Reading

• Hypotheses on Systems and Complexity
• San Francisco Japanese Tea Garden
• Photos and History of the San Francisco Japanese Tea Garden, compiled by Steven Pitsenbarger, will give you a sense of the history present in the garden
• California Midwinter International Exposition of 1894
• Tending Your Software Garden
• Programming is Gardening, not Engineering
• Software Gardening: Yet Another Crappy Analogy or Reality?
• Extreme Programming
• Continuous Delivery

For other takes on the metaphor of “software as gardening”, see:

• Tending Your Software Garden
• Programming is Gardening, not Engineering
• Software Gardening: Yet Another Crappy Analogy or Reality?