Exploring Serverless CI/CD for Embedded Devices

Today we have a guest post from Kayode Alade. Kayode is an IoT Solutions Architect with a knack for making things work. He considers himself a perpetual learner, always enthusiastic about acquiring new knowledge, whether from an engineering perspective or a business standpoint. You can find Kayode on GitHub and LinkedIn. Embedded DevOps is changing …

Dealing with Signed Commits When Creating and Splitting a Monorepo

A major roadblock on the way to the goal of developing in a monorepo and distributing to standalone repositories arose with signed commits. This article logs the problems I faced, options for addressing the issue, and my thoughts on the matter. This is still an open investigation, and I am logging my ideas and results. …

Embedded Artistry’s GitHub Process

This document describes the Embedded Artistry development process. Following these guidelines shows that you respect the time and effort of the developers managing our open source software projects. In return, you will be shown respect in addressing your issue, reviewing your changes, and incorporating your contributions.

Table of Contents:

  1. Development Process
    1. Forking
    2. Protected Branches
    3. Submitting Changes
    4. Continuous Integration
  2. Code Review Requirements
    1. Addressing Feedback
  3. Rebasing and Updating
  4. CCC Process
  5. git-lfs

git-lfs

Many Embedded Artistry projects store binary artifacts, images, and reference documents using git-lfs. If you do not have git-lfs installed, you will have small (100–200B) placeholder files instead of the actual content. For many projects, this will not be a problem, as the git-lfs files are often images or pdfs. Some projects may store binary files, and these require git-lfs usage.

To install git-lfs on Linux or with WSL:

sudo apt install git-lfs

To install git-lfs on OS X:

brew install git-lfs

Additional installation instructions can be found on the git-lfs website.

Development Process

We do not currently used the widespread NVIE git branching model for our open source projects. Instead, we maintain a much simpler GitHub flow.

For all Embedded Artistry projects, master contains the latest code. The master branch shall always remain in a usable state. Changes which render the master branch unusable will be immediately corrected or reverted.

When using git, it is best to isolate each topic or feature into a “topic branch”. Branches are a great way to group commits related to one feature together, or to isolate different efforts when you might be working on multiple topics at the same time.

While it takes some experience to get the right feel about how to break up commits, a topic branch should be limited in scope to a single issue. If you are working on multiple issues, please create multiple branches and submit them for review separately.

# Checkout the master branch - you want your new branch to come from master
git checkout master
# Create a new branch named newfeature (give your branch its own simple informative name)
git branch newfeature
# Switch to your new branch
git checkout newfeature

Please create your branch from master when making any new changes. Once you are ready to merge changes, open a pull request. The build server will test and analyze the branch to ensure it can be safely merged.

Forking

Note: Members with write access can make branches directly in the repository. Creating a fork is optional in this case.

You will need to fork the main repository to work on your changes. Simply navigate to our GitHub page and click the “Fork” button at the top. Once you’ve forked the repository, you can clone your new repository and start making edits.

For more information on creating a fork and submitting a PR using a fork, please see this helpful guide.

Protected Branches

The master branch is marked as “protected” for most of our projects. Changes cannot be submitted to the master branch unless the proper process is followed:

  1. Changes cannot be pushed directly to master; a PR must be opened
  2. The Continuous Integration (CI) server must verify that the changes satisfy our code quality processes
  3. The code must be reviewed.

These protections apply to all contributors, including administrators.

Commit Requirements

Please see our Source Code Commit Guidelines for information on structuring your commits and commit messages.

In general:

  1. Commits should be “atomic”, which means all changes are related and accomplish a single goal
  2. Avoid large commits with lots of changes
  3. Include a subject line that summarizes the change
  4. In the commit body, describe why you’re making the change and what it is supposed to accomplish

Continuous Integration

We strive to utilize continuous integration (CI) on all of our projects. We maintain a Jenkins build server that runs automated checks for Embedded Artistry organization projects.

The CI process is automatically launched when:

  1. A pull request is opened
  2. A new commit is pushed to a pull request
  3. A new commit is pushed to master

Many of our projects will upload test results and code annotations to the GitHub pull request using Report.CI. This allows you to view test failures within the GitHub interface.

For more information about our CI process, see this Field Atlas entry.

Submitting Changes

All changes to Embedded Artistry open source projects shall be handled through the pull request process. If you are new to pull requests, please see our Open Source Contribution guide.

Please fill out the pull request template when populating the PR body. This helps provide the proper context for maintainers and reviewers. Additionally, the manual checklist helps eliminate common errors prior to code reviews.

Once you open a Pull Request, the CI process will automatically kick off. The build server will report either successful or failed status checks. You will see something similar to the following:

Here is what a successful CI build looks like: If the status checks fail, you must resolve the issues. Please tag @phillipjohnston in your PR if you cannot see why your build failed.

Once the CI status checks pass, your changes will be reviewed by a project maintainer.

When all of the steps are complete, your PR will be merged.

Code Review Requirements

Your changes will be reviewed before they are accepted. Before reviews begin, changes must build correctly on the Jenkins CI server, be free of formatting errors, and pass tests.

Project maintainers will reveiw your changes and provide feedback. We work on a variety of projects, so please expect some delay in getting back to you with a code review. I will notify you as soon as I have seen the PR and provide insight into the expected review timeline.

If you are a new code reviewer or contributor, please see the Field Atlas entry on Giving and Receiving Feedback.

Addressing Feedback

During the review, constructive feedback may be provided by the project maintainers. Feedback isn’t meant as an attack, but to help make sure the highest-quality code makes it into our project. Changes will be approved once required feedback has been addressed.

Rebasing and Updating

If a maintainer asks you to “rebase” your PR, they’re saying that a lot of code has changed on master since you created your branch/fork. You need to update your branch/fork with the latest changes from master so it is easier to merge.

If you are working from the primary repository, you need to update master and then rebase your changes onto it.

git checkout master
git pull
git checkout -
git rebase master

If you’ve forked the repository, you will need to add the primary repository as a “remote” so you can pull in changes. You can also update your fork directly in GitHub using the Pull Request interface.

To add the primary repository to your fork as another “remote”, follow these steps:

git remote add upstream <https://github.com/organization/project>

Note: You only need to do this once.

To update your forked repository from the command line, follow these steps:

# Fetch upstream master and merge with your repo's master branch
git fetch upstream
git checkout master
git merge upstream/master
# An alternate for the last step, which throws away any changes on 
# your copy of master and replaces it with the original project's master branch
git reset upstream/master --hard
# Now, rebase your development branch
git checkout newfeature
git rebase master

If too much code has changed for git to automatically apply your branches changes to the new master, you will need to manually resolve the merge conflicts yourself.

Once your new branch has no conflicts and works correctly, you can override your old branch using this command:

git push -f

Note that this will overwrite the old branch on the server, so make sure you are happy with your changes first!

If you have any questions about this process, please ask for help!

CCC Process

Occasionally, projects enter a period of high scrutiny, especially before a new release. During these periods, the Code Change Control (CCC) process is enacted. You may see a new pull request template during those periods.

The CCC PR template provides much more rigor and documentation than a normal PR message. When we need stability and strict testing, such as when attempting to stabilize for a customer release, the CCC process helps ensure that we include changes that are thoroughly tested and justified.

Please understand that this process is not meant to keep you out, but to ensure that changes are sufficiently vetted before they are accepted.

Further Reading

Open Source Contribution Guide

Welcome! We love receiving contributions from the community, so thanks for stopping by! There are many ways to contribute, including submitting bug reports, improving documentation, submitting feature requests, reviewing new submissions, or contributing code that can be incorporated into the project.

Some of the information described below is specific to our processes, but most of it is generally useful for GitHub-based open source work.

If you are working on your first open source project or pull request, here are some helpful tutorials:

Table of Contents:

  1. Code of Conduct
  2. Finding Something to Work on
  3. Feature Requests
  4. Reporting Problems
  5. Improving Documentation
  6. Branching
  7. Pull Request Guidelines
  8. Style Guidelines
  9. Further Reading

Code of Conduct

By participating in an Embedded Artistry project, you agree to abide by the Embedded Artistry Code of Conduct.

Finding Something to Work on

The best way to find something to work on is to check the GitHub issue tracker for the project you are interested in. Pick an unassigned issue that you think you can accomplish and add a comment that you are attempting to do it.

If you’re new to a project, or programming in general, keep an eye out for these labels:

  • good first issue labeled issues are deemed to be good low-hanging fruit for newcomers to the project
  • help wanted labeled issues may be more difficult than good first issue and may include new feature development
  • documentation labeled issues apply to new documents or improvements to existing documents

Feature Requests

We welcome feature requests!

For all requests, please file a new GitHub issue in the relevant project. Please provide the details about the feature/function you would like to see and why you need it. If you’re interested in tackling the feature yourself, please discuss your ideas and feedback before proceeding. We may have insights and advice from working on these projects that can benefit your design.

For new features or functions which have not been implemented, please label your request with the “Feature” label. If you are requesting an improvement on an existing function, please use the “Enhancement” label. If you are not a contributor to the project, you may not be able to label issues. It’s fine to stay blank – we’ll label it later.

Small Changes can directly be crafted and submitted directly to a project’s GitHub repository as a Pull Request. See the section regarding Pull Request Submission Guidelines.

Reporting Problems

Before you submit your issue, please search the project’s GitHub Issue archive – maybe your question or issue has already been identified or addressed.

If you find an error in the source code, you can help us by submitting an issue to our GitHub issue tracker. Even better, you can submit a Pull Request with a fix!

Most of our open sourece projects use an issue template, which GitHub will auto-populate when you create a new issue. Please make sure to follow the template and provide the necessary information. This will help us track down the bug as quickly as possible.

For more information about our issue template, please see this article.

Improving Documentation

Should you have a suggestion or correction for the project’s documentation, you can open an issue and outline the problem or improvement. However, creating the documentation fix yourself is extremely helpful!

For larger changes, we recommend creating a new issue (or commenting on a related existing one) to let others know what you’re working on. If you’re making a small change (typo, phrasing) don’t worry about filing an issue first – just open a Pull Request.

Branching

Those users with Contributor permissions can directly clone the repository and work on a branch within it.

Those without Contibutor permissions will need to fork the main repository to work on your changes. Simply navigate to our GitHub page and click the “Fork” button at the top. Once you’ve forked the repository, you can clone your new repository and start making edits.

When using git, it is best to isolate each topic or feature into a “topic branch”. Branches are a great way to group commits related to one feature together, or to isolate different efforts when you might be working on multiple topics at the same time.

While it takes some experience to get the right feel about how to break up commits, a topic branch should be limited in scope to a single issue. If you are working on multiple issues, please create multiple branches and submit them for review separately.

# Checkout the master branch - you want your new branch to come from master
git checkout master
# Create a new branch named newfeature (give your branch its own simple informative name)
git branch newfeature
# Switch to your new branch
git checkout newfeature

For more information on the GitHub fork and pull-request processes, please see this helpful guide.

Pull Request Guidelines

When you are ready to generate a pull request for preliminary review or merging into the project, you must first push your local topic branch back up to GitHub:

git push origin newfeature

Once you’ve committed and pushed all of your changes to GitHub:

  • Go to the page for your fork on GitHub
  • Select your development branch
  • Click the pull request button

If you need to make any adjustments to your pull request, just push the updates to your branch. Your pull request will automatically track the changes on your development branch and update.

Please follow the instructions in the Pull Request Template for this project. GitHub will automatically populate the pull request description with this template. For more information on our pull request template, please see this article.

Once you have opened your pull request, it will be processed as described in the Embedded Artistry GitHub Process guide.

Style Guidelines

Many of our projects use automated code formatting with clang-format. Otherwise, please do your best to mimic the existing style of whatever file(s) you are working in.

You can auto-format your code to match the style guidelines by issuing the following command:

make format

Formatting is enforced by the Jenkins build server which runs continuous integration for this project. Your Pull Request will not be accepted if the formatting check fails. If you cannot install clang-format on your machine for some reason, please let us know and we can help you with formatting updates.

Further Reading

Source Control Commit Guidelines

Preparing quality commits in a source control system is an underrated skill. Commit messages help explain the context for set of changes. We can’t always find an explanation or context for why code was changed by looking at the code. We must keep a detailed project history in some other way, and the commit tree is one of the best places. This history is invaluable when debugging future problems or trying to understand why a previous change was made.

Additionally, how we structure our commits matters. With commits that are too large, reviewers will have trouble processing all of the information. Future debugging efforts are also hindered, because developers cannot pinpoint exactly where an error was introduced when large commits are the norm. We want to build a project history that is truly a future resource for future developers.

The following commit guidelines are used on Embedded Artistry open source software projects to provide a consistent organization style. If your commits and messages are not of sufficient quality, you may be asked to amend them before a pull request is accepted.

In general, these guidelines will serve you well no matter your organization. For other takes on writing quality commit messages, see the Further Reading section below.

Table of Contents:

  1. Prefer Atomic Commits
    1. Splitting Commits
    2. Whitespace and Formatting Cleanup
  2. Commit Message Guidelines
    1. The Subject
    2. The Message Body
    3. Reference Issue Numbers
    4. On Cherry-Picked Commits
  3. Revise Your Commits Before Submitting a PR
  4. Further Reading

Prefer Atomic Commits

In general, the smaller the commit, the better. We prefer atomic commits, which apply a distinct and related set of changes in a single action.

What does this mean in practical terms?

  1. All changes in a commit should be related.
    1. Don’t combine changes that address different problems into a single commit.
  2. All changes in a commit should address a single aspect of a problem or change.
    1. Don’t be afraid to break up a new feature, bug fix, or refactoring effort into multiple distinct changes.
    2. This will help you keep track of what works and what doesn’t: if there’s a problem, you can always revert back to the last known good state. If you wait too long between commits, you may lose a lot of work or spend too long finding the source of the problem.
  3. Prefer small commits to large commits.
    1. This helps reviewers by allowing them to focus on a small set of related changes.
    2. This helps future debugging efforts by increasing the probability that a git bisect operation will quickly identify the source of the problem.

Splitting Commits

Sometimes, we accidentally commit distinct sets of changes together. Or we might mess up a rebase effort, accidentally squashing two unrelated commits.

To undo this problem, review our process on splitting up git commits in the middle of a branch.

Whitespace and Formatting Cleanup

Don’t mix code changes with whitespace or formatting changes! If you are fixing code formatting, include those changes separately from your code changes. If your request is unreadable due to whitespace changes, it will be rejected.

Many of our projects enforce automated code formatting using clang-format. Formatting is enforced by the Jenkins build server which runs continuous integration for this project. Your Pull Request cannot be accepted if the formatting check fails. If your changes are rejected by the server, please include the auto-format updates in a separate commit.

You can auto-format your code to match the style guidelines by issuing the following command:

make format

If you don’t have clang-format installed, we can apply the format changes for you.

Commit Message Guidelines

Below is a visual representation of the Embedded Artistry git commit message format:

50-char-ish"subject" which summarizes the change

After the first line, include additional explanatory text. Include
one blank line in between the summary and the body. Various tools
like git log, git shortlog, and git rebase get confused if the two
sections run together.

You can also include bullet points:
* A
* B
* C

Reference issues at the bottom, like this:

Fixes #102
Relates to #100, #8

The Subject

Treat the first line of a commit message like an email subject line.

We recommend a maximum length of 50 characters because it works best with git’s command line tools.

We also recommend using the imperative mood for subjects. This might be what you think of as “commanding” or “instructing”. Examples include:

  • Refactor subsystem X
  • Implement feature Y
  • Remove deprecated methods
  • Update version to 1.2.3

Another way to look at it is to have your subject line complete the following sentence:

If applied, this commit will [subject line here].

We don’t worry about manual line breaks in our commit message bodies. Text wrapping works well in modern tools.

The Message Body

Please separate the body from the subject with a blank line.

Most importantly, the commit message body should be used to explain what you are doing and why you did it that way, rather than how you did it. The code itself serves to explain the how. Focus on side effects, compatibility changes, or other consequences that are not immediately obvious from reviewing the code. Also include any important factors that helped you arrive at your particular approach.

Not all commit messages require both a subject and a body. You can include only a subject if it is sufficient for a given commit.

Reference Issue Numbers

When working with GitHub projects, reference relevant issue numbers in your commit message body.

You can have a commit automatically close an issue when it is merged to master by saying something like the following in your message body:

Fixes #123
Fixes: #123
Resolves: #123

You can also reference issues in other repositories by adding a prefix for organization/repository:

Fixes octo-org/octo-repo#100
Further reading

The full list of keywords that will close an issue is found in the GitHub documentation.

If other issues or PRs are related to this commit, but it doesn’t address the problem, you can include the issue/PR number without using a keyword like “fixes” or “resolves”. This will link the commit and the issue in GitHub’s web interface.

On Cherry-Picked Commits

Sometimes, we cherry-pick changes from one branch to another. We recommend using the -x flag with git cherry-pick, which references the original commit ID.

For more information, see Git Cherry-pick Recommendations.

Revise Your Commits Before Submitting a PR

While you’re working on a problem on your own branch, don’t worry about getting these commit details perfect. Instead, clean up your branch before you submit a PR. You can perform a git rebase to squash commits, amend commit messages, and more.

We often use a visual tool, such as Sourcetree, for complex rebase process.

Further Reading

For more on writing quality git commit messages:

For more valuable processes to follow:

June 2019: Programmer vs Engineer, Coroutines, Report CI

3 June 2019 by Phillip Johnston • Last updated 27 September 2019 Welcome to the June 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there. This month we’ll cover: Programming vs engineeringCoroutinesReport CIEmbedded news from around the webUpdates to the Embedded Artistry Website We also want to share some exciting news with you! One of our articles was published in the June 2019 issue of the Software Quality Professional Journal. You can read The …

Seeing Intermittent GitHub Clone Failures on Jenkins? Check Your Repo Size

One of my clients noticed occasional build failures while using Jenkins. It was a strange situation, as their builds would suddenly see a burst of failures with no apparent change. I have been using the same Jenkins setup internally for the past year, and I have never observed such behavior. Their software builds for three …