Managing Embedded Linux Deployment

Embedded Linux developers face numerous challenges from the perspective of maintaining and deploying products developed with Open Source Software:

Yet, at the core of it, all Embedded Linux products have the same foundational components:

The challenges are numerous:

Mission Critical Approaches

Far too often developers get lost in a sea of multiple choices, each with their advantages, each with their disadvantages, and through this, get “tunnel vision”, focused only on what tools are available today, and how to work with them. This natural tendency will get a product out the door this year, but will cause numerous headaches on a multi-year product cycle.

The most omnipotent approach for any developer is to avoid tunnel vision through building a Open Source Agnostic internal baseline. Through this approach, all of the fears of tomorrow are removed, and the developer is free to focus on what is most important: Getting the product out the door.

This article focuses on how to approach Embedded Linux based software development in a means that will survive a multi-year product cycle that easily incorporates into it the never ending changes in Open Source that surround them, but cannot be avoided in any way.

Figure 1 depicts an approach that incorporates the primary components which are discussed in this article:

Missing

Figure 1 ‒ Primary Approaches

Available Tools and Packages

Before discussing in detail approaches to managing an Embedded Linux project, let's take a look at some of the options for developing a baseline:

Open Embedded

A robust set of tools that have been used by vendors such as MontaVista, NXP, Wind River, Intel and others over the past several years.

Advantages:

Disadvantages:

BuildRoot

A robust set of tools that have been used by many organizations for years, and continues to be extremely viable. However, generally not supported by vendors.

Advantages:

Disadvantages:

Cross Compilers and Tools

Tools like Open Embedded and BuildRoot do allow for a tool chain to be built automatically, and this is a suitable approach. However, it's important to evaluate whether vendor specific optimizations are present in a present release of OE/Yocto or BuildRoot. It may be prudent to see if the vendor provides their own pre-built toolchain, and if so, determine if there is a significant reason for that (such as optimization).

Design for Open Source Changes

Open Source will change: this is inevitable. What is valid today may not be valid in a month. And most certainly, will not be valid over a multi-year cycle, as packages that are available today from upstream feeds may not necessarily be available over a multi-year period.

Key foundational components to consider are these:

Product Development ‒ A Practical Approach

The above sections outlined the issues and concepts from 8,000' ‒ A view from the sky. This following discussions provide a practical set of approaches that will endure a multi-year, multi-platform, multi-customer implementation strategy.

Let's revisit our foundational components, with a little more detail on what matters:

Through tying the above items together, we now have a “product release” in place. Each component depends on the other components, and together, they form a product. That product is good within the framework of the “snapshot” of what is taken today in every single sense: it works. It is manageable. And, it is not tied to any changes in the Open Source community; whether in GIT or SVN, it's a framework.

What holds it together is the product developer developed scripting. That scripting is the orchestra of music, as it makes all of these components come to together. And, it's that scripting that allows us to work with the never ending changes from Open Source, as through this, the product developer maintains full control over what happens.

Figure 2 depicts this structure:

Missing

Figure 2 ‒ A Practical Implementation

Breaking down what is in Figure 2 is as follows:

  1. Root FS Source: This can come from a variety of sources. The application developer need not be concerned, this is provided by an OS Team; over time, this will change, and there will be multiple sources for a Root FS, each applied to a specific customer/product release environment.
  2. Kernel Source: This can come from a variety of sources. A realistic approach to this is that these various Kernels are vendor stabilized, without any developer updates; they are a snapshot, and the number of versions will grow with time.
  3. U-Boot Source: Could be a boot loader from another vendor, but in keeping with mainstream, it's best to use U-Boot. These are vendor supplied releases, without any developer updates. A snapshot, and the number of versions will grow with time.

The additional components in this figure are:

  1. User Applications: These are vendor developed tools. They remain as agnostic to above items as possible, but when require changes, are done through #ifdef tied to a version.
  2. Product Specific Changes: Contain patch sets to Kernel and Boot Loader, as well as updates to Root Filesystem. Generally best to have a generic set of updates which are product developer centric, but allow the option of customization on a “per customer” basis.
  3. Support Tools: A set of scripts, either Python or Shell based, which perform specific functions based on each target build requirement.
  4. Master Coordinator: Basically a script or Makefile that can be invoked to build a particular platform. Typically, it would be provided with a Product and Customer specific build target, and from that, invoke specific scripting that that existed either in the Support Tools environment, or in the Customer Specific environment to carry out the build. A shell per say, as each end customer build should be able to be handled stand-alone.

Summary

Designing a baseline that allows for keeping current with the Open Source Community is a conscious decision, and has the following characteristics:

By adopting this approach, the developer provides a stable means to maintain a product over a multi-year cycle.