Feb 09, 2022 Linux

From Prototype to Post Deployment: Linux Decision Points

By Brandy Goolsby


We have all, at least once in our lives, made decisions that led to unintended consequences — sometimes they can be the kind that creates financial and quality risks. We recently hosted an interesting webinar that discussed the decision process an organization might go through when deciding whether to build a product based on Linux and open source. You can find the video for that webinar here.

The decision process starts with prototyping, then moves on to definition and development. Next come production, deployment, maintenance, and end-of-life. In the embedded industry, the product lifecycle can last 5, 10, or even 15 years or more.

If your project makes it past prototype and gets funded, your Linux solution will also pass through multiple decision points. These decisions will impact speed, quality, and resources. They can also directly impact future scalability, profitability, and the overall success of your project. Some decisions will create a debt you may have to pay off or a need to revisit releases down the road. Specifically, I am talking about technical debt, which the webinar discusses in some detail.

What brings us to this decision process in the first place? Usually, it is because we want to create an innovative product or deliver a needed service. Typically, we don’t start with, “I’m going to build a product based on Linux”! Rather, Linux and open source in general are components in the software platform we use to achieve our aim. We must source our Linux from somewhere, and this is one of the first important decisions to make during the project. The Wind River webinar used a poll to see how audience members sourced their Linux, and the results were a bit surprising. There were four options given:

1)     Choose roll-your-own based on Yocto Project Linux, Debian, or something similar: 27 %

2)     Choose Linux provided through a semiconductor or board vendor: 17 %

3)     Choose Linux from a commercial open source provider: 10 %

4)     Continue evaluating whether you want to use Linux: 45 %

It is interesting that the number of respondents choosing Linux from a commercial OS provider is not higher than 10%. Additionally, given recent trends around the migration to Linux for embedded development, it is surprising that 45% of the audience was still evaluating whether to use Linux at all. (If you are evaluating, check out “OS Options for Edge Use Cases (RTOS vs Linux)”.  Many  who have been in the open source community for a long time might have expected that number to be much lower. However, this may be due to  the webinar audience consisting of primarily  embedded software developers, not enterprise or web application developers. The embedded world has not yet embraced Linux to the extent that enterprise and internet/web developers have, because many embedded use cases require features and functionality — safety-critical certifications, for example — that are not yet practical with Linux.

Back to the decision-making process: Let’s say you’ve sourced your Linux from one of the choices above and you’re beginning the definition and development phase. Soon you will have to make another design decision: whether you’ll use the Linux distribution “as is” or you’ll make modifications to the distribution. More often than not, this decision will determine how much technical debt you’ll carry.

First, let’s define technical debt. According to the Linux Foundation, “Technical debt refers to the cost of maintaining source code that was caused by a deviation from the main branch where joint [community] development happens.” Essentially, this means that if you do not use a given release of Linux as is, then you will end up creating patches or modifications called “forks.” And most likely you will have to carry the cost of maintaining those forks for several releases, if not for the entire life of your product. You may have valid reasons for doing this, reasons that seem worth the risk and commitment. The list below includes some of the more common reasons for carrying technical debt:

Time-to-release: You need to release on a fixed schedule, and you don’t align on a given release of Linux. It is faster to do it yourselves; you will come back and align with open source later.

●  Innovation: Your development team believes they have a better way to handle a particular function than what the open source community has provided. You don’t want to take the time to upstream your feature(s) and reconcile with open source.

Competitive features: You want to add a unique feature to your product to differentiate, and you have to make patches to open source to support that functionality. You do not upstream the patches to open source, either because the community will not accept these changes or because you don’t want to upstream but prefer to keep the patches unique to your product.

Policy: Some organizations have strict formal policies regarding contributing to open source, and thus you have internal reasons for not upstreaming and aligning with the community.

Skills/knowledge: Your organization may not yet have the skills or knowledge to upstream, or it may not have the needed relationship with the open source community to shepherd the code through the process.

Any of these reasons can seem more than valid at the time a design decision has to be made. And in reality, it is nearly impossible not to carry some level of technical debt in a complex, competitive embedded device. Ultimately, your decision might come down to deciding to use a Linux platform that is “good enough” versus an ideal version that has all the functionality and time-to-market attributes you would desire.

Where the pain occurs is when you start to develop the next release(s) of your product. You will want to use the “latest, greatest” Linux release — but you have all those patches, all that technical debt, from your previous release. So now you have engineers working on forward-porting and testing patches rather than working on innovation. The disadvantages pile up as more versions of the product are released. I once worked on a project that involved more than 3,000 technical debt patches that accumulated over three releases of the product. It took approximately 30 engineers almost a year to eventually remove and/or replace the technical debt and get aligned with the main branch of open source.

Delays and higher expenses are only part of the cost you incur as the technical debt adds up. Less measurable but perhaps equally as important is the cost your organization may pay in community standing. Open source communities frown on forks. Your organization might also suffer from poor morale among engineers, if they’re spending all their time forward-porting debt. And finally, it may become more challenging to hire quality talent over the long haul, especially when you must find engineers who have to look forensically at the code to determine where changes were made.

An organization should typically use “good enough” open source solutions. If you do need to differentiate your Linux or do something differently to support a driver or a feature, then engage with the open source community and do whatever is necessary to push your patches into the main branch.

Despite all your efforts to stay on a straight path, you might still find yourself with technical debt based on those in-the-moment decisions to accelerate time-to-market. You will need to treat this code just like the rest of your product: You will have to engage in long-term support, or perform a CVE vulnerability audit, or include the patches in your IP compliance documents. All of this comes at a cost that continues to compound with each release.

Should you find yourself in this situation, however, know that there are organizations that can help. Wind River offers a comprehensive suite of support and integration services to enhance the speed and quality of a market-differentiated Linux solution for deployment — regardless of which Linux you chose. Whether you sourced your Linux from the Yocto Project, from a board vendor, or from another commercial vendor, Wind River has deep expertise in Linux that can help with the delivery of a commercial Linux solution.

Additional Resources

Previous StarlingX R6.0 is here!
Next Trailblazing the GROW Bootcamp