DO 178B: Looking at Developer Preferences, Issues and Vendor Cost of Development

Jerry Krasner, Ph.D., MBA
Dolores A. Krasner
January 2012

Copyright 2012 by American Technology International, Inc, 638 Main St., Ashland, MA 01721. All rights reserved. No part of this book covered by copyright hereon may be reproduced or copied in any manner whatsoever. Every effort has been made to provide accurate data. To the best of the editor’s knowledge, data is reliable and complete, but no warranty is made for this.

About EMF

EMF is the premier market intelligence and advisory firm in the embedded technology industry. Embedded technology refers to the ubiquitous class of products which use some type of processor as a controller. These products include guided missiles, radars, and avionics as well as robots, automobiles, telecom gear, and medical electronics.

Embedded Market Forecasters (EMF) is the market research division of American Technology International, Inc. EMF clients range from startups to Global 100 companies worldwide. Founded by Dr. Jerry Krasner, a recognized authority on electronics markets, product development and channel distribution, EMF is headquartered in Framingham, Mass.

www.embeddedforecast.com

About the Authors

Jerry Krasner, Ph.D., MBA is Vice President of Embedded Market Forecasters and its parent company, American Technology International. A recognized authority with over 30 years of embedded industry experience, Dr. Krasner was formerly Chairman of Biomedical Engineering at Boston University, and Chairman of Electrical and Computer Engineering at Wentworth Institute of Technology and Bunker Hill Community College. In addition to his academic appointments, Dr. Krasner served as President of Biocybernetics, Inc. and CLINCO, Inc., Executive Vice President of Plasmedics, Inc. and Clinical Development Corporation, and Director of Medical Sciences for the Carnegie-Mellon Institute of Research. Earlier, he was Senior Engineer at the MIT Instrumentation Laboratory. Dr. Krasner earned BSEE and MSEE degrees from Washington University, a Ph.D. in Medical Physiology / Biophysics from Boston University and an MBA from Nichols College.

Dolores A. Krasner received BS and MEd degrees from Fitchburg State College and Framingham State College with specializations in special education, literacy and educational research. While a full time teacher, she continued her studies with 60 credits beyond her MEd degree in educational research and best practices. Ms. Krasner retired from her full-time educational roles in 2007. Ms. Krasner is currently Senior Editor and Researcher for Embedded Market Forecasters.


Executive Summary

Embedded Market Forecasters (EMF) used its 2011 survey of embedded developers (642 respondents) to look at the habits and concerns of DO178B developers and how their activities and designs compared with the broad embedded industry. Using its unique to the industry Embedded Dashboard, comparisons and correlations were undertaken.

Among the reported findings in this paper are:

  • The total developer cost per project was about 10% higher for DO 178B projects
  • When looking at comparative project and management practices, there was a huge difference in requirements tracking tool use (82.5% for DO 178B developers versus 54.4%)
  • DO 178B developers had a voice in selection of development processes (42.1% versus 28.2%) and the use of cost & time linked management systems to development processes (49.1% versus 38.8%). Developer participation is clearly more a part of the DO 178B management process than that found in industry projects
  • Modeling-simulation and CMMI use are much more prevalent for DO 178B projects whereas Agile methodologies and Common Criteria requirements are less considered a Best Practice by DO 178B developers
  • Incomplete/vague requirements and insufficient resources are of greater concern to DO 178B developers whereas insufficient time and design complexity are of less concern to them
  • When asked about the most important criteria for their OS selection, “safety certifiable” and realtime performance were overwhelming selections for DO 178B developers. Microprocessor support, compatibility with Linux nd availability of source code were of far less concern
  • EMF made Total Project Cost calculations for three of the most recognized OSes (VxWorks Secure, Integrity Secure and LynxOS Secure). Micrium uC/OS-II was selected as a representative of lesser known OSes for purposes of comparison. Micrium has been available for more than a decade and is deployed in 10’s of millions of applications. These cost calculations were also compared to the DO 178B user community at large as well as to the industry average for all RTOSes. Interestingly, Micrium has the least cost per project – in all but one case it had a 50% cost advantage

Background

The RTOS marketplace is highly competitive, if not zero sum close to it, and is filled with claims, counter claims and FUD. Vendors will have you believe that a certified OS is superior to one that has not undergone a certification process. Linux, for example, is not certifiable under DO 178B. However billions of devices run with commercial Linux OSes. Certainly, mission critical applications are expected to run at a higher level of certainty – we can’t have airplanes falling out of the sky. The idea that DO 178B certification is necessary for medical patient monitoring is ridiculous (the highest frequency response is 100 Hz) and the use of FUD to scare medical developers into using such has backfired.

The decision to absorb the expense of DO 178B certification is most often a strategic marketing decision. If one doesn’t sell to the military or avionics marketplace there is no need to incur such expense.

We have read and listened to the many advocates and users of DO 178B based developments. What we haven’t found is data that reflects the concerns, design choices, and experiences of DO178B developers.

EMF conducts annual detailed and comprehensive surveys of embedded developers in a manner so as to maintain statistical accuracy. Between March 2009 and April 2011, EMF gathered data from 1975 developers to look at design outcomes (time-to-market, design completions ahead of or behind schedule, closeness of final design results to pre-design expectations, etc.) and to look at the relative levels of lines-of-code for different application verticals. For the purpose of this report we rely on the results of the 2011 EMF Embedded Developer Survey (642 respondents)

This data has also been used to look at project costs and design outcomes measured by comparing final design results to pre-design expectations for the software, tools, merchant boards and processors used by embedded DO 178B Level A developers.

For many military and aerospace/avionics applications, application software requires adherence to such standards as DO 178B/Level A, ARINC 653, Common Criteria and MILS, among others. Developers are exposed to a myriad of claims and counter claims.

In this report, EMF data is used to determine project costs and design outcomes for several RTOSes certified to DO 178B Level A. This should certainly be of interest to OEMs and systems integrators who make OS and processor choices for their specific applications. We also report which issues these developers identify as having the greatest impact on their design efforts.

Recently, smaller companies have had their OSes certified under DO 178 B Level A – and it raises the question whether they offer, in addition to their intrinsic capabilities, a comparable return on investment (ROI). In this report we compare project costs for well known certified RTOSes (Integrity, VxWorks and LynxOS) as well as with Micrium uC/OS II, a lesser known but certified DO 178B RTOS.

Methodology

The data that is referred to in this report is statistically accurate and authentic and is based on:

  • A statistically generated comprehensive and detailed survey of embedded developers and managers who reported on their design results (number of developers per project, vertical market of their design, time to market, percent of designs completed behind schedule or cancelled, closeness of final design outcomes to pre-design expectations, testing outcomes, etc.), the tools they used (development, modeling, Java, Eclipse, and other development tools), their choice of OS, IDE, communication middleware, processors used as well as where they go to learn about new products, tools and concepts.
  • An EMF Dashboard – a unique tool that allows the user to simultaneously compare similar products (vendors can do competitive comparative analysis); that marketing executives can use for sales promo and strategic planning; that allows developers beginning a project to compare the experiences of hundreds of fellow developers that undertook similar projects to gain insights before making a commitment; and that allows CFOs and senior managers to look at what tools and processes resulted in the greatest cost savings. The interested reader can view a Dashboard demo at http://www.embeddedforecast.com/EMF_DashboardIntro/EMF_DashboardIntro.html
  • Six hundred and forty two developers responded to the online survey, of which 60 were hardware engineers, 204 were software engineers, 61 were systems developers, 61 were systems architects, 55 were firmware engineers and 138 were engineering managers. In addition 63 developers gave titles other than these listed. This provided an excellent distribution of experiences and viewpoints from which to draw inferences and conclusions. Statistically, the response is at a 95% confidence level, plus or minus 4.1%.
  • 72.0% of respondents came from North America, while 8.3% were from Asia and 19.8% were from Europe.

Comparing DO 178B Development Activities with Worldwide Embedded Development Activities

Average Cost of Development

Table I presents the comparative costs of development between DO 178B and embedded developments.

The EMF Dashboard was filtered to create two cross tabs that were simultaneously compared, side-by-side. The Dashboard was interrogated to determine (from questions within the survey) the comparative costs using:

a) Number of software developers per project
b) Number of months from design start to product shipment
c) Percent of designs completed behind schedule
d) Number of months behind schedule
e) Percent of designs cancelled
f) Number of months between project start and cancellation

Multiplying a) and b) provides the total average number of man-months expended in a project. Multiplying a), c) and d) provide the average number of man-months lost to schedule. Multiplying a), e), and f) provide the number of man months lost to cancellation. Adding these calculations provides the total number of project man-months.

Table I: Comparative Costs for Certified (DO 178B) and non-certified RTOSes
Ind ave DO 178B
All RTOSes Users
Devel time Months 13.9 17.9
% behind schedule 47.0% 50.0%
Months behind 3.8 5.7
% Cancelled 11.0% 8.9%
Months before cancellation 4.7 6.3
SW Developers/project 14.7 12.0
Average Developer months/project 204.3 214.8
Developer months lost to schedule 26.3 34.2
Developer months lost to Cancellation 7.6 6.7
Total developer months/project 238.2 255.7
At $10K/developer month
Average developer cost/project $2,043,300 $2,148,000
Average cost to delay $338,541 $409,284
Total developer cost/project $2,381,841 $2,557,284

Interestingly, DO 178B developments had fewer cancellations (this might be explained for mil/aero developments), and longer project times. Surprisingly DO 178B developments used fewer developers per project.

Table II presents project management practices that are most used by DO 178B developers and how these practices are reflected in industry responses (percent of respondents choosing the practice).

Table II: Comparative Project & Management Practices
Do 178B Industry
Requirements tracking tools 82.5% 54.4%
Budgets based on both management and engineering input 56.1% 50.9%
Cost & Time linked management systems to development progress 49.1% 38.8%
Developers have a voice in selection of development processes 42.1% 28.2%
Management practices assigned to the right technical talent 38.6% 34.2%

Table III presents Best Practices as perceived by developers. It is interesting to observe that only 60.3% of DO 178B developers consider it a Best Practice (compared with 12.0% of the industry). Peer reviews and Modeling-Simulation based development were the 2nd/3rd considered Best Practice.

Table III: Processes Considered being a Best Practice
Do 178B Industry
DO 178B 60.3% 12.9%
Modeling-Simulation 34.5% 22.7%
Peer reviews 34.5% 35.9%
Process Management (CMMI) 22.4% 10.1%
Agile Methodologies 19.0% 26.3%
Common Criteria 1.7% 5.2%

Given the strict oversight of software development it is surprising to see that Common Criteria is dismissed by DO 178B developers as a Best Practice. It makes sense that modeling-simulation would be a benefit to these developers – particularly with the need for code reuse. We’d like to believe that CMMI is a hold over as a bad habit from prior years – it means that you have to employ a consistent process; not necessarily a good process.

Developers were asked to identify the most significant issues impacting their software development. They were given a list of 15 possible responses (they could do a write in as well) and they were limited to a maximum of four choices. In this manner we were able to develop a hierarchy of issues illustrated by the percentage of respondents that chose that option. The comparative responses are presented in Table IV.

Table IV: Most Significant Issues Confronting Embedded Developers
Do 178B Industry
Incomplete/vague requirements 62.1% 53.4%
Insufficient resources 46.6% 40.9%
Insufficient time 34.5% 44.7%
Design complexity 31.0% 38.5%
Standards compliance 24.1% 13.5%
Lack of enabling tools 3.4% 11.5%

Clearly DO 178B developers are not concerned about enabling tools or time constraints. Design complexity is not as great a concern for DO 178B developers and this is consistent with their use of modeling-simulation tools (Table III).

Concluding our look at comparative developer characteristics and preferences, Table V presents the reported criteria that is most important to developers in selecting an operating system.

Table V: Criteria Most Important in Selecting their Next OS
Do 178B Industry
Safety certifiable 67.9% 15.0%
Realtime performance 51.8% 37.3%
Acquisition cost 35.7% 37.3%
Prior experience with OS 21.4% 14.8%
Microprocessor support 23.2% 30.5%
Compatible with Linux 7.1% 20.8%
Availability of source code 16.1% 27.8%

Interestingly, cost is not a differentiating factor whereas microprocessor support is less of an issue with DO 178B developers. Not surprisingly, Linux support is a non-issue with DO 178B developers.

Comparing Associated Costs for Comparable DO 178B Operating Systems

For more than a decade, the mission critical and standards-based market has been dominated by Wind River (VxWorks), Green Hills Software (Integrity), and LynuxWorks (LynxOS). Recently there has been competition from such vendors as Micrium, Express Logic and commercial Linux vendors.

The question that hasn’t been answered is costs associated with projects using alternative OSes for DO 178B required developments compare with VxWorks, Integrity or LynuxWorks OS. We chose to include Micrium as an example of an alternative choice. All four OSes presented are certified to DO 178B Level A

The results are presented in Table VI and the calculations were made the same as for Table I. This Table was derived from data that was based on the following number of lines of code:

  • Green Hills: 1054 thousand lines of code
  • LynxOS – 840 thousand lines of code
  • Micrium – 829 thousand lines of code
  • VxWorks – 577 thousand lines of code
  • Industry average – 787 thousand lines of code
Table VI: Comparative Project Costs for DO 178B Operating Systems
Ind ave DO 178B Integrity VxWorks LynxOS Micrium
All RTOSes Users Secure Secure Secure uC/OS-II
Devel time Months 13.9 17.9 12.4 12.2 16.9 11.3
% behind schedule 47.0% 50.0% 35.7% 45.9% 31.0% 41.2%
Months behind 3.8 5.7 6.1 5.2 5.0 3.5
% Cancelled 11.0% 8.9% 12.8% 11.4% 12.0% 10.6%
Months before cancellation 4.7 6.3 7.1 4.6 4.5 3.4
SW Developers/project 14.7 12.0 14.8 12.2 12.4 8.5
Average Developer months/project 204.3 214.8 183.5 148.8 209.6 96.1
Developer months lost to schedule 26.3 34.2 32.2 29.1 19.2 12.3
Developer months lost to Cancellation 7.6 6.7 13.5 6.4 6.7 3.1
Total developer months/ project 238.2 255.7 229.2 184.4 235.5 111.4
At $10K/developer month
Average developer cost/project $2,043,300 $2,148,000 $1,835,200 $1,488,400 $2,095,600 $960,500
Average cost to delay $338,541 $409,284 $456,802 $355,166 $259,160 $153,204
Total developer cost/project $2,381,841 $2,557,284 $2,292,002 $1,843,566 $2,354,760 $1,113,704

Note that all of the certified RTOSes presented in Figure VI outperformed the industry average.

Summary

It is interesting to note that a lesser known OS compared very favorably compared to the more established DO 178B OSes.

It is clear from the presented analysis that embedded developers looking for a certified RTOS take into account the design particulars of their applications and the compute power required. EMF hopes that this analysis will help developers to look at certified software from a different perspective.

First, choose the RTOS that best meets your application. Such considerations as the following should be first considered:

  • Power requirements
  • Worst case interrupt latency
  • Worst case context-switch times
  • Time-to-market considerations
  • Ease-of-use
  • Footprint
  • Budget

The latency and context-switch times are usually processor specific and can be gotten from the chip vendors. For a better discussion of this topic see Michael Barr’s view at:

http://www.netrino.com/Embedded-Systems/How-To/RTOS-Selection

It is not immediately clear as to why the project cost of the lesser known OSes is significantly less that those of the “big three”. It might have to do with them having been selected specifically for the application at hand, ease of learning a new OS, or other considerations.

What is clear is that DO178B developers have a broader range of choices and that switching OSes might be less troublesome than previously thought.

Tags: ,

Questions or Comments?

Have a question or a suggestion for a future article?
Don't hesitate to contact us and let us know!
All comments and ideas are welcome.