DO 178B – Redux: Looking at Developer Preferences, Issues and Vendor Cost

DO 178B - Redux: Looking at Developer Preferences, Issues and Vendor Cost

Jerry Krasner, Ph.D., MBA
April 2013

About EMF

EMF is the premier market intelligence and advisory firm in the embedded technology industry. Embedded technology refers to the ubiquitous class of products which use some type of processor as a controller. These products include guided missiles, radars, and avionics as well as robots, automobiles, telecom gear, and medical electronics.

Embedded Market Forecasters (EMF) is the market research division of American Technology International, Inc. EMF clients range from startups to Global 100 companies worldwide. Founded by Dr. Jerry Krasner, a recognized authority on electronics markets, product development and channel distribution, EMF is headquartered in Framingham, Mass.

www.embeddedforecast.com

About the Author

Jerry Krasner, Ph.D., MBA is Vice President of Embedded Market Forecasters and its parent company, American Technology International. A recognized authority with over 30 years of embedded industry experience, Dr. Krasner was formerly Chairman of Biomedical Engineering at Boston University, and Chairman of Electrical and Computer Engineering at Wentworth Institute of Technology and Bunker Hill Community College. In addition to his academic appointments, Dr. Krasner served as President of Biocybernetics, Inc. and CLINCO, Inc., Executive Vice President of Plasmedics, Inc. and Clinical Development Corporation, and Director of Medical Sciences for the Carnegie-Mellon Institute of Research. Earlier, he was Senior Engineer at the MIT Instrumentation Laboratory. Dr. Krasner earned BSEE and MSEE degrees from Washington University, a Ph.D. in Medical Physiology / Biophysics from Boston University and an MBA from Nichols College.

Regarding the Data in this Report

The data that is referred to in this report is statistically accurate and authentic and is based on:

  • A statistically generated comprehensive and detailed survey of embedded developers and managers who reported on their design results (number of developers per project, vertical market of their design, time to market, percent of designs completed behind schedule or cancelled, closeness of final design outcomes to pre-design expectations, testing outcomes, etc.), the tools they used (development, modeling, Java, Eclipse, and other development tools), their choice of OS, IDE, communication middleware, processors used as well as where they go to learn about new products, tools and concepts.
  • An EMF Dashboard – a unique tool that allows the user to simultaneously compare similar products (vendors can do competitive comparative analysis); that marketing executives can use for sales promo and strategic planning; that allows developers beginning a project to compare the experiences of hundreds of fellow developers that undertook similar projects to gain insights before making a commitment; and that allows CFOs and senior managers to look at what tools and processes resulted in the greatest cost savings.

For the interested reader, the following link demonstrates the power of the Dashboard and how we used it in developing the data that is presented herein:

http://www.embeddedforecast.com/EMF_DashboardIntro/EMF_DashboardIntro.html

Executive Summary

Embedded Market Forecasters (EMF) used its 2011 survey of embedded developers (642 respondents) to look at the habits and concerns of DO178B developers and how their activities and designs compared with the broad embedded industry. Using its unique to the industry Embedded Dashboard, comparisons and correlations were undertaken and published in 2012.

In this report, EMF reports on the results of the EMF 2013 Survey of Embedded Developers to compare the current data with that of the 2012 publication.

Among the reported findings in this paper are:

  • The total developer cost per project was about 10% higher for DO178B projects in both the 2012 and 2013 data
  • Modeling-simulation and CMMI use ranked higher among DO178B developers in 2012 then in 2013. In 2013, CMMI and Common Criteria requirements dropped off the list of the top 5 items considered a Best Practice by DO 178B developers
  • Incomplete/vague requirements and insufficient resources continue to be of greater concern to DO 178B developers in 2013 as the most significant issues confronting embedded developers. Design complexity as a significant issue rose from 31% of DO 178 respondents to 51.9% in 2013
  • In 2013, when asked about the most important criteria for their OS selection, “safety certifiable”, “realtime performance”, “prior experience with OS”, and “Linux compatibility” fell out of favor as criteria for OS selection whereas “acquisition cost”, “microprocessor support” and “availability of source code” became the criteria of greatest importance.
  • EMF made Total Project Cost calculations for three of the most recognized OSes (VxWorks Secure, Integrity Secure and LynxOS Secure). Micrium uC/OS-II was selected as a representative of lesser known OSes for purposes of comparison. Micrium has been available for more than a decade and is deployed in 10’s of millions of applications. Interestingly, Micrium again in 2013 had the least cost per project

Background

Eighteen months ago EMF published its findings illustrating the activities of DO178B developers compared with the activities of embedded developers whose designs don’t require certified operating systems. We had read and listened to the many advocates and users of DO 178B based developments. What we didn’t find was data that reflected the concerns, design choices, and experiences of DO178B developers. So in January 2012 we made our findings public.

The information was welcomed by the industry and some of the findings were not expected. EMF was encouraged to revisit the DO178B topic to see if the 2012 findings (based on the extensive 2011 EMF Survey of Embedded Developers) were a one-time look into the industry or if they were repeatable.

Revisiting a topic such as this is always a good idea. The findings from the 2013 EMF Survey of Embedded Developers are presented here. It is interesting to observe what, if anything, has changed during the two year interval.

Introduction

The RTOS marketplace is highly competitive, if not zero sum close to it, and is filled with claims, counter claims and FUD. Vendors will have you believe that a certified OS is superior to one that has not undergone a certification process. Linux, for example, is not certifiable under DO 178B. However billions of devices run with commercial Linux OSes. Certainly, mission critical applications are expected to run at a higher level of certainty – we can’t have airplanes falling out of the sky. The idea that DO 178B certification is necessary for medical patient monitoring, for example, is ridiculous (the highest frequency response required for patient monitoring is 100 Hz) and the attempted use of FUD to scare medical developers into using such has backfired.

The decision to absorb the expense of DO 178B certification is most often a strategic marketing decision. If one doesn’t sell to the military or avionics marketplace there is little need to incur such expense.

EMF conducts annual detailed and comprehensive surveys of embedded developers in a manner so as to maintain statistical accuracy. Between March 2009 and April 2011, EMF gathered data from 1975 developers to look at design outcomes (time-to-market, design completions ahead of or behind schedule, closeness of final design results to pre-design expectations, etc.) and to look at the relative levels of lines-of-code for different application verticals. For the purpose of the 2012 report we relied on the results of the 2011 EMF Embedded Developer Survey (642 respondents). The 2013 survey contained the responses of 595 developers (95% +/- 4.0% accuracy).

For many military and aerospace/avionics applications, application software requires adherence to such standards as DO 178B/Level A, ARINC 653, Common Criteria and MILS, among others. Developers are exposed to a myriad of claims and counter claims.

In this report, EMF data is used to determine project costs for several RTOSes certified to DO 178B Level A. This should certainly be of interest to OEMs and systems integrators who make OS and processor choices for their specific applications. We also
report which issues these developers identify as having the greatest impact on their design efforts.

Recently, smaller companies have had their OSes certified under DO 178 B Level A – and it raises the question whether they offer, in addition to their intrinsic capabilities, a comparable return on investment (ROI). In this report we compare project costs for well known certified RTOSes (Integrity, VxWorks and LynxOS) as well as with Micrium uC/OS II, a lesser known but certified DO 178B RTOS.

Methodology

The data that is referred to in this report is statistically accurate and authentic and is based on:

  • A statistically generated comprehensive and detailed survey of embedded developers and managers who reported on their design results (number of developers per project, vertical market of their design, time to market, percent of designs completed behind schedule or cancelled, closeness of final design outcomes to pre-design expectations, testing outcomes, etc.), the tools they used (development, modeling, Java, Eclipse, and other development tools), their choice of OS, IDE, communication middleware, processors used as well as where they go to learn about new products, tools and concepts.
  • An EMF Dashboard – a unique tool that allows the user to simultaneously compare similar products (vendors can do competitive comparative analysis); that marketing executives can use for sales promo and strategic planning; that allows developers beginning a project to compare the experiences of hundreds of fellow developers that undertook similar projects to gain insights before making a commitment; and that allows CFOs and senior managers to look at what tools and processes resulted in the greatest cost savings. The interested reader can view a Dashboard demo at http://www.embeddedforecast.com/EMF_DashboardIntro/EMF_DashboardIntr o.html
  • Five hundred and ninety-five developers responded to the online survey, of which 51 were hardware engineers, 124 were software engineers, 73 were systems developers, 43 were systems architects, 90 were firmware engineers and 126 were engineering managers. In addition 75 developers gave titles other than these listed. This provided an excellent distribution of experiences and viewpoints from which to draw inferences and conclusions. Statistically, the response is at a 95% confidence level, plus or minus 4.0%.
  • 49.5% of respondents came from North America, while 18.4% were from Asia and 32.1% were from Europe.

Comparing DO 178B Development Activities with Worldwide Embedded Development Activities

Average Cost of Development:

Table I presents the comparative costs of development between DO 178B and other embedded developments.

The EMF Dashboard was filtered to create two cadres that were simultaneously compared, side-by-side. The Dashboard was used to determine (from questions within the survey) the comparative costs using:

a) Number of software developers per project
b) Number of months from design start to product shipment
c) Percent of designs completed behind schedule
d) Number of months behind schedule
e) Percent of designs cancelled
f) Number of months between project start and cancellation

Multiplying a) and b) provides the total average number of man-months expended in a project. Multiplying a), c) and d) provide the average number of man-months lost to schedule. Multiplying a), e), and f) provide the number of man months lost to cancellation. Adding these results provides the total number of project man-months.

Table I: Comparative Costs for Certified (DO 178B) and non-certified RTOSes
2012 Ind ave
All RTOSes
2012 DO 178B
Users
2013 Ind ave
All RTOSes
2013 DO 178B
Users
Devel time Months 13.9 17.9 13.8 17.4
% behind schedule 47.0% 50.0% 38.5% 38.4%
Months behind 3.8 5.7 6.3 3.8
% Cancelled 11.0% 8.9% 9.0% 9.2%
Months before cancellation 4.7 6.3 4.4 5.5
SW Developers/project 14.7 12 12.3 13.3
Average Developer months/project 204.33 214.8 169.7 231.4
Developer months lost to schedule 26.3 34.2 29.8 19.4
Developer months lost to Cancellation 7.6 6.7 4.9 6.7
Total developer months/ project 238.2 255.7 204.4 257.6
At $10,000/developer month
Average developer cost/project $2,381,841 $2,557,284 $2,044,445 $2,314,200
Average cost to delay $338,541 $409,284 $347,045 $261,372
Total developer cost/project $2,720,382 $2,966,568 $2,391,489 $2,575,572

Interestingly, the average project cost between DO 178B developments and non-DO 178B development remained remarkably consistent between the 2011 and 2013 surveys. This adds to the confidence level of the findings. The longer development times for DO 178B projects is not surprising given the nature of mission critical applications for which certified OSes are required (this might be explained for mil/aero developments).

Table II presents Best Practices as perceived by developers for the years 2012-2013. Since the 2012 report was based on 2011 survey data, the time span is actually two years. It is interesting to observe that only 68.4% of DO 178B developers consider it a Best Practice (compared with 14.1% of the industry).

Table II: Processes Considered being a Best Practice
2013 Do 178B Industry
DO-178B/C 68.4% 14.1%
Peer reviews 26.3% 26.8%
V-Model 25.0% 13.7%
Agile Methodologies 23.7% 33.6%
Modeling/simulation 19.7% 16.0%
2012 Do 178B Industry
DO 178B 60.3% 12.9%
Modeling-Simulation 34.5% 22.7%
Peer reviews 34.5% 35.9%
Process Management (CMMI) 22.4% 10.1%
Agile Methodologies 19.0% 26.3%

Given the strict oversight of software development, it is surprising to see that Common Criteria is not ranked high by DO 178B developers as a Best Practice. It makes sense that modeling-simulation would be a benefit to these developers – particularly with the need for code reuse. We’d like to believe that CMMI is a hold over as a bad habit from prior years – it means that you have to employ a consistent process; not necessarily a good process.

Developers were asked to identify the most significant issues impacting their software development. They were given a list of 15 possible responses (they could do a write in as well) and they were limited to a maximum of four choices. In this manner we were able to develop a hierarchy of issues illustrated by the percentage of respondents that chose that option. These comparative responses are presented in Table III.

Table III: Most Significant Issues Confronting Embedded Developers
2013 Do 178B Industry
Incomplete or vague requirements 57.0% 51.4%
Design complexity 51.9% 41.7%
Insufficient resources 35.4% 39.5%
Insufficient time 27.8% 38.4%
Prolonged development cycle 26.6% 24.2%
2012 Do 178B Industry
Incomplete/vague requirements 62.1% 53.4%
Insufficient resources 46.6% 40.9%
Insufficient time 34.5% 44.7%
Design complexity 31.0% 38.5%
Standards compliance 24.1% 13.5%

Clearly DO 178B developers are not as concerned about enabling tools or time constraints. Design complexity was reported in 2013 as a major concern for DO 178B developers and this is consistent with their use of modeling-simulation tools (Table II). Concluding our look at comparative developer characteristics and preferences, Table IV presents the reported criteria that are most important to developers in selecting an operating system.

Table IV: Criteria Most Important in Selecting their Next OS
2013 Do 178B Industry
Acquisition cost 43.9% 44.9%
Microprocessor support 42.5% 35.9%
Availability of source code 40.7% 25.6%
Real time performance 37.1% 44.9%
Compatibility with our development tools 32.9% 21.8%
Reliability 27.3% 26.9%
Safety certifiable 13.8% 44.9%
2012 Do 178B Industry
Safety certifiable 67.9% 15.0%
Realtime performance 51.8% 37.3%
Acquisition cost 35.7% 37.3%
Microprocessor support 23.2% 30.5%
Prior experience with OS 21.4% 14.8%
Availability of source code 16.1% 27.8%
Compatible with Linux 7.1% 20.8%

EMF has no idea why safety certifiable dropped by 50% in importance between 2012 and 2013. Interestingly, cost and microprocessor became more of an issue with DO 178B developers during this time period. Not surprisingly, Linux support appears to be a non-issue with DO 178B developers. We will need to wait for the 2014 survey to see if there is a trend.

Comparing Associated Costs for Comparable DO 178B Operating Systems

For more than a decade, the mission critical and standards-based market has been dominated by Wind River (VxWorks), Green Hills Software (Integrity), and LynuxWorks (LynxOS). Recently there has been competition from such vendors as Micrium, Express Logic and commercial Linux vendors. Of particular interest are the costs associated with projects using alternative OSes for DO 178B required developments and how they compare with VxWorks, Integrity or LynuxWorks OS. We chose to include Micrium as an example of an alternative choice. All four OSes presented are certified to DO 178B/C Level A. The results are presented in Table V and the calculations were made the same as for Table I. This Table was derived from data that was based on the following number of lines of code:

  • Green Hills: 1008 thousand lines of code
  • LynxOS – 717 thousand lines of code
  • Micrium – 579 thousand lines of code
  • VxWorks – 542 thousand lines of code
  • Industry average – 569 thousand lines of code
Table V: Comparative Project Costs for DO 178B/C Operating Systems
Micrium uC/OS GHS Secure LynxOS Secure VxWorks Secure
Devel time Months 12.1 18.9 15.2 16.8
% behind schedule 38.5% 44.5% 41.7% 43.8%
Months behind 2.8 4 4.2 4.4
% Cancelled 12.4% 6.9% 8.0% 5.6%
Months before cancellation 3.7 5.7 7.3 5.1
SW Developers/project 5 16 12.9 14.7
HW Developers/project 3.4 10.7 5.6 4.8
Total Developers/Project 8.4 26.7 18.5 19.5
Average Developer months/project 101.6 504.6 281.2 327.6
Developer months lost to schedule 9.1 47.5 32.4 37.6
Developer months lost to cancellation 3.9 10.5 10.8 5.6
Total Developer months lost cancel/delay 12.9 58.0 43.2 43.1
Total developer months/ project 114.5 562.7 324.4 370.7
At $10,000/developer month
Average developer cost/project $1,145,491 $5,626,571 $3,244,049 $3,707,496
Average cost to delay $129,091 $580,271 $432,049 $431,496
Total developer cost/project $1,274,582 $6,206,842 $3,676,098 $4,138,992

Note that Table V includes the project cost of hardware developers.

Summary

It is interesting to note that a lesser known OS compared very favorably compared to the more established DO 178B OSes. This repeats the findings reported in 2012.

It is clear from the presented analysis that embedded developers looking for a certified RTOS take into account the design particulars of their applications and the compute power required. EMF hopes that this analysis will help developers to look at certified software from a different perspective.

First, choose the RTOS that best meets your application. Such considerations as the following should be first considered:

  • Power requirements
  • Worst case interrupt latency
  • Worst case context-switch times
  • Time-to-market considerations
  • Ease-of-use
  • Footprint
  • Budget

The latency and context-switch times are usually processor specific and can be gotten from the chip vendors. For a better discussion of this topic see Michael Barr’s view at:

http://www.netrino.com/Embedded-Systems/How-To/RTOS-Selection

It is not immediately clear as to why the project cost of the lesser known OSes is significantly less that those of the “big three”. It might have to do with them having been selected specifically for the application at hand, ease of learning a new OS, or other considerations.

What is clear from this year-over-year data is that DO178B developers have a broader range of choices and that switching OSes might be less troublesome than previously thought.

Tags: ,

Questions or Comments?

Have a question or a suggestion for a future article?
Don't hesitate to contact us and let us know!
All comments and ideas are welcome.