Estimating the Value of Ceritifed RTOSes in Embedded Development

Jerry Krasner, Ph.D., MBA
American Technology International, Inc.
www.embeddedforecast.com

About EMF

EMF is the premier market intelligence and advisory firm in the embedded technology industry. Embedded technology refers to the ubiquitous class of products which use some type of processor as a controller. These products include guided missiles, radars, and avionics as well as robots, automobiles, telecom gear, and medical electronics.

Embedded Market Forecasters (EMF) is the market research division of American Technology International, Inc. EMF clients range from startups to Global 100 companies worldwide. Founded by Dr. Jerry Krasner, a recognized authority on electronics markets, product development and channel distribution, EMF is headquartered in Framingham, Mass.

About the Author

Jerry Krasner, Ph.D., MBA is Vice President of Embedded Market Forecasters and its parent company, American Technology International. A recognized authority with over 30 years of embedded industry experience, Dr. Krasner was formerly Chairman of Biomedical Engineering at Boston University, and Chairman of Electrical and Computer Engineering at Wentworth Institute of Technology and Bunker Hill Community College. In addition to his academic appointments, Dr. Krasner served as President of Biocybernetics, Inc. and CLINCO, Inc., Executive Vice President of Plasmedics, Inc. and Clinical Development Corporation, and Director of Medical Sciences for the Carnegie-Mellon Institute of Research. Earlier, he was Senior Engineer at the MIT Instrumentation Laboratory. Dr. Krasner earned BSEE and MSEE degrees from Washington University, a Ph.D. in Medical Physiology / Biophysics from Boston University and an MBA from Nichols College.

Copyright 2010 by American Technology International, Inc, 638 Main St, Ashland, MA 01721 All rights reserved. No part of this book covered by copyright hereon may be reproduced or copied in any manner whatsoever. Every effort has been made to provide accurate data. To the best of the editor’s knowledge, data is reliable and complete, but no warranty is made for this.


Background

EMF conducts annual detailed and comprehensive surveys of embedded developers in a manner so as to maintain statistical accuracy. Between March 2009 and April 2010, EMF gathered data from 1325 developers to look at design outcomes (time-to-market, design completions ahead of or behind schedule, closeness of final design results to pre-design expectations, etc.) and to look at the relative levels of lines-of-code for different application verticals.

This data has also been used to look at project costs and design outcomes measured by comparing final design results to pre-design expectations for the software, tools, merchant boards and processors used by embedded developers.

For many military and aerospace/avionics applications, application software requires adherence to such standards as DO 178B/Level A, ARINC 653, Common Criteria and MILS, among others. Developers are exposed to a myriad of claims and counter claims. Green Hills Software, for example, touts their EAL 6+ certification as an example of their product superiority – but a reading of their certification shows a very limited coverage.

In this report, EMF data is used to determine project costs and design outcomes for several RTOSes certified to DO 178B Level A. This should certainly be of interest to OEMs and systems integrators who make OS and processor choices for their specific applications.

Recently, smaller companies have had their OSes certified under DO 178 B Level A – and it raises the question whether they offer, in addition to their intrinsic capabilities, a comparable return on investment (ROI). In this report we compare project costs and design outcomes for well known certified RTOSes (Integrity, VxWorks and LynxOS) as well as with Micrium uC/OS II, a lesser known RTOS.

EMF’s comprehensive embedded developer surveys and our unique Dashboard tool enable detailed comparisons to be made between competing DO 178B OSes.

Calculating the ROI

The following parameters are used to calculate the ROI:

a) Number of software developers per project
b) Number of months from design start to product shipment
c) Percent of designs completed behind schedule
d) Number of months behind schedule

Multiplying a) and b) provides the total average number of man-months expended in a project. Multiplying a), c) and d) provide the average number of man-months lost to schedule. Adding these calculations provides the total number of project man-months.

Table I presents a side-by-side comparison between the secure OSes from Wind River, Green Hills, LynuxWorks and Micrium. The Industrial Average is for all OSes.

Table I: Comparative ROIs for Certified RTOSes
Ind ave Micrium Integrity VxWorks LynxOS
Devel time Months 13.9 12.5 17.9 16.3 13.7
% behind schedule 47.0% 43.1% 43.0% 51.1% 36.4%
Months behind 3.8 2.8 3.6 3.3 2.5
Ave Delay Months 1.79 1.21 1.55 1.69 0.91
Delay/proj time 12.8% 9.7% 8.6% 10.3% 6.6%
SW Developers/project 14.7 6.1 15.2 14.2 12.3
Average Developer months/project 204.3 76.3 272.1 231.5 168.5
Developer months lost to schedule 26.3 7.4 23.5 23.9 11.2
Total developer months/project 230.6 83.6 295.6 255.4 179.7
At $10K/developer month
Average developer cost/project $2,043,300 $762,500 $2,720,800 $2,314,600 $1,685,100
Average cost to delay $262,542 $73,615 $235,296 $239,455 $111,930
Total developer cost/project $2,305,842 $836,115 $2,956,096 $2,554,055 $1,797,030

This Table was derived from data that was based on the following number of lines of code:

  • Green Hills: 1054 thousand lines of code
  • LynxOS – 840 thousand lines of code
  • Micrium – 829 thousand lines of code
  • VxWorks – 577 thousand lines of code
  • Industry average – 787 thousand lines of code

Design outcomes are another contributor to the total cost of ownership. The EMF survey asks developers to report on how close to their pre-design expectation their final design outcome was for Performance, Systems Functionality and Features & Schedule. EMF believes that acceptable design outcomes are those that are within 30% of pre-design expectation.

Table II presents the survey results for these certified RTOSes:

Table II: Comparative Design Outcomes within 30% of Pre-Design Expectation
Ind Ave Micrium Integrity VxWorks LynxOS
Performance 73.8% 88.8% 72.3% 80.4% 77.8%
Systems Functionality 76.0% 84.6% 83.4% 80.4% 81.5%
Features & Schedule 79.3% 78.8% 70.2% 73.2% 81.4%

Note that all of the certified RTOSes outperformed the industry average.

Summary

It is interesting to note that a lesser known OS compared very favorably compared to the more established DO 178B OSes. In a previous report such non-certified (but frequently used) OSes as ThreadX and Nucleus also produced more favorable results than the better known OSes.

It is clear from the presented analysis that embedded developers looking for a certified RTOS take into account the design particulars of their applications and the compute power required. EMF hopes that this analysis will help developers to look at certified software from a different perspective.

Tags: ,

Questions or Comments?

Have a question or a suggestion for a future article?
Don't hesitate to contact us and let us know!
All comments and ideas are welcome.