AM Tutorial  (8:00 a.m. - Noon)

How to Define and Implement Practical Software Engineering Metrics

by Timothy G. Olson and Jairus Hihn
Intermediate level

Summary: Most organizations struggle with metrics.  Some metrics are easy to collect but are not very useful.  Other metrics are too expensive to collect.  Some organizations collect too many metrics, and then don’t use them effectively.  What is a good metric?  What are the vital few metrics?  This half-day tutorial will describe, “what is a good metric”, provide a baseline of the vital few software engineering metrics, and provide a detailed example of the implementation and evolution of a software engineering metrics system at NASA’s Jet Propulsion Laboratory (JPL).  This tutorial is based on an ASQ best paper entitled, “Successfully Using a Measurement Framework to Rapidly Achieve Measurable Results.”

Abstract: Most organizations struggle with metrics.  Some metrics are easy to collect but are not very useful.  Other metrics are too expensive to collect.  Some organizations collect too many metrics, and then don’t use them effectively.  What is a good metric?  What are the vital few metrics?  This half-day tutorial will describe, “what is a good metric”, provide a baseline of the vital few software engineering metrics, and provide a detailed example of the implementation and evolution of a software engineering metrics system at NASA’s Jet Propulsion Laboratory (JPL). This tutorial is based on an ASQ best paper entitled, “Successfully Using a Measurement Framework to Rapidly Achieve Measurable Results.”
The first session will cover an award winning Lean Measurement Framework.  The Lean Measurement Framework is based upon the popular Goal/Question/Metric (G/Q/M) paradigm, the Juran Quality Trilogy, and the initial core measures recommended by the Software Engineering Institute (SEI).  The G/Q/M Paradigm is applied to the goals of planning, control, and improvement and based on powerful metrics that have a proven track record.  In order to illustrate the power of the Measurement Framework, real examples from industry are used. Four examples will be described to demonstrate how to use the framework to instrument: 1) a process, 2) a project, 3) an organization, and 4) a complex metric.  Finally, the Measurement Framework helps to ensure that all metrics are collected (e.g., on a form) and stored (e.g., in a database).  There will be time for questions and answers.

The second session will cover the JPL measurement program and how we instantiated the Lean Metrics Approach described in the first session. JPL is a Federally Funded Research and Development Center for NASA with around a 1000 software engineers, actively developing and maintaining over 35 million lines of code, in well over 20 languages.  There are several hundred tasks varying from one-two person tasks to large tasks averaging forty people.  The JPL metrics system needs to effectively support this very diverse community.  As part of the measurement program we create models, tools, repositories, processes, handbooks, and training that enables the evolution of a quantitative management culture at JPL. In this session we will share our specific lessons learned in implementing a metrics program in an organization that has evolved from a CMMI Level 1 to level 3 organization over the last seven years, including what worked and what has not worked.  We will describe our measurement and estimation processes, share much of our training material and tips for training software engineers to engage quantitative management.  We will describe our core metrics, along with ways to analyze them and incorporate them into your decision making process.  Finally we will provide externally released versions of two of our key Excel based tools to tutorial participants:

Presenters Bios:

Timothy G. Olson Timothy G. Olson is Founder and President of Lean Solutions institute, Inc. (LSI).  While performing training and consulting, Mr. Olson has helped numerous organizations measurably improve quality, productivity, and performance, save millions of dollars in costs of poor quality, and has helped numerous organizations reach higher Software Engineering Institute (SEI) maturity levels.  Mr. Olson is a leader of applying Lean SolutionsTM (e.g., lean processes, metrics, requirements) to systems and software engineering.  Mr. Olson has been formally trained in Baldrige, Crosby, Deming, Juran, ISO, CMMI® and Six Sigma quality approaches.  Mr. Olson is a Malcom Baldrige National Quality Award (MBNQA) Examiner (2008) and a Juran Institute Associate.  Mr. Olson was a lead-author of a Software Quality Course for the University of Minnesota, and he is currently a senior member of ASQ, and a member of IEEE and NDIA.

Dr. Jairus HihnDr. Jairus Hihn is a Principal Member of the Engineering staff at NASA’s Jet Propulsion Laboratory and is currently the manager for the Software Quality Improvement Projects Measurement Estimation and Analysis Element, which is establishing a laboratory wide software metrics and software estimation program at JPL. M&E’s objective is to enable the emergence of a quantitative software management culture at JPL. He has a Ph.D. in Economics from the University of Maryland. He has been developing estimation models and providing software and mission level cost estimation support to JPL’s Deep Space Network and flight projects since 1988. He has over 60 publications and regularly presents at international management and software conferences. Jairus has extensive experience in simulation and Monte Carlo methods with applications in the areas of decision analysis, institutional change, R&D project selection cost modeling, and process models.