SG 1 Align Measurement and Analysis Activities
Measurement objectives and activities are aligned with identified information needs and objectives. The specific practices covered under this specific goal may be addressed concurrently or in any order:
· When establishing measurement objectives, experts often think ahead about necessary criteria for specifying measures and analysis procedures. They also think concurrently about the constraints imposed by data collection and storage procedures.
· It often is important to specify the essential analyses that will be conducted before attending to details of measurement specification, data collection, or storage.
SP 1.1 Establish Measurement Objectives
Establish and maintain measurement objectives that are derived from identified information needs and objectives.
Measurement objectives document the purposes for which measurement and analysis are done, and specify the kinds of actions that may be taken based on the results of data analyses.
The sources for measurement objectives may be management, technical, project, product, or process implementation needs.
The measurement objectives may be constrained by existing processes, available resources, or other measurement considerations.
Judgments may need to be made about whether the value of the results will be commensurate with the resources devoted to doing the work.
Modifications to identified information needs and objectives may, in turn, be indicated as a consequence of the process and results of measurement and analysis. Sources of information needs and objectives may include the following:
· Project plans
· Monitoring of project performance
· Interviews with managers and others who have information needs
· Established management objectives
· Strategic plans
· Business plans
· Formal requirements or contractual obligations
· Recurring or other troublesome management or technical problems
· Experiences of other projects or organizational entities
· External industry benchmarks
· Process-improvement plans
Typical Work Products
1. Measurement objectives
Subpractices
1. Document information needs and objectives.
Information needs and objectives are documented to allow traceability to subsequent measurement and analysis activities.
2. Prioritize information needs and objectives.
It may be neither possible nor desirable to subject all initially identified information needs to measurement and analysis. Priorities may also need to be set within the limits of available resources.
3. Document, review, and update measurement objectives.
It is important to carefully consider the purposes and intended uses of measurement and analysis.
The measurement objectives are documented, reviewed by management and other relevant stakeholders, and updated as necessary. Doing so enables traceability to subsequent measurement and analysis activities, and helps ensure that the analyses will properly address identified information needs and
objectives.
It is important that users of measurement and analysis results be involved in setting measurement objectives and deciding on plans of action. It may also be appropriate to involve those who provide the measurement data.
4. Provide feedback for refining and clarifying information needs and objectives as necessary.
Identified information needs and objectives may need to be refined and clarified as a result of setting measurement objectives. Initial descriptions of information needs may be unclear or ambiguous. Conflicts may arise between existing needs and objectives. Precise targets on an already existing measure may be
unrealistic.
5. Maintain traceability of the measurement objectives to the identified information needs and objectives.
There must always be a good answer to the question, "Why are we measuring this?"
Of course, the measurement objectives may also change to reflect evolving information needs and objectives.
SP 1.2 Specify Measures
Specify measures to address the measurement objectives.
Measurement objectives are refined into precise, quantifiable measures. Measures may be either "base" or "derived." Data for base measures are obtained by direct measurement. Data for derived measures come
from other data, typically by combining two or more base measures.
Examples of commonly used base measures include the following:
· Estimates and actual measures of work product size (e.g., number of pages)
· Estimates and actual measures of effort and cost (e.g., number of person hours)
· Quality measures (e.g., number of defects, number of defects by severity)
Examples of commonly used derived measures include the following:
· Earned Value
· Schedule Performance Index
· Defect density
· Peer review coverage
· Test or verification coverage
· Reliability measures (e.g., mean time to failure)
· Quality measures (e.g., number of defects by severity/total number of defects)
Derived measures typically are expressed as ratios, composite indices,or other aggregate summary measures. They are often more quantitatively reliable and meaningfully interpretable than the base
measures used to generate them.
Typical Work Products
1. Specifications of base and derived measures
Subpractices
1. Identify candidate measures based on documented measurement objectives.
The measurement objectives are refined into specific measures. The identified candidate measures are categorized and specified by name and unit of measure.
2. Identify existing measures that already address the measurement objectives.
Specifications for measures may already exist, perhaps established for other purposes earlier or elsewhere in the organization.
3. Specify operational definitions for the measures.
Operational definitions are stated in precise and unambiguous terms. They address two important criteria as follows: · Communication: What has been measured, how was it measured, what are the units of measure, and what has been included or excluded?
· Repeatability: Can the measurement be repeated, given the same definition, to get the same results?
4. Prioritize, review, and update measures.
Proposed specifications of the measures are reviewed for their appropriateness with potential end users and other relevant stakeholders. Priorities are set or changed, and specifications of the measures are updated as necessary.
SP 1.3 Specify Data Collection and Storage Procedures
Specify how measurement data will be obtained and stored.
Explicit specification of collection methods helps ensure that the right data are collected properly. It may also aid in further clarifying information needs and measurement objectives.
Proper attention to storage and retrieval procedures helps ensure that data are available and accessible for future use.
Typical Work Products
1. Data collection and storage procedures
2. Data collection tools
Subpractices
1. Identify existing sources of data that are generated from current work products, processes, or transactions.
Existing sources of data may already have been identified when specifying the measures. Appropriate collection mechanisms may exist whether or not pertinent data have already been collected.
2. Identify measures for which data are needed, but are not currently available.
3. Specify how to collect and store the data for each required measure.
Explicit specifications are made of how, where, and when the data will be collected. Procedures for collecting valid data are specified. The data are stored in an accessible manner for analysis, and it is determined whether they will be saved for possible reanalysis or documentation purposes. Questions to be considered typically include the following:
· Have the frequency of collection and the points in the process where measurements will be made been determined?
· Has the time line that is required to move measurement results from the points of collection to repositories, other databases, or end users been established?
· Who is responsible for obtaining the data?
· Who is responsible for data storage, retrieval, and security?
· Have necessary supporting tools been developed or acquired?
4. Create data collection mechanisms and process guidance.
Data collection and storage mechanisms are well integrated with other normal work processes. Data collection mechanisms may include manual or automated forms and templates. Clear, concise guidance on correct procedures is available to those responsible for doing the work. Training is provided as necessary to
clarify the processes necessary for collection of complete and accurate data and to minimize the burden on those who must provide and record the data.
5. Support automatic collection of the data where appropriate and feasible.
Automated support can aid in collecting more complete and accurate data.
Examples of such automated support include the following:
· Timestamped activity logs
· Static or dynamic analyses of artifacts
However, some data cannot be collected without human intervention (e.g.,customer satisfaction or other human judgments), and setting up the necessary infrastructure for other automation may be costly.
6. Prioritize, review, and update data collection and storage procedures.
Proposed procedures are reviewed for their appropriateness and feasibility with those who are responsible for providing, collecting, and storing the data. They also may have useful insights about how to improve existing processes, or be able to suggest other useful measures or analyses.
7. Update measures and measurement objectives as necessary.
Priorities may need to be reset based on the following:
· The importance of the measures
· The amount of effort required to obtain the data
Considerations include whether new forms, tools, or training would be required to obtain the data.
SP 1.4 Specify Analysis Procedures
Specify how measurement data will be analyzed and reported.
Specifying the analysis procedures in advance ensures that appropriate analyses will be conducted and reported to address the documented measurement objectives (and thereby the information needs and
objectives on which they are based). This approach also provides a check that the necessary data will in fact be collected.
Typical Work Products
1. Analysis specification and procedures
2. Data analysis tools
Subpractices
1. Specify and prioritize the analyses that will be conducted and the reports that will be prepared.
Early attention should be paid to the analyses that will be conducted and to the manner in which the results will be reported. These should meet the following criteria:
· The analyses explicitly address the documented measurement objectives
· Presentation of the results is clearly understandable by the audiences to whom the results are addressed
Priorities may have to be set within available resources.
2. Select appropriate data analysis methods and tools.
Issues to be considered typically include the following:
· Choice of visual display and other presentation techniques (e.g., pie charts, bar charts, histograms, radar charts, line graphs, scatter plots, or tables)
· Choice of appropriate descriptive statistics (e.g., arithmetic mean, median, or mode)
· Decisions about statistical sampling criteria when it is impossible or unnecessary
to examine every data element
· Decisions about how to handle analysis in the presence of missing data elements
· Selection of appropriate analysis tools
· The work does not cost more to perform than is justified by the benefits that it provides.
Criteria for evaluating the conduct of the measurement and analysis might include the extent to which the following apply:
· The amount of missing data or the number of flagged inconsistencies is beyond specified thresholds.
· There is selection bias in sampling (e.g., only satisfied end users are surveyed to evaluate end-user satisfaction, or only unsuccessful projects are evaluated to determine overall productivity).
· The measurement data are repeatable (e.g., statistically reliable).
· Statistical assumptions have been satisfied (e.g., about the distribution of data or about appropriate measurement scales).
