Performance Measurement Metrics for IT Governance

During the past 30 years, enterprises have been embracing new methods to transform their operations to use IT and related technology to provide a higher level of customer service. The pace at which enterprises are adopting these new methods is rapid. To handle the speed of this transformation, management relies on technology resources and vendors, resulting in an increased dependency on technology and skilled resources. The pace and dependencies can create a lack of enterprise control; therefore, enterprises use key performance indicators (KPIs) to measure the performance of IT service delivery.

Although many enterprises today conduct return on investment (ROI) analysis of new IT projects and sometimes incorporate the total cost of ownership (TCO) calculation into the business case that they present to the board of directors for approval, only about 25 percent of enterprises conduct ROI analysis after the completion of a project.^{1, 2, 3} However, ROI and TCO are not the only criteria for approving IT projects; they are only two of the many considerations in the decision-making process. A positive ROI does not necessarily mean that the project will be approved. It is a strategic decision that is based on business requirements and stakeholder expectations. Therefore, enterprises should conduct a cost-benefit analysis that may require quantitative and qualitative indicators.

Enterprises that want to effectively monitor the activities of IT so that they are in line with the business goals use KPIs or key measurement metrics. Performance indicators/metrics not only help to monitor achievements compared against goals, but also help to evaluate the effectiveness and efficiency of business processes. Metrics also help enterprises allocate and manage resources. Performance metrics enhance and influence decisions that are related to business such as budgets, priorities, resourcing and activities.

KPI and metrics are essential tools for management that are implemented in all areas of the business. Today, enterprise use of IT and related technology requires huge investments in IT. Therefore, stakeholders are interested in confirming that IT investments are strategically aligned, managed effectively and help the achievement of common business goals. To ensure stakeholder expectations are met, management uses IT governance practices that are defined by the global standard from the International Organization for Standardization (ISO) ISO 38500 and COBIT 5.

IT Governance and Metrics

The IT governance mechanism ensures that stakeholder needs, conditions and options are evaluated to determine balanced, agreed-on enterprise objectives. IT governance also ensures that direction is set through prioritization and decision making and that performance and compliance are monitored against agreed-on direction and objectives. Management plans, builds, runs and monitors activities in alignment with the direction that is set by the governance body to achieve the enterprise objectives.⁴

The IT governance processes are evaluate, direct and monitor (EDM). Metrics are a monitoring mechanism and help management monitor the achievements of the enterprise’s business-related goals and IT-related goals. Appropriate metrics help the governing body provide direction that is based on defined goals and an evaluation of metrics. Metrics help enterprises answer valuable questions, such as:⁵

• Is IT performance better than last year?
• What is the enterprise getting from IT investments?
• How can the enterprise benchmark performance?
• What should the enterprise do in the absence of measureable metrics? Can it use risk management, loss expectancy, attack vectors or correlation?

Metrics describe a quality and require a measurement baseline, e.g., 87 percent of incidents reported were resolved within two hours. These measurements demonstrate workloads and activity. Metrics are useful for evaluating compliance and process effectiveness and measuring success against established objectives. Enterprises expect positive outcomes from IT and IT resources, including skilled human resources. To manage the performance of IT, management is interested in getting the answers to the questions in the first column in figure 1. The second column shows the type of indicators that are required to get the answers to these questions.⁶

Developing Performance Metrics

Developing performance metrics usually follows a process of:

1. Establishing critical processes to meet customer requirements (This helps enterprises with developing manageable metrics.)
2. Identifying specific, quantifiable outputs of work from the identified processes in step 1.
3. Establishing targets against which results can be scored

Developing metrics includes defining a balanced set of performance objectives, metrics, targets and benchmarks. Metrics should cover activities and outcomes that are measured using lead and lag indicators and an appropriate balance of financial and nonfinancial measures. The metrics should be reviewed and agreed on with IT, other business functions and other relevant stakeholders.⁷

Metrics and indicators are based on information received from operations. When this information represents the measurement of performance, it is referred to as means-based metrics. Metrics that are designed to monitor the achievement of objectives are called ends-based metrics. Ends-based metrics may include:

• Changes in an enterprise’s inventory of risk exposure (use risk profile report)
• Comparing defined goals of business growth with investment in IT and establishing relationship

Means-based metrics may include:

• The number of application vulnerabilities over one year
• Percentage of automated teller machine (ATM) downtime during active hours (must be less than two percent of active hours)
• Percentage of incidents that are resolved within the SLA time (including escalated incidents)

The following recommendations should be observed while developing metrics and identifying performance indicators:⁸

• Normalize metrics to a common attribute parameter—To understand trends properly, normalize metrics to a common parameter; for example:
  • Time—Is time defined as per year occurrence, transactions per second/minute/hour, average interval between events, mean time between failures (MTBF)
  • Cost—Is cost per unit or per million?
• Understand the characteristics of a good metric—A good metric allows accurate and detailed comparisons, leads to correct conclusions, is well understood by everyone and has a quantitative basis. A good metric helps to avoid erroneous conclusions. A good metric is linear, reliable, repeatable, easy to use, consistent and independent.
• Avoid comparisons against other similar enterprises—Each enterprise is different and may have different goals and objectives. The exception to this is metrics for benchmarking performance. Develop metrics that focus on specific quantified comparisons of documented operational activities that are responsible for contributing to outcomes. Use routines and ratios and avoid wholesale comparisons of business lines because this comparison is subjective and often broad and qualitative.
• Minimize cost-related comparisons—Limit cost-related metrics to measure only the benefits (value) from IT. A comparison with the industry or other enterprises may not be relevant. The challenge that enterprises face often is quantifying the outcomes for comparison against costs. For example, a service for improving customer satisfaction may cost enterprises; however, quantifying improved satisfaction of customers may not be possible. In such cases, indirect indicators, e.g., repeat/more business opportunities, may be more useful.
  Another problem is common costs, both internal and external, that include allocations for multiple operations or business lines that cannot be segregated for charging to a specific operation or business line.
• Focus on work activities and outcomes—While developing metrics and indicators, the focus should be on processes and activities for generating and providing data, e.g., the number of cash-related transactions on an automated teller machine (ATM) or the percentage of servers that were patched during a month.
• Keep metrics to a manageable quantity—Top management may not be interested in a multipage analytical report or dashboard. Although a large amount of relevant data might be collected during activities, only the most critical metrics should be included in the management report/dashboard.

What Are Good Metrics?

Good metrics generally satisfy the following criteria:⁹

• Consistently measured—Metrics must provide similar analysis over a period of time.
• Easy-to-gather data—The cost of collecting data for metrics should be low, and the data must be collected through routine operational processes. However, this data collection must satisfy the requirement of being contextually specific. Some service metrics for IT may need more effort and, hence, cost to get data, e.g., how many customers could not be serviced due to an ATM that was not working?
• Expressed as numbers, percentage or unit of measure—Numbers and percentages are easy to understand and compare; therefore, as much as possible, metrics should be represented as a number or a percentage.
• Contextually specific—IT metrics must measure the achievement of goals or objectives of business; therefore, the metrics must represent the context.

Types of Indicators and Metrics

The need for metrics and indicators is underlined by many organizations, such as the Information Technology Infrastructure Library (ITIL), ISACA (COBIT 5) and ISO. Although ISO expects a measurement of performance, it does not prescribe any specific indicators. Measurement methods may be defined by organizations.

ITIL defines three types of metrics: technology metrics, process metrics and service metrics. Note that technology and process metrics are also referred to as operational metrics.¹⁰

Technology Metrics
Technology metrics measure specific aspects of the IT infrastructure and equipment, e.g., central processing unit (CPU) utilization of servers, storage space utilized, network status (e.g., speed, bandwidth utilization) and average uptime (availability of technology).

Most technology metrics provide inputs on IT utilization, which is a very small part of service, to the chief information officer (CIO) or data center manager; however, unless this metric is compared with another metric, it may not provide meaningful information for top management. For example, consider a network response of 100 milliseconds, (i.e., a message reaches its destination in 100 milliseconds). If management expects network response to be 10 milliseconds, the response time requires attention, and if management expects network response to be 300 milliseconds, the response time is more than satisfactory.

Process Metrics
Process metrics measure specific aspects of a process, e.g., number of changes that are rolled back within a month, average incident response time in a month, percentage of employees who attended to task on time, average time to complete a process.

Process metrics provide information about the functioning of processes. These metrics are generally used for compliance conformance that is related to internal controls. However, too many process metrics may not serve the purpose of monitoring. Metrics that are related to critical processes may be considered for management reporting.

Service Metrics
The primary focus of ITIL is on providing service. Service metrics are essential metrics for management to monitor. They provide an end-to-end measurement of service performance. Defining service metrics can be difficult due to the intangible nature of service levels. Service metrics are more like assessments about what is already known about a problem and are measured in a way that provides ballpark results.¹¹ When it is difficult to measure the service levels due to associated uncertainty (e.g., unpredictable human behavior) such uncertainty in measuring service levels can be reduced at indicative levels and can be brought within ballpark measurements.

Examples of service-level metrics include the following:

• Results of a customer satisfaction survey indicating how much IT contributes to customer satisfaction
• Cost of executing a transaction (banks use this metric to measure the cost of a transaction that is carried out via different service channels, such as Internet, mobile, ATM and branch)
• Efficiency of service, which is based on the average time to complete a specific service. A service is not just a process; a service can consist of multiple processes.

Many types of metrics are required for a comprehensive understanding of the health of service management throughout the enterprise.

COBIT 5

COBIT 5 is primarily an IT governance framework. Effective governance management must be able to manage risk and meet stakeholder expectations by optimizing resources. COBIT 5 identifies the following seven enablers that help to achieve governance objectives:

• Principals, policies and frameworks
• Processes
• Organizational structures
• Culture, ethics and behavior
• Information (data)
• Services, infrastructure and applications
• People, skills and competencies

Stakeholder expectations help management to arrive at a method for benefits realization, which helps to determine enterprise goals. Because enterprises deploy IT, these goals cascade into IT-related goals, which cascade into enabler goals (see figure 2).¹²

To monitor goal achievement, management uses indicators and metrics. COBIT 5 identifies two types of indicators:

• Lead indicators are activities that predict the achievement of goals. These indicators are not measurable, e.g., implementing global or industry best practices or following the life-cycle approach for resources (enablers).
• Lag indicators are measurable and help measure the achievement of goals. Most metrics are defined for lag indicators.

COBIT 5 identifies three levels of metrics: enterprise goal metrics, IT goal metrics and process goal metrics.¹³

Enterprise Goals and Sample Metrics
COBIT 5 identifies 17 generic enterprise goals that are based on dimensions of a balanced scorecard (BSC). These dimensions are financial, customer, internal, and learning and growth.

Generic metrics for IT goals and process goals are defined in the process description for each COBIT 5 process. The COBIT 5 process reference model identifies 37 IT-related generic processes. Metrics can be defined using COBIT 5. Consider the enterprise goal of customer-oriented service culture. COBIT 5 suggests using the following metrics:

• Number of customer service disruptions due to IT service-related incidents (reliability)
• Percent of business stakeholders satisfied that customer service delivery meets agreed-on levels
• Number of customer complaints
• Trend of customer satisfaction survey results

Depending upon the organization’s customer services offered using IT solutions, the following metrics (which shall be a subset of metrics defined previously) may be considered:

• Impact on customer satisfaction due to service disruptions because of IT-related incidents
• Percent of business stakeholders satisfied that customer service delivery meets agreed-on levels
• Reduction or increase in number of customer complaints related to nonavailability of IT-based services

There are two IT-related goals that primarily map to the enterprise goal of customer-oriented service culture.¹⁴ They are IT-related goals 01, Alignment of IT and Business strategy and 07, Delivery of IT services in line with business requirements. Metrics suggested for IT-related goal 07 from COBIT 5 are (for simplicity, only those IT goals that primarily map to the enterprise goal in the example have been considered):

• Number of business disruptions due to IT service incidents
• Percent of business stakeholders satisfied that IT service delivery meets agreed-on service levels
• Percent of users satisfied with the quality of IT service delivery

Based on business requirements, the following metrics may be considered:

• Number of IT incidents affecting business service
• Percent of IT incidents affecting business service to total IT incidents
• Number of customer complaints related to service delivery due to issues related to IT

COBIT 5 suggests metrics for each process in the process reference model. The next step is to identify the processes that are associated with the IT-related goal 07. Delivery of IT services in line with business requirements and select the metrics for processes. The following processes depend upon this IT goal:

Conclusion

Developing, implementing and monitoring performance measurement metrics is key for implementing monitoring mechanisms for goals and objectives that are set by the IT governance processes. Performance measurement metrics should not be copied from similar enterprises. Every enterprise has unique objectives and, thus, unique metrics. This uniqueness is due to many reasons, including business strategy and objectives, enterprise culture, difference in risk factors, risk assessment results, and geopolitical and economic situations. Enterprises can use generic metrics that are provided by global standards and frameworks such as ITIL and COBIT 5 to define enterprise-specific metrics, which should be mapped to enterprise objectives and goals.

Endnotes

¹ Jeffery, M.; “Return on Investment Analysis for E-business Projects,” Northwestern University, Evanston, Illinois, USA, www.kellogg.northwestern.edu/faculty/jeffery/htm/publication/roiforitprojects.pdf
² Myers, R.; “Measuring the Business Benefit of IT,” CFO.com, 20 October 2004, http://ww2.cfo.com/strategy/2004/10/measuring-the-business-benefit-of-it/
³ Bidgoli, H.; The Internet Encyclopedia, Volume 3, John Wiley & Sons, USA, April 2004
⁴ ISACA, COBIT 5, USA, 2012
⁵ Jaquith, A.; Security Metrics: Replacing Fear, Uncertainty, and Doubt, Addison-Wesley, USA, 2007
⁶ Op cit, ISACA, COBIT 5
⁷ OpsDog, Inc., “What are KPIs & Benchmarks?” 2016, https://opsdog.com/tools/kpis-and-benchmarks
⁸ Ibid.
⁹ Op cit, Jaquith
¹⁰ Scarborough, M.; “Three Types of Metrics Defined by ITIL,” Global Knowledge Training LLC, 12 December 2013, http://blog.globalknowledge.com/PROFESSIONAL-DEVELOPMENT/ITIL/THREE-TYPES-OF-METRICS-DEFINED-BY-ITIL/
¹¹ Hubbard, D.; How to Measure Anything: Finding the Value of Intangibles in Business, John Willey & Sons, USA, 2007
¹² ISACA, COBIT 5: Enabling Processes, USA, 2012
¹³ Ibid.
¹⁴ Ibid.
¹⁵ Ibid.