Indicator Management: A Question That Should Start With “Why?”

Enterprises’ operational functions often need their data on a real-time basis to know about business performance and have an overall assessment of their strategic and tactical goals. Simultaneously, the risk management function wants to define analytical models based on historical data to forecast potential risk materialization. In addition, the audit team wants to analyze its data to perform periodic reviews and establish if business owners are mitigating relevant risk. This is a burden carried by most enterprises worldwide.

In scenarios such as these, data users should begin by asking what should be measured, then they can start defining indicators, data sources, data frequency and type of metrics. However, other concerns often appear, such as demonstrating the benefit and the outcome of long working hours, getting visibility from top management and board members and obtaining proper funding to make professional dreams come true. These concerns will not be adequately addressed unless a different approach is adopted, which is why it is beneficial to explore a different perspective on establishing critical indicators’ purpose based on data analysis while defining a strategy to meet stakeholders’ expectations. Management can make more informed decisions and does not have to rely on static measurements by asking for the “why” behind a particular metric to understand its relevance and inclusion in the structured plan.

As outlined in the methodology proposed in a 2018 ISACA^® Journal article,¹ there are general guidelines used to address this issue based on the following pillars:

• Planning on indicators
• Capturing relevant data
• Defining an indicator’s maturity
• Organizing and presenting the measurement strategy
• Monitoring and refining

Planning on Indicators

There are many examples of indicators,² but in a holistic approach, it is imperative that indicators have a clear purpose for the plan owner and the organization. Whether the indicators main objective is to measure risk (key risk indicators [KRIs]), process performance (key performance indicators [KPIs]) or control operations (key control indicators [KCIs]), every indicator’s purpose must address a defined goal according to its audience, relationship with strategic or tactical plans and life cycle:

• Audience—No indicator will be relevant for all stakeholders within an organization. There must be a set of indicators for each audience because their interests may vary, depending on several factors (e.g., the knowledge they have on the subject, their role within the organization, the visualization method used for the indicators). Furthermore, it is essential to identify which indicator is the actual target for each audience.
• Relationship with plans—Not every indicator will be a showstopper. Sometimes having too many metrics can be misleading because it is challenging to link them to specific objectives or activities. Therefore, the best recommendation is to ensure that indicators have traceability and correlation with causes, effects and benefits. Simultaneously, the thresholds and expected outcomes will help with the definition of the next steps.
• Life cycle—No indicator will be relevant and useful forever. As the only constant is change itself, there must be a strategy that addresses:
  • Related processes or business objectives
  • Timeframe to measure the indicator
  • Desired outcome before changing to an evolved version of the indicator
  • Road map considering its current and desired stages

Capturing Relevant Data

Once an organization determines what indicators it will use based on a strategy, the next step is to identify the data sources for the selected metrics. Besides information systems and databases, the scan must also include shared databases (regardless of their technology, from spreadsheets to evolved data warehouses), middleware, and business intelligence solutions. Although it poses an additional challenge regarding data confidentiality, availability, and integrity (ergo, data security), there must be a proper definition of data flows feeding the indicators to ensure integrity. The organization must document guidelines to maintain such data flows and periodically review them to avoid deviations that may alter alignment with the established strategy. Always remember that long-term data strategies depend on reliable data analysis to ensure well-informed and transparent decisions.

Defining an Indicator’s Maturity

Based on the definitions regarding the previously noted three elements (audience, relationship with plans, life cycle), organizations need to state the gap between the current and future status. With this input, the organization will have clarity on its deficiencies and improvement opportunities. This helps organizations to define associated initiatives and road maps to establish the steps that must be carried out to achieve the desired or expected maturity. At this point, it is fundamental to avoid subjective metrics (e.g., qualitative metrics based on people’s expertise, indicators not supported by data) to ensure that all parties are on the same page. Once the metrics are clear, the path ahead will be straightforward, thus allowing findings prioritization, initiatives differentiation and traceability for the indicators’ strategy.

Organizing and Presenting the Measurement Strategy

The target audience has already been mentioned. Now, it is fundamental to consider the approval party with accountability and sponsorship on the critical indicator strategy. Given the importance of having a clear message for each audience, the strategic owner will have the additional challenge of showing an intricate stratagem in a friendly, easy-going manner. Nevertheless, items such as benefits, conclusions, initiatives and potential impacts of not achieving measurable goals must be highlighted. Traditional tools such as dashboards, Gantt charts and flow diagrams can be useful, but considering innovative drivers such as design thinking and Agile methodologies is a plus.

Monitoring and Refinement

Plainly stated, if the owner cannot measure a plan, the plan may be inadequate. Like any other program, the organization needs to periodically assess the strategy to ensure everything is going according to the initial guidelines. The plan owner must define performance measurement metrics regarding IT governance.³

However, that does not mean that plans cannot change. As previously stated, metrics can deviate, data sources may become unreliable, business owners may lose engagement with indicators, the enterprise’s business model could change, and regulators may issue new obligations. Therefore, constant refinement becomes necessary for a successful indicator strategy.

Conclusion

Defining indicators is relatively easy because it requires only data, analysis, and insights. However, organizations should not consider generating and maintaining indicators as a lightweight duty. Due to the constant evolution enterprises go through, the complex environment of stakeholder interaction and the fast-paced changes in technology and regulations, it is practically impossible to define an indicator that will last forever. Every measure must be part of a strategy that will enable the possibility of determining the following:

• What are the IT processes, business processes or strategic goals related to the indicators?
• What is the measurement frequency for KPIs/KCIs/KRIs and the audience for each?
• How long will the indicators be usable?
• How long is the organization planning to measure a particular set of indicators?
• What is the threshold that will denote when an indicator change is required?
• What are the expectations of the stakeholders using the indicators?

However, none of those questions will lead organizations to better results than knowing why an indicator is measured. Once the organization can clearly state the indicator’s objective and expected outcome, metrics will support any investment, monitor deviations on the most critical risk and decipher how to boost performance, thanks to an intelligent data analysis approach.

Endnotes

¹ Satyanarayana Tammineedi, R.; "Integrating KRIs and KPIs for Effective Technology Risk Management," ISACA^® Journal, vol. 4, 2018
² ISACA^®, Risk Scenarios Using COBIT^® 5 for Risk, 2014
³ Bakshi, S.; "Performance Measurement Metrics for IT Governance," ISACA Journal, vol. 6, 2016

Julian Marquez, CISA, CRISC, CDPSE, COBIT Foundation, ISO 27001 Lead Auditor, ITIL Foundation

Is an experienced risk management professional specializing in digital, IT and data risks. During his 13-year career, he has advised multiple enterprises within the financial services and retail industries, providing consulting services in Brazil, Canada, Chile and Colombia. He has also facilitated training events on internal control, IT risk management, data privacy and security management based on his experience and knowledge of best applicable practices.