IS Audit Basics: Auditors and Large Software Projects, Part 2: Can Auditors Prevent Project Failure?

This three-part series within the IS Audit Basics column continues a discussion on a matter that, over many years, has been of concern to business managers everywhere: What causes large software projects to have significant cost and timescale overruns and/or fail to meet expectations or, in a worst case, be abandoned before completion?

Part 1 explored just three areas that can cause pain as a project is approved and launched: the business case, the project risk analysis and the requirements definition. In this second part, three other items are explored: buy vs. build, the project plan and the project manager. It includes some lessons learned and provides hints on how auditors can help their organizations.

The Choice: Buying vs. Building a Custom System

There is a large portfolio of business systems available for purchase or for licensing in the cloud. Some can be used without modification when the user organization is able to adapt to the features of the system, thus avoiding all software development, while others require customization, which becomes, in itself, a software development project.

A license for such systems can be purchased, e.g., for an enterprise resource planning (ERP) application. An application of this kind often requires customization to meet an organization’s way of working. As the extent of customization grows, so does the complexity and risk of the project, which can extend for several years.

Designing a custom system, i.e., one of a kind, to meet a particular business purpose is a more complex proposition. The scope of doing this is vastly larger than licensing and customizing a commercial product. Still, such projects are carried out to significantly change the working model of an organization. There are many examples of successful projects, such as the applications developed by large Internet companies (e.g., Google, Yahoo, Amazon, Salesforce).

However, many others have failed or been abandoned, such as the Connecting for Health system attempted by the UK National Health Service, mentioned in Part 1 of this series. Articles published over many years about software failures suggest that success in such projects is the exception rather than the rule.

When an organization does not have the resources to undertake such projects, the usual course of action is to procure them by using individual contractors and managing their work internally or to contract out the whole project to a specialized firm, either locally or internationally.

In theory, the latter appears to be a sound approach. The specialized firm is supposed to have the required expertise and, therefore, produce reliable estimates of costs and timescales. It is also expected to take responsibility for finding and managing the skilled and experienced human resources for the task and replacing them should they become unavailable.

This approach can fall down when management insists on awarding the contract to the lowest bidder regardless whether they have a track record of success in comparable projects. When the firm under consideration is based in another country (or continent), issues of cross-cultural communications, collaboration and jurisdiction may become elements to consider.

The auditor should work with the project sponsor, the project manager, and procurement and legal officers to determine whether the negotiations leading to a contract take the previously mentioned elements into consideration and ensure that any changes initiated by the vendor, the supplier or the client are evaluated, approved through an appropriate change control mechanism (to be discussed in part 3 of this series, which will publish in the ISACA Journal, volume 4, 2016) and recorded.

The Project Plan

A quotation attributed to both Benjamin Franklin¹ and Winston Churchill declares, “If you are failing to plan, you are planning to fail.” This remains true today.

Planning a large project is not a trivial task because it requires a detailed breakdown of the tasks to be accomplished, the dependencies between them (tasks that cannot start until another task has been completed), estimating the duration of each task and the accuracy of such estimates, and the identification of a critical path (the shortest time in which the project can be completed), among other things.

A person who has not participated in a large project before and thus has no experience in designing a project plan is at a disadvantage and risks missing key tasks and/or underestimating their size and complexity.

Anyway, a project plan is certain to require adjustments as the project develops to reflect the inevitable changes in scope or requirements that will emerge and the likely slippage of the completion dates for whatever reason. Without access to a high-quality crystal ball to tell the future, the project is unlikely to develop as initially planned.

A large database project that was audited a couple of years ago was supported by a very elementary project plan consisting of fewer 100 activities, of which only two were labeled “critical.” Optimism is great and, in large projects, dangerous.

The auditor should examine the evolving project plan and how changes are reflected in it as well as seek evidence of how the changes were assessed and approved prior to their implementation.

The Project Manager

This individual has a complex and critical role that requires communication with numerous stakeholders, monitoring progress, and dealing with unexpected and unplanned situations, facilitating the work of all those working on the project by removing administrative obstacles, distractions and unnecessary meetings.

People with a suitable profile are in short supply. Wise management will seek them out and recognize that tasking them with this responsibility will make them unavailable for whatever they were doing before.

Less wise management may prefer an easier option and appoint someone who not only does not have that profile, but also has never managed a complex project. The lesson to be learned from doing this is that experience is the best teacher, but also the most expensive.

A scenario that I personally experienced occurred when the project champion was also the project manager. This person’s background was in personnel administration. This person was unique in many ways: He had a dominating personality and was temperamental while also having good political sense and knowledge of using salami tactics² to increase the project budget by 400 percent over 10 years. This did not help the project.

This resulted in somewhat fluid requirements leading to time delays and cost increases. In the end, the system, as delivered, was disappointing, but that is another story. The replacement system now under development risks having the same outcome.

The auditor should find robust evidence of the risk of such situations and report accordingly. This will not make the auditor many friends, but it is part of what auditors get paid to do.

Conclusions

The auditor is not responsible for fixing projects going bad. Their role is to provide independent and objective findings and observations to management, possibly including recommendations for actions they should consider and take.

Having said this, the auditors should also make management aware that there are some facts of life surrounding projects:

1. All projects have calculated risk factors, and risk assessments should be taken seriously.
2. No project ever develops as planned as there are too many factors outside the control of the project team.
3. Estimates are rarely robust as they are based on assumptions that may be incorrect and predictions that are not reliable.
4. Every project has an “impossible region”³ for the time of its completion and adding resources to shorten timescales is almost certain to fail.
5. The larger the project (in cost and duration), the more likely that there will be unwelcome surprises.
6. By the time a large project is completed, the world for which it had been designed may have changed.

The next column in this series will examine the audit requirements of change control.

Endnotes

¹ GoodReads, www.goodreads.com/quotes/460142-if-you-fail-to-plan-you-are-planning-to-fail
² Atherton, Tony; “Negotiating Skills: The Salami Tactic,” Atherton Training Consultants Ltd.
³ Brooks, Fredrick; The Mythical Man Month, Addison Wesley, 1975, reissued 1995

Ed Gelbstein, Ph.D., 1940 – 2015, worked in IS/IT in the private and public sectors in various countries for more than 50 years. Gelbstein did analog and digital development in the 1960s, incorporated digital computers in the control systems for continuous process in the late ‘60s and early ‘70s, and managed projects of increasing size and complexity until the early 1990s. In the 1990s, he became an executive at the preprivatized British Railways and then the United Nations global computing and data communications provider. Following his (semi) retirement from the UN, he joined the audit teams of the UN Board of Auditors and the French National Audit Office. Gelbstein also taught postgraduate courses on business management of information systems.