Because the income statement factors in depreciation, but depreciation is not an out-of-pocket expense. For instance, if revenue of $10,000 is reduced to $7,000 of income because of a $3,000 depreciation deduction, you still have the use of the full $10,000. So, the cash flow figure of $10,000 is the more instructive one to look at.
Introduction
This document provides guidelines for examining the I.R.C. § 41 credit for increasing research activities (“research credit”) claimed relative to software development. Specifically, this document provides guidelines on applying the process of experimentation test of I.R.C. § 41(d)(1)(C) and selected exclusions from qualified research of I.R.C. § 41(d)(4) to software development activities, whether for internal-use or for commercial sale, lease or license.[i] These guidelines are not an official pronouncement of the law and cannot be used, cited, or relied upon as such.
If a decision is made to examine a taxpayer’s software development activities for purposes of the research credit, these guidelines will aid in risk analysis and will help focus limited audit resources by ranking software development activities at lowest to highest risk of not constituting qualified research under I.R.C. § 41(d). These guidelines, however, do not address all potential research credit issues, some of which may be more material and less burdensome to examine than an I.R.C. § 41(d) activity evaluation. Accordingly, examiners should also refer to the Research Credit Audit Techniques Guide.
Nothing in these guidelines precludes an examiner from proposing any proper adjustment, even if the adjustment arises from an activity that may not be identified in these guidelines as being at greatest risk of not constituting qualified research under I.R.C. § 41(d).
Summary of Recommended Audit Procedures
To assist in focusing upon those software development activities at greatest risk of not constituting qualified research under I.R.C. § 41(d), activities are identified, analyzed, and rated as “high risk”, “moderate risk”, or “low risk”. For the purpose of these guidelines, risk is defined as follows:
- High risk means that these activities usually fail to constitute qualified research under I.R.C. § 41(d).
- Moderate risk means that these activities often fail to constitute qualified research under I.R.C. § 41(d).
- Low risk means that these activities rarely fail to constitute qualified research under I.R.C. § 41(d).
Law
1. Process of Experimentation
I.R.C. § 41(d)(1) provides that in order to constitute qualified research, substantially all of the activities of the research must constitute elements of a process of experimentation related to a new or improved function, performance, or reliability or quality. The legislative history explains that the term process of experimentation means “a process involving the evaluation of more than one alternative designed to achieve a result where the means of achieving that result is uncertain at the outset.” H.R. Conf. Rep. No. 99-841, at II-71, 72 (1986). In addition, a process of experimentation may involve developing one or more hypotheses, testing and analyzing those hypotheses (through, for example, modeling or simulation), and refining or discarding the hypotheses as part of a design process to develop the overall business component. Id.
Final regulations issued in January 2004 address the application of the process of experimentation test. The rule is as follows:
I.R.C. § 41(d)(1) provides that in order to constitute qualified research, substantially all of the activities of the research must constitute elements of a process of experimentation related to a new or improved function, performance, or reliability or quality. The legislative history explains that the term process of experimentation means “a process involving the evaluation of more than one alternative designed to achieve a result where the means of achieving that result is uncertain at the outset.” H.R. Conf. Rep. No. 99-841, at II-71, 72 (1986). In addition, a process of experimentation may involve developing one or more hypotheses, testing and analyzing those hypotheses (through, for example, modeling or simulation), and refining or discarding the hypotheses as part of a design process to develop the overall business component. Id.
Final regulations issued in January 2004 address the application of the process of experimentation test. The rule is as follows:
A process of experimentation must fundamentally rely on the principles of the physical or biological sciences, engineering, or computer science and involves the identification of uncertainty concerning the development or improvement of a business component, the identification of one or more alternatives intended to eliminate that uncertainty, and the identification and the conduct of a process of evaluating the alternatives (through, for example, modeling, simulation, or a systematic trial and error methodology). A process of experimentation must be an evaluative process and generally should be capable of evaluating more than one alternative. A taxpayer may undertake a process of experimentation if there is no uncertainty concerning the taxpayer’s capability or method of achieving the desired result so long as the appropriate design of the desired result is uncertain as of the beginning of the taxpayer’s research activities. Uncertainty concerning the development or improvement of the business component (e.g., its appropriate design) does not establish that all activities undertaken to achieve that new or improved business component constitute a process of experimentation.
Treas. Reg. § 1.41-4(a)(5). Notably, there is no statement either in the regulatory rule (or in the examples applying the rule) that any activity, including software development, per se constitutes a process of experimentation. To the contrary, the preamble to these final regulations states as follows:
The final regulations state that the mere existence of uncertainty regarding the development or improvement of a business component does not indicate that all of a taxpayer's activities undertaken to achieve that new or improved business component constitute a process of experimentation, even if the taxpayer, in fact, does achieve the new or improved business component. The Treasury Department and the IRS believe that the inclusion of a separate process of experimentation requirement in the statute makes this proposition clear. However, the Treasury Department and the IRS have included this clarification in the final regulations out of concern that taxpayers have not been giving sufficient weight to the requirement that a taxpayer engage in a process designed to evaluate one or more alternatives to achieve a result where the capability or the method of achieving that result, or the appropriate design of that result, is uncertain as of the beginning of the taxpayer’s research activities. In particular, this clarification is intended to indicate that merely demonstrating that uncertainty has been eliminated (e.g., the achievement of the appropriate design of a business component when such design was uncertain as of the beginning of a taxpayer's activities) is insufficient to satisfy the process of experimentation requirement. A taxpayer bears the burden of demonstrating that its research activities additionally satisfy the process of experimentation requirement.
T.D. 9104 (January 2, 2004) (emphasis added).
2. Selected Exclusions from Qualified Research
There are certain research activities that are specifically excluded from qualified research under I.R.C. § 41(d)(4). The following activities are not qualified research:
I. Exclusion for Research after Commercial Production
I.R.C. § 41(d)(4) states that qualified research does not include any research conducted after the beginning of commercial production. A business component is considered ready for commercial production when it is developed to the point where it is ready for use or meets the basic functional and economic requirements of the taxpayer.
I.R.C. § 41(d)(4) states that qualified research does not include any research conducted after the beginning of commercial production. A business component is considered ready for commercial production when it is developed to the point where it is ready for use or meets the basic functional and economic requirements of the taxpayer.
The following activities are deemed to occur after the commencement of commercial production:
a) Preproduction planning for a finished business component;
b) Tooling up for production;
c) Trial production runs;
d) Trouble shooting involving detecting faults in production equipment or processes;
e) Accumulating data relating to production processes; and
f) Debugging flaws in a business component.
Treas. Reg. § 1.41-4(c)(10), Examples 1 and 2, illustrate the application of the exclusion for research after commercial production.
a) Preproduction planning for a finished business component;
b) Tooling up for production;
c) Trial production runs;
d) Trouble shooting involving detecting faults in production equipment or processes;
e) Accumulating data relating to production processes; and
f) Debugging flaws in a business component.
Treas. Reg. § 1.41-4(c)(10), Examples 1 and 2, illustrate the application of the exclusion for research after commercial production.
II. Exclusion for Adaptation
This exclusion applies if the taxpayer's activities relate to adapting an existing business component to a particular customer's requirement or need. This exclusion does not apply merely because a business component is intended for a specific customer. A contractor’s adaptation of an existing business component to a taxpayer’s particular requirement or need is not qualified research.
Treas. Reg. § 1.41-4(c)(10), Examples 3 and 7, illustrate the application of the adaptation exclusion.
III. Exclusion for Duplication
This exclusion applies if the taxpayer reproduced an existing business component, in whole or in part, from a physical examination of the business component, plans, blueprints, detailed specifications, or publicly available information with respect to such component. This exclusion does not apply merely because the taxpayer evaluates another's business component in the course of developing its own business component.
Treas. Reg. § 1.41-4(c)(10), Example 8, illustrates the application of the duplication exclusion.
IV. Exclusion for Surveys, Studies, Research Relating to Management Functions
The following activities are excluded under this provision:
(a) Efficiency surveys;
(b) Management functions or techniques, including such items as preparation of financial data and analysis, development of employee training programs and management organization plans, and management based changes in production processes (such as rearranging work stations on an assembly line);
(c) Market research, testing, or development (including advertising or promotions);
(d) Routine data collections; or
(e) Routine or ordinary testing or inspections for quality control.
Treas. Reg. § 1.41-4(c)(10), Example 9, illustrates the application of this exclusion.
(a) Efficiency surveys;
(b) Management functions or techniques, including such items as preparation of financial data and analysis, development of employee training programs and management organization plans, and management based changes in production processes (such as rearranging work stations on an assembly line);
(c) Market research, testing, or development (including advertising or promotions);
(d) Routine data collections; or
(e) Routine or ordinary testing or inspections for quality control.
Treas. Reg. § 1.41-4(c)(10), Example 9, illustrates the application of this exclusion.
V. Exclusion for Research in the Social Sciences, etc.
Qualified research does not include research in the social sciences (including economics, business management, and behavioral sciences, arts, or humanities).
Treas. Reg. §1.41-4(c)(10), Example 10, illustrates the application of this exclusion.
Software Development - Overview
An overview of the software development process is helpful in determining whether a process of experimentation, as defined in the Code and Treasury Regulations, is present. Software development generally involves a cycle of requirements specification, design, coding, testing, performance tuning, product release, maintenance, and bug fixing. Based upon the results of any or all of these steps, the software development activity may need to repeat (i.e., iterate through) one or more of these steps. For example, assume a product development cycle was in its testing phase and a new requirement came in that for competitive reasons had to be incorporated into the product before it could be released. Then, the new requirement would have to be designed into the product, which in turn could affect the existing design and coding of software already written and tested.
All software development projects follow a development methodology, whether structured or informal. Some common methodologies include:[ii]
Waterfall;
Iterative;
RAD (Rapid Application Development);
SEI (Software Engineering Institute);
ISO 9000;
Extreme Programming; and
Object-Oriented.
Iterative;
RAD (Rapid Application Development);
SEI (Software Engineering Institute);
ISO 9000;
Extreme Programming; and
Object-Oriented.
There are, in general, many different ways a given piece of software can be designed and implemented. For example, if one were to ask ten different software engineers to write a payroll program, one would likely end up with ten completely different programs.
Software, unlike tangible business components, does not “wear out” in the traditional sense, so it tends to last years longer than other products. For example, manufacturers may build a tangible product for a short period of time, and then replace it with an entirely new model (e.g., cars, TVs, cell phones). However, core software often tends to be used for years, if not decades.
Software requirements often change during the development process. The requirements, and/or business rules, specified for a piece of software are rarely, if ever, complete at the beginning of the process, and often conflict with each other. Moreover, the requirements can, and often do, change throughout the software development activity. New or changing requirements may necessitate major or minor redesign efforts in order to incorporate the new or changed requirements. It is commonplace in software development to work with incomplete, imprecise requirements, and/or changing requirements throughout the life cycle of a software development project.
As software design is inherently iterative, it follows that in order to incorporate new, modified, or deleted requirements, the design must also change.
Technology, especially software technology, is constantly changing. Technologies that are new one year may become commonplace in subsequent years. It is important to be aware of the timeframe of the software development activity in question and the technologies being utilized. Software developers that utilize newer technologies (e.g., currently, the Internet, JAVA, XML, Web Services, grids, etc.), like any individual learning a new language, may need to spend time learning those technologies. This learning process is commonplace in software development.
Software projects fail for many reasons. Interestingly, the most common reasons that software projects fail relate to business or project management issues, not the failure to eliminate uncertainty concerning the development or improvement of the software in question. These reasons are: [iii]
Incomplete and/or changing requirements;
Lack of user involvement;
Lack of resources (i.e., budget or staff);
Unrealistic expectations;
Lack of executive support;
Lack of planning;
“Didn’t need it any longer”;
Lack of IT management; and
Technological illiteracy.
Lack of user involvement;
Lack of resources (i.e., budget or staff);
Unrealistic expectations;
Lack of executive support;
Lack of planning;
“Didn’t need it any longer”;
Lack of IT management; and
Technological illiteracy.
Software Development – Common Activities
This section provides a list of examples of software development activities that are common to the development or improvement of a software business component:
- Project planning and project management.
Project planning and project management activities typically involve obtaining estimates of the resource requirements, developing and tracking the schedule and budget, and acquiring necessary resources (e.g., people, hardware, software, money, etc.). - Developing user requirements that define the software’s functionality.
This activity is typically done by the “functional” areas within the company (e.g., financial analysts, marketing, human resources, etc.) and not by software developers. These activities specify the features desired in the new software, but tend not to address any software development uncertainties. - Developing specifications, such as functional, design or test specifications.
- Estimating resource requirements.
This activity involves estimating the resource needs (e.g., staff, budget, timeframe) based upon the functionality requirements of the project. If the resource needs exceed management expectations (e.g., the project costs too much), then an iterative process of re-estimating the resources (e.g., adding staff to shorten the schedule, or dropping requirements to reduce cost, etc.) generally occurs until management approves the resources for the project. - Programming.
These activities involve the actual writing of the software programs that comprise the new or improved business component. - Tuning and benchmarking of software.
These activities involve running benchmark programs and comparing their performance and answers (i.e., program results) to the business component that was developed or improved to determine how the new or improved business component performs. These activities typically involve running existing software programs, building databases with known data in order to be able to compare results, and/or using software tools to determine where the performance bottlenecks exist in the software. - Performing software maintenance and debugging.
Software maintenance typically consists of debugging a problem, reviewing, analyzing, and understanding existing program code. Debugging may uncover errors in requirements, programming errors, misunderstandings in interfaces, unexpected system behavior, deficiencies, or defects in vendor or subcontractor products, and integration errors, etc. - Improving performance and/or scalability of an application.
- Quality assurance (QA)/Testing (module, systems, integration, alpha, beta, performance, stress, regression, user acceptance, and manual testing, etc.) occurs throughout the development and maintenance processes. Beta and subsequent testing typically is done against the entire software business component, not just the portion of the software that was modified.
Processes of Experimentation in Software Development
The preamble to the final regulations states that –
“The final regulations do not provide detailed guidance as to how the regulatory provisions are to be applied to a given factual situation. Rather, the Treasury Department and the IRS have concluded that the application of these provisions will depend on the specific activities being claimed by a taxpayer as qualified research, the nature of the taxpayer’s business and industry, and the uncertainties being addressed by the taxpayer’s research activities.”
T.D. 9104 (January 2, 2004)
Accordingly, some preliminary discussion of the nature of uncertainty in software development is in order. Uncertainty in this context is the software development uncertainty that has been identified concerning a functional aspect of a software business component.
In software development, as with the development of tangible business components, there is a distinction between a software development uncertainty, that is resolved through a process of experimentation, and a software development uncertainty that is resolved by other means. For example, a taxpayer may have to configure a software application, and may be uncertain about which configuration choices to make. This uncertainty, in and of itself, does not indicate that the taxpayer subsequently engaged in a process of experimentation to eliminate the configuration uncertainty. The activities undertaken to eliminate the configuration uncertainty are determinative.
In addition to software development uncertainties, there are other types of uncertainties, namely business and project uncertainties. Business uncertainties could, for example, be whether or not potential customers will react favorably to the new product, and/or whether or not the product will be competitive. Project uncertainties could be whether or not the existing staff is adequately trained to use a technology, and/or whether the project can be completed within a given schedule and budget. Such uncertainties do not meet the requirements of I.R.C. § 41(d).
Software Development Risk Categories
There are many different categories of software development, such as, but not limited to, the initial development of software products and new product application areas, feature extensions and enhancements to existing products, combining existing products to create a new product or product suite, and creating new versions of existing products by removing features (e.g., to provide a more entry-level product).
The facts and circumstances of each case must be taken into account in making a determination as to whether or not a taxpayer’s activities constitute elements of a process of experimentation under the Code and the Treasury Regulations. The following list is intended to provide assistance in identifying categories of software development at a low risk, moderate risk, and high risk of not being qualified research under I.R.C. § 41(d).
High Risk Categories of Software Development - means that these activities usually fail to constitute qualified research under I.R.C. § 41(d).
- Maintenance of existing software applications/products.
Software applications generally last for years, if not decades. Thus, once a vendor releases a new product, subsequent minor releases (e.g., release 1.1, 1.2, 2.4, 3.2, etc.) usually involve corrective maintenance to debug, diagnose, and fix design, logic, and/or programming errors, adapt the product to externally imposed changes, such as regulatory matters or competitive developments, and perfect it to match features in competitive products. Critical fix or patch releases may occur more frequently, such as to fix software bugs or security issues that cannot wait for the next minor or major release of the software. Perfective enhancements include modifications and improvements to the software to extend the product’s useful life (e.g., improve the product’s ease of use, performance, responsiveness, space requirements, etc.). Corrective, adaptive, and perfective maintenance generally utilize the same existing architecture and technologies of the released product.[iv] Such maintenance efforts are needed to keep the product operational and competitive in the marketplace. These maintenance activities typically involve analysis, debugging, reverse engineering, making program modifications, and testing of the modifications that were made. Such maintenance activities, in general, are not directed at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science. - Software application configuration.
Applications purchased from outside vendors frequently have to be configured to meet the specific needs and requirements of the taxpayer. For example, configuration activities may involve defining a chart of accounts, defining the number of users, setting access privileges to various functions or reports, etc. This process may involve selecting among vendor defined options, and/or modifying vendor specified default capabilities (e.g., which users can access the payroll report) using vendor provided software. Such configuration activities, in general, are not directed at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science. - Reverse engineering.
Reverse engineering activities typically consist of examining existing programs, databases, and files in order to figure out how an existing application really works. For example, if a new application was supposed to interface with an existing legacy application, then the software engineers might need to reverse engineer the data interface supported by the legacy application. Reverse engineering is not qualified research under I.R.C. § 41(d). See Treas. Reg. § 1.41-4(c)(10), Example 8. - Performing studies, or similar activities, to select vendor products.
For example, choosing between database management systems (like Oracle or DB2), choosing between computer manufacturers (e.g., H-P, IBM, or Dell), choosing between enterprise resource program suppliers (e.g., SAP, Oracle, or PeopleSoft), or choosing between web portal platforms (e.g., WebLogic or Web Sphere) are generally routine due diligence product selection studies. Such studies are generally not directed at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science. See Treas. Reg. § 1.41-4(c)(10), Example 9. - Detecting flaws and bugs in software.
Encountering flaws and bugs in software, including vendor software, usually occurs during debugging and routine testing of the software. These debugging and testing activities are generally not directed at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives, which fundamentally relies on the principles of computer science, but instead are activities directed toward the verification and validation that the software was programmed as intended and works correctly. See I.R.C. § 41(d)(4)(A); Treas. Reg. § 1.41-4(c)(2). - Modifying an existing software business component to make use of new or existing standards or devices, or to be compliant (i.e., certified, validated, etc.) with another vendor’s product or platform.
Activities associated with modifying software to use new devices or standards generally involve an examination of the existing software to locate where these changes need to be inserted into the software. In order to locate where these changes should be inserted, activities such as reviewing program code and/or documentation, reverse engineering, debugging, and routine testing are typically performed. In order to be compliant with another vendor’s product or platform, the activities usually involve running a target vendor’s validation test suite, examining the results, and then, through a process of debugging and reverse engineering, resolving any problems that the certification test suite encountered. These activities are generally not directed at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives, which fundamentally relies on the principles of computer science, but are instead directed at verifying that the software works according to the standards or devices, or successfully passes another vendor’s certification tests. - Developing a business component that is substantially similar in technology, functionality and features to the capabilities already in existence at other companies.
These activities usually involve an analysis of the products already in the marketplace in order to determine, for example, functions and features, what the user interface screens look like, and what must be done in order to develop a similar business component. In addition, since these other competitive products are already on the market, the technologies, the training to learn these technologies, and the available software engineering skills to develop a similar business component, are generally available in the marketplace. The development of these business components generally does not involve activities directed at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science. - Upgrading to newer versions of hardware or software, or installing vendor fix releases.
These activities involve installing new releases of software, or installing new hardware, each of which typically involves following the vendor’s installation instructions. There are occasions in which an installation or upgrade fails. However, the resultant activities would then generally be directed at debugging the installation and/or talking with the vendor to determine what went wrong, but not at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science. - Re-hosting or porting an application to a new hardware (e.g., from mainframe to PC) or software platform (e.g., Windows to UNIX), or rewriting an existing application in a new language (e.g., rewriting a COBOL mainframe application in C++).
Re-hosting or porting an application to a new hardware platform generally involves the adaptation of the existing software business component to the new hardware or software platform, not resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally rely on the principles of computer science. Note that such an adaptation of an existing business component to a new hardware platform is not qualified research under I.R.C. § 41(d).
Rewriting an application in a new language generally involves reverse engineering the existing code to determine all of the requirements and processing algorithms that the rewritten software must support, and possibly dividing the existing large components into smaller components. There is generally no new application architecture, and the original program code is typically converted into the new language constructs. There are generally no software development uncertainties, resolved through a process of experimentation, associated with breaking a large program into smaller components, or in converting program code written in one language into another language. - Writing hardware device drivers to support new hardware (e.g., disks, scanners, printers, modems, etc.).
The vendors of new hardware devices generally provide documentation as to what the capabilities and functions of the devices are, as well as a description of the commands that must be programmed in order to make the devices work. For example, a hard disk vendor would provide the software commands to read data from and write data to a hard disk drive. The software activities are then generally directed at writing software to use these documented device interfaces, not at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science. - Data quality, data cleansing, and data consistency activities.
These activities include such activities as designing and implementing software to validate and/or clean data fields, and/or make the data fields consistent across databases and applications. For example, an address field could be defined differently in every database, or when transferring data between systems, such as between a mainframe and a UNIX server, the format of the data may need to be translated, such as from EBCDIC to ASCII. There is generally no software development uncertainty, resolved through a process of experimentation, with respect to reading data from a file or database, converting that data to a different format (e.g., mm/dd/yy converted to mm/dd/yyyy), or writing the newly formatted data into a file or database. The decisions related to what format to select for any given piece of data, such as a date or an address, are business uncertainties, not software development uncertainties. - Bundling existing individual software products into product “suites” (e.g., combining existing word processor, spreadsheet, and slide presentation software applications into a single “suite”).
Software suites usually involve the development or improvement of a common user interface to allow users to invoke each of the existing individual applications in the bundled suite, or to select configuration or runtime options for the programs in the suite. The products themselves, including any configuration and runtime options, are generally not modified in this process. In addition, the interfaces to the individual applications are not generally uncertain, as the creation or modification of a user interface to invoke a software application is based upon information available to the taxpayer. The activities associated with the bundling of software applications into a suite of products are not generally directed at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science. - Expanding product lines by purchasing other products.
Commercial software vendors may also purchase software products from other vendors, and then may need to develop software to migrate customers from one product to another in order to consolidate the vendor’s overall product offerings, and/or integrate the products so that they work properly together. For example, a database company could purchase another company which had financial software products, and then modify the financial software to work only with the acquiring company’s new database product.
Modifying a purchased software product to access a different database, for example, means that you have to review the product’s program code to find all of the database references, and then change each of them to reference the new database. These activities require reading the documentation for both products, perhaps reverse engineering the purchased product to extract more information about the data items, and then making the appropriate software modifications. Just as in Y2K work, these activities are directed at finding all of the location in the software than need to be modified, and are not generally directed at resolving software development uncertainties.
If the vendor decides to migrate existing customers off of the purchased products and onto the vendor’s current products, then the vendor must develop software to convert the customer’s current data into the vendor’s data structures. This involves understanding the purchased product’s data requirements, as well as an understanding of their own data requirements. Once these requirements are known, which may require reverse engineering activities, it is a straightforward process to write software to read the old data, convert that data into the new data formats, and then to write that data into new data structures (i.e., files or databases). There is generally no software development uncertainty associated with reading or writing data, or in converting data items from one format to another, that is resolved by identifying and conducting a process, designed to evaluate alternatives, which fundamentally relies on the principles of computer science.
- Interfaces.
There are many ways in which a software business component can interface with other software components and/or users. For example, one type of interface development includes designing and implementing electronic interfaces between software applications, wherein one software application can “talk” to another software application and exchange data, or execute a business transaction. A software business component might involve interfacing with one or more other software applications, whether internal (e.g., an inventory software application might “talk” to a warehouse software application), or external (e.g., a business-to-business software interaction) to the taxpayer.
A software business component might also interface with a user, such as via an Internet browser. For example, a taxpayer could have an application that runs on a PC, and then decide to add an Internet browser interface to essentially the same application (e.g., one could have a PC application to calculate your taxes and print your tax forms, or have an Internet browser interface to a similar application to do your taxes).
Interface software development activities involve defining the data and transaction requirements that must be supported on each end of the interface, which may also involve reverse engineering of the interfaces. These activities are generally directed at defining the requirements of the interface, and writing the software accordingly, but not at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science.
- Y2K Program Changes.
Y2K activities typically involved the examination of application program code in order to identify when two-character fields were used to represent year (e.g., ‘89), and once identified, these date fields had to be modified to hold a four character year (e.g., 1989). Y2K activities also involved the modification of program code to correctly make date calculations when using this larger date field. Once all of these changes were made, extensive testing had to be done in order to ensure that the Y2K compliant software produced the same answers as the existing, non-Y2K compliant, application. There is generally no software development uncertainty associated with reading existing program code to locate code and data that may need to be modified, or in testing an existing application to verify that it still works correctly. See Rev. Proc. 97-50, 1997-2 C.B. 525. - Vendor Product Extensions.
This software is typically written by customers to fill gaps in the functionality of a product. For example, one could develop software to perform some calculation not provided by a spreadsheet product, or develop software to customize the format of a report. In order to develop these product extensions, vendors typically provide well-defined interfaces (a/k/a user exits), and/or APIs (Application Programming Interfaces), to permit new code to interact with the vendor’s code. Herein, the vendor’s product is not modified, and the new user code must conform to the vendor’s defined interface or API. These activities of determining which interface or API to use, and then implementing the modifications, are generally not directed at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science. - Graphical User Interfaces (“GUI”).
The design of the graphical user interface screens generally involves issues such as determining what data items and graphics should be displayed on each screen, and where on each screen each item should be displayed. These are business and/or project uncertainties. However, there is typically no software development uncertainty, resolved through a process of experimentation, with respect to the appropriate design of the graphical user interface, or in the capability or method of developing the actual graphical user interface screens.
Moderate Risk Categories of Software – means that these activities often fail to constitute qualified research under I.R.C. § 41(d).
- Functional enhancements to existing software applications/products.
Software applications tend to last for years, if not decades. Thus, once a vendor releases a new product, subsequent major releases (e.g., release 2.0, 3.0, “product name 2004”, “product name 2005”, etc.) of the software typically include new functional enhancements (i.e., product features or capabilities), within the same existing architecture and technologies, and built using the same tools, proven during the development of the original software application. However, particular software development uncertainties that need to be resolved through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science are more likely to arise if, for example, the underlying architecture or technologies also need to be modified.
Some major releases may contain features and functions that require software development activities which are generally not directed at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science. For example, a new version of an income tax product may contain updated income tax tables and modified tax forms, or an anti-virus product may contain modified screens to permit a user to choose when to automatically run an anti-virus scan.
Alternatively, major releases may contain features and functions that involve software development activities that are directed at resolving software development uncertainties through a process of experimentation by identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science. For example, a new version of an income tax product may involve the development or modification of data encryption algorithms to protect the privacy and integrity of the electronic submission of tax returns, or a new version of an anti-virus product may involve the development or modification of heuristic algorithms to detect new incoming computer viruses.
- Software developed as an embedded application, such as in cell phones, automobiles, airplanes.
Embedded software applications generally utilize the software already included in the system in order to execute the new application. In addition, embedded applications tend to be developed using existing architectures, languages, tools, and methodologies. Once developed, the application is then downloaded using established tools to the target environment, such as a cell phone.
For example, particular software development uncertainties, such as very small screen sizes, software that must be failsafe, or a requirement to develop a new communication protocol, may need to be resolved through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science.
Alternatively, embedded software to display information on a user’s screen, such as caller ID names and phone numbers displayed on a cell phone screen, or to produce a report, such as the time and speed a car was going when an accident occurred, generally do not involve activities which are directed at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science.
- The development of software utility programs, such as debuggers, backup systems, performance analyzers, data recovery, etc.
Utility software applications generally utilize software interfaces, or application programming interfaces (APIs) that already exist in the system. Typical activities may involve understanding the capabilities of the underlying system, including the existing APIs, devices, file systems, and/or databases, that may be needed as part of the business component. For example, to develop a business component to backup data on a computer generally involves an understanding of the file system on the computer (i.e., how to find the location of a file on a hard disk, and how to retrieve the entire file, etc.), the devices to which the backup files will be written (e.g., hard disk, tape, floppy disk, CD, etc.), and the underlying application programming interfaces provided by the operating system (e.g., Microsoft Windows, IBM’s MVS, UNIX, etc.). Such software development does not generally involve activities which are directed at resolving software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science.
Alternatively, for example, a software utility program designed to recover data from a hard disk, even though the file was previously deleted and its disk space reused by other applications, may involve design uncertainties which can only be resolved through identifying and conducting a process designed to evaluate alternatives, which fundamentally relies on the principles of computer science.
- Changing from a product based on one technology to a product based on a different or newer technology (e.g., switching from a hierarchical database technology to a relational database technology).
The functions and features of the old technology and new technology products tend to be similar, if not identical. Activities generally involve a comparison of the features and requirements in the current system versus the capabilities provided by the newer technology. In addition, there is generally an analysis of the current design to understand how the existing system works, followed by the subsequent development of a design that incorporates the newer technology. For example, the activities related to extracting data from a hierarchical database, converting that data into a new format, and then loading that data into a relational database, generally do not involve software uncertainties which can only be resolved through identifying and conducting a process designed to evaluate alternatives, which fundamentally relies on the principles of computer science.
Alternatively, changing a technology upon which an existing product was originally based may involve changing the underlying architecture of the software in order to accommodate the new technology. For example, changing an application that had run on a mainframe, to an application that runs in a grid computing environment may involve software development uncertainties that need to be resolved through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science.
- The adaptation and commercialization of a technology developed by a consortium or an open software group.
A vendor may, for example, join a consortium to jointly define a software model for how to perform electronic transactions with the suppliers to that industry (e.g., business-to-business transactions). Each vendor in the consortium may then commercialize parts, or all, of the model. Such models defined by a consortium typically specify in detail the requirements of the model which the software must then satisfy. A vendor can then proceed to design, develop, and test the software which satisfies the model. While the models define the requirements, they usually, for example, do not define the underlying architecture of the software that must conform to the model, and thus there may be particular software development uncertainties which can only be resolved through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science.
Low Risk Categories of Software – means that these activities rarely fail to constitute qualified research under I.R.C. § 41(d).
- The initial release of an application software product may require new constructs, such as, for example, new architectures, new algorithms, or new database management techniques. The design of these new constructs often requires a process of experimentation to resolve specific software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science.
- The development of system software, such as operating systems and compilers, frequently entails the resolution of software development uncertainties related to, for example, process scheduling and memory management designs, and instruction execution optimization. Such software development uncertainties are typically resolved through identifying and conducting a process designed to evaluate technical alternatives which fundamentally relies on the principles of computer science.
- The development of specialized technologies, such as image processing, artificial intelligence, or speech recognition, often requires a process of experimentation to resolve software development uncertainties through identifying and conducting a process designed to evaluate alternatives which fundamentally relies on the principles of computer science.
- Software developed as part of a hardware product wherein the software interacts directly with that hardware in order to make the hardware/software package function as a unit.
The activities involved in developing a hardware/software product generally include evaluating the designs and tradeoffs among the hardware and software components by running, for example, performance benchmarks on the combined product to measure operations per minute, graphical display times, or data throughput. The hardware and software designs will generally be modified together in order to develop the final hardware/software business component. This process of concomitantly modifying the hardware and the software in order to develop the hardware/software product frequently entails the identification and conducting of a process designed to evaluate alternatives which fundamentally relies on the principles of computer science.
Conclusion
The intent of this document is to provide guidelines for examining software development activities by identifying categories of software development that are at a low risk, moderate risk, and high risk of not constituting qualified research under I.R.C. § 41(d). Thus, these guidelines can aid in risk analysis and can help focus limited audit resources by ranking software development activities at lowest to highest risk of not constituting qualified research.
[i] These guidelines do not, however, address the I.R.C. § 41(d)(4)(E) internal-use software exclusion and the applicable exceptions. For guidance on this exclusion, see Advance Notice of Proposed Rulemaking, Credit for Increasing Research Activities (REG-153656-03), 69 F.R. 43 (January 2, 2004). Furthermore, these guidelines do not address several other exclusions that might apply to the taxpayer’s activities. For further guidance, refer to the Research Credit Audit Techniques Guide.
[ii] For more information, an Internet search for “xxx software development”, such as “iterative software development”, will provide more information on these and other software development methodologies.
[iii] Case Study Conclusions, CHAOS, Charting the Seas of Information Technology, The Standish Group, 1994, page 11.
[iv] Section 3.1, Definitions, IEEE Standard for Software Maintenance, Software Engineering Standards Committee of the IEEE Computer Society, June 25, 1998, pages 3 & 4.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
INTRODUCTION
Ensuring that adequate and timely risk identification is performed is the responsibility of the owner, as the owner is the first participant in the project. The sooner risks are identified, the sooner plans can be made to mitigate or manage them. Assigning the risk identification process to a contractor or an individual member of the project staff is rarely successful and may be considered a way to achieve the appearance of risk identification without actually doing it.
It is important, however, that all project management personnel receive specific training in risk management methodology. This training should cover not only risk analysis techniques but also the managerial skills needed to interpret risk assessments. Because the owner may lack the specific expertise and experience to identify all the risks of a project without assistance, it is the responsibility of DOE’s project directors to ensure that all significant risks are identified by the integrated project team (IPT). The actual identification of risks may be carried out by the owner’s representatives, by contractors, and by internal and external consultants or advisors. The risk identification function should not be left to chance but should be explicitly covered in a number of project documents:
- Statement of work (SOW),
- Work breakdown structure (WBS),
- Budget,
- Schedule,
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
- Acquisition plan, and
- Execution plan.
There are a number of methods in use for risk identification. Comprehensive databases of the events on past projects are very helpful; however, this knowledge frequently lies buried in people’s minds, and access to it involves brainstorming sessions by the project team or a significant subset of it. In addition to technical expertise and experience, personal contacts and group dynamics are keys to successful risk identification.
Project team participation and face-to-face interaction are needed to encourage open communication and trust, which are essential to effective risk identification; without them, team members will be reluctant to raise their risk concerns in an open forum. While smaller, specialized groups can perform risk assessment and risk analysis, effective, ongoing risk identification requires input from the entire project team and from others outside it. Risk identification is one reason early activation of the IPT is essential to project success.
The risk identification process on a project is typically one of brainstorming, and the usual rules of brainstorming apply:
- The full project team should be actively involved.
- Potential risks should be identified by all members of the project team.
- No criticism of any suggestion is permitted.
- Any potential risk identified by anyone should be recorded, regardless of whether other members of the group consider it to be significant.
- All potential risks identified by brainstorming should be documented and followed up by the IPT.
The objective of risk identification is to identify all possible risks, not to eliminate risks from consideration or to develop solutions for mitigating risks—those functions are carried out during the risk assessment and risk mitigation steps. Some of the documentation and materials that should be used in risk identification as they become available include these:
- Sponsor mission, objectives, and strategy; and project goals to achieve this strategy,
- SOW,
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
- Project justification and cost-effectiveness (project benefits, present worth, rate of return, etc.),
- WBS,
- Project performance specifications and technical specifications,
- Project schedule and milestones,
- Project financing plan,
- Project procurement plan,
- Project execution plan,
- Project benefits projection,
- Project cost estimate,
- Project environmental impact statement,
- Regulations and congressional reports that may affect the project,
- News articles about how the project is viewed by regulators, politicians, and the public, and
- Historical safety performance.
The risk identification process needs to be repeated as these sources of information change and new information becomes available.
There are many ways to approach risk identification. Two possible approaches are (1) to identify the root causes of risks—that is, identify the undesirable events or things that can go wrong and then identify the potential impacts on the project of each such event—and (2) to identify all the essential functions that the project must perform or goals that it must reach to be considered successful and then identify all the possible modes by which these functions might fail to perform. Both approaches can work, but the project team may find it easier to identify all the factors that are critical to success, and then work backward to identify the things that can go wrong with each one.
Risk identification should be performed early in the project (starting with preproject planning, even before the preliminary concept is approved) and should continue until the project is completed. Risk identification is not an exact science and therefore should be an ongoing process throughout the project, especially as it enters a new phase and as new personnel and contractors bring different experiences and viewpoints to risk identification. For this reason, the DOE project director should ensure that the project risk management plan provides for periodic updates.
The goal of risk identification is not only to avoid omissions but also to avoid the opposite pitfall—of being distracted by factors that are not root causes but only symptoms. Treating the symptoms, rather than the root causes, will give the appearance of activity but will not solve the
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
problem. Unfortunately, identification of symptoms is far easier than identification of root causes. Project owners should ensure that the risk identification process goes beyond the symptoms. While outside, disinterested reviewers can sometimes help perform this function, the following sections describe methods that can be used by project personnel to identify risks and their causes.
Following the initial risk identification phase, the project director should have a working list of risks that have been identified as potentially affecting the project. From this list, the project director should differentiate those that seem minor and do not require further attention from those that require follow-up, qualitative analysis, quantitative analysis, and active mitigation and management. This process requires some qualitative assessment of the magnitude and seriousness of each identified risk. Various methods that have been developed to assess failures in physical equipment and systems have also been applied in one form or another to project risks.
The commonly used risk tool shown in Table 4-1 is a two by two matrix that allows assigning a risk to one of four quadrants based on a qualitative assessment of its relative impact (high or low) and the likelihood of its occurrence (high or low). Risks in the upper right quadrant
TABLE 4-1 Risk Screening Based on Impact and Probability
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
need the most attention. Finer gradations of impact and likelihood—for example, very high, high, medium, low, and very low (a five by five matrix)—would allow a more nuanced consideration of the attention needed.
Risks that can be characterized as both low impact and low likelihood of occurrence are essentially negligible and can usually be eliminated from active consideration. The main concern of the owner’s project director is to monitor these factors sufficiently to determine that the impact or likelihood does not increase.
High Impact, High Probability
Risks that are characterized as both high impact and high likelihood of occurrence often cause a project to be terminated, or to fail if it is continued in spite of the risks. In this situation, the owner’s management must determine if the project should be terminated or if the project is so mission critical or the potential benefits are so great that taking the risks is justified. Risk management does not imply that no risks are taken; it means that the risks taken should be calculated risks. For example, an owner may decide to proceed if there is a reasonable expectation that enough engineering or management effort can reduce either the impact or the likelihood of the events, such that the risk can become either low impact, high probability or low probability, high impact. Often such a decision is contingent on achieving the necessary risk reductions by some deadline.
Low-impact, high-probability risks are those largely due to uncertainties about a number of elements that may be individually minor risks but that in the aggregate could amount to a significant risk. These include uncertainties concerning the actual costs of labor and materials (such as steel), the actual durations of activities, deliveries of equipment, productivity of the workforce, changes due to design development or the owner’s preferences, and other uncertainties that are typically considered to lie within the natural variability of project planning, design, construction, and start-up (they do not include catastrophic events or radical design changes). Each of these uncertainties, taken alone, would have little impact on the project. However, taken together, there is the possibility that many of the estimates of these factors would prove to be too optimistic, leading
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
to cumulative effects such as performance shortfalls, schedule overruns, and cost overruns. Methods for dealing with such risks include
- Provision for adequate contingencies (safety factors) for budget and schedule (contingencies are discussed in Chapter 6).
- Improvement in the work processes in order to reduce the uncertainties. Prefabrication of major components to avoid the uncertainties of construction at a job site is one example of changing the normal process to reduce risks (although in this example the change may also introduce new risks, such as transportation of the components to the job site; thus the resolution of one risk may give rise to another).
By definition, high-impact, low-probability events are rare occurrences, and therefore it is very difficult to assign probabilities to them based on historical records. Data do not exist and so subjective estimates of probabilities are necessary. However, the objective is not the scientific determination of accurate probabilities of rare events but the determination of what management actions should be taken to monitor, mitigate, and manage the risks. For example, if a certain risk is identified and management determines that some specific mitigation actions should be taken if the risk has a likelihood of more than 1 in 100 of occurring, then a precise characterization of the probability is unnecessary; the only issue is whether it is assessed to be more than 1 in 100 or less than 1 in 100.
Pareto Diagrams
One of the important uses of a good risk analysis is to determine where to apply management resources and what to leave alone, as management resources are not unlimited. One approach is to break down the uncertainties into manageable parts. Pareto diagrams are one way to show the sources of uncertainty or impact in descending order. This form of presentation makes explicit those activities that have the greatest effect on the project completion date or cost and that therefore require the greatest management attention. The project director or manager must then determine whether the high-ranking events are (1) truly root causes or (2) simply work packages or activities that may reflect underlying causes but are themselves symptoms. The resulting analysis can provide guidance for managers to reduce, mitigate, buffer, or otherwise manage these sources of uncertainty.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
As a simple illustration, suppose we are interested in determining which work packages have the greatest effects on the uncertainty in the total cost. First, we estimate the uncertainty, or variance, in the cost of each individual work package. Second, we estimate the correlations or associations between each pair of work packages. Then, by elementary second-moment theory (Benjamin and Cornell, 1970),1 the sensitivity of the uncertainty in the total project cost with respect to each work package is proportional to the combination of the activity uncertainties and the correlations between activities. That is, the uncertainty in the total cost is affected not only by the uncertainty in each work package but also by how much each work package affects, and is affected by, the others. As an elementary example, the uncertainty in the cost of a construction project may be more sensitive to outdoor activities than to indoor activities because unusually bad weather can cause a number of outdoor activities to run over budget and over schedule simultaneously, whereas indoor activities are typically not linked so tightly to the weather. By tabulating these values for all work packages, and sorting them from largest to smallest, we can identify those work packages with the largest sensitivities, which are those to which the project manager should give the highest priority. If we do this for a project of, say, 20 work packages and sort them according to the largest values of the sensitivities, we can then plot a Pareto diagram, as shown in Figure 4-1. (The absolute values of the sensitivities have no importance; the only concern is the relative values.)
In project risk assessment, a failure can be any significant event that the sponsor does not want to happen—a budget overrun, a schedule overrun, or a failure to meet scope, quality, or mission performance objectives. While risks may arise from specific causes, they may also be the result of general environmental conditions that are not limited to specific times and places but are pervasive throughout the project. The objective of failure modes and effects analysis is the identification of root or common causes, which may affect the project as a whole. Often this identification is facilitated by methodically considering the project function by function,
1 | All probability distributions may be characterized by their moments. Second-moment theory is the use of the second moments of probability distributions—that is, means, variances, and covariances (or correlation coefficients), instead of full probability distribution functions. As probability distributions are subjective and therefore not capable of precise definition, this approximate method can greatly simplify many calculations and, more importantly, provide the risk analyst with insight into the effects of uncertainty on project outcomes. |
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
to try to avoid omissions. Identification of potential risks that turn out, upon further assessment, to be negligible is a waste of time; however, failure to identify potential risks that turn out to be serious is a threat to the project. Therefore, the project director should err on the side of caution when identifying possible risks.
Failure modes and effects analysis (FMEA) is a discipline or methodology to assist in identifying and assessing risks qualitatively. It is a method for ranking risks for further investigation; however, it is not a method for quantifying risks on a probabilistic basis (Breyfogle, 1999). FMEA is typically based on a subjective assessment of the relative magnitudes of the impacts of the risk events on the project (often on a scale from 1 to 10), multiplied by the relative likelihood that the risk event will occur (also on a scale from 1 to 10). In addition, a third parameter may be included to assess the degree of warning that the project will have regarding the actual occurrence of the risk event (again on a scale from 1 to 10). This third parameter may give some management support by establishing early warning indicators for specific serious risks, which might not otherwise have been established.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
The purpose of assigning these values for all significant risks is only to rank the risks and to set priorities for subsequent quantitative analysis of the significant risks. In the absence of more quantitative factors, such as sensitivity analysis, the failure modes, or better, all root causes, can be used to rank the risks. One can prepare a Pareto chart that shows the risks ordered by possible impact or by the combination of impact and likelihood of occurrence. Then risk mitigation efforts can first address the failure mode or root cause with the highest impact and work from there.
The three factors—severity, likelihood, and leading indicators—interact. For example, if the project is the construction of a facility in a flood plain or an area with poor drainage, then a failure mode could be flooding of the work site. Project management cannot affect the frequency of floods, so risk management must focus on trying to reduce the severity of the impact of a flood. If the control method is to buy flood insurance and then evacuate personnel and abandon the site if the water rises, then measuring the height of the water (the “Nilometer” method) may be a sufficient indicator. If the control method is to reduce the severity of loss by placing sandbags around the perimeter and renting pumps, then measuring the water height may have little impact on the mitigation effort; but measuring the rainfall across the watershed may be more appropriate because it allows time to implement the control. If the control method is to build a cofferdam around the site before constructing anything else, then the choice of leading indicator may be irrelevant.
Efforts to mitigate the risks will focus on the impact, likelihood, and detectability of the most serious risk or its root causes and will try to reduce these factors until this risk becomes as low as or lower than the next higher risk. As this process continues, the most important risks will be reduced until there are a number of risks essentially the same and a number of other risks all lower than the first group. The first group will require specific management actions and may require constant monitoring and attention throughout the project. The second group will be monitored, but with lower priority or frequency. The first group is considered the critical group, much like the critical-path activities in a network schedule; the second group is the noncritical group, which must be watched primarily to see that none of the risks from this group become critical.
It should be emphasized that this form of risk assessment is qualitative and relative, not quantitative and absolute. It is primarily for distinguishing between risks that require follow-up and management, because of high impact or high likelihood (or both), and risks that do not appear to require follow-up, because of both low impact and low likelihood. It should be clearly understood that there is no quantitative assessment of the overall risk to the total project: The severity factors are not estimated
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
in terms of loss of dollars, the likelihoods of occurrence are not probabilities, and there is no cost-benefit analysis of the risks versus the control methods. The analysis only identifies risk priorities in a methodical way to help direct further risk management activities. It is left to the judgment of the project engineers, designers, and managers to determine the appropriate risk mitigation and control measures to achieve an acceptable level of risk. Note especially that risks with a low likelihood of occurrence but very high severities may require follow-up and management action.
Due to changes in project conditions or perceptions, even risks that appear to have low impact and high likelihood at one time may appear differently at another. Therefore, the owner’s representatives have the responsibility to reevaluate all failure modes and effects periodically to ensure that a risk previously considered negligible has not increased in either impact or likelihood to a level requiring management attention.
The Project Definition Rating Index (PDRI) is an example of an additive qualitative risk assessment tool (CII, 1996, 1999). The PDRI is used in front-end project planning to help the project team assess project scope definition, identify risk elements, and subsequently develop mitigation plans. It includes detailed descriptions of issues and a weighted checklist of project scope definition elements to jog the memory of project team participants. It provides the means to assess risk at various stages during the front-end project planning process and to focus efforts on high-risk areas that need additional definition. The PDRI facilitates the project team’s assessment of risks in the project scope, cost, and schedule. Each risk element in the PDRI has a series of five predetermined weights. Once the weights for each element are determined they are added to obtain a score for the entire project. This score is statistically correlated with project performance to estimate the level of certainty in the project baseline.
METHODS OF QUANTITATIVE RISK ANALYSIS
After risk factors are assessed qualitatively, it is desirable to quantify those determined by screening activities to be the most significant. It cannot be repeated too often that the purpose of risk assessment is to be better able to mitigate and manage the project risks—not just to compute project risk values. The assessment of risks attributed to elements completely out of project management control—such as force majeure, acts of God, political instability, or actions of competitors—may be necessary to reach an understanding of total project risk, but the risk assessment should
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
be viewed as a step toward identifying active measures to manage all risks, even those considered outside the control of project managers, not to support a passive attitude toward risks as inevitable.
It is often desirable to combine the various identified and characterized risk elements into a single quantitative project risk estimate. Owners may also be interested in knowing the total risk level of their projects, in order to compare different projects and to determine the risks in their project portfolios. (See the discussion of program risk and project portfolios in Chapter 8.) This estimate of overall project risk may be used as input for a decision about whether or not to execute a project, as a rational basis for setting a contingency, and to set priorities for risk mitigation.
While probabilistic risk assessment methods are certainly useful in determining contingency amounts to cover various process uncertainties, simple computation methods are often as good as, or even better than, complex methods for the applications discussed here. Owner’s representatives should be proficient in simple statistical approaches for computing risk probabilities, in order to be able to check the numbers given to them by consultants and contractors. When addressing probabilistic risk assessment, project directors should keep in mind that the objective is to mitigate and manage project risks and that quantitative risk assessment is only a part of the process to help achieve that objective.
There are many available methods and tools for quantitatively combining and assessing risks. Some of the most frequently used methods are discussed briefly below.
Multivariate statistical models for project costs or durations are derived from historical data. Also known as regression analysis, statistical models are one of two methods of analysis explicitly cited in OMB Circular No. A-94 (OMB, 1992). The models are typically either top-down or parametric and do not contain enough detail to validate bottom-up engineering estimates or project networks.
These methods are objective in that they do not rely on subjective probability distributions elicited from (possibly biased) project advocates. Analysts build linear or nonlinear statistical models based on data from multiple past projects and then compare the project in question to the models. The use of such statistical models is desirable as an independent benchmark for evaluating cost, schedule, and other factors for a specific project, but statistically based methods require a large database of projects, and many owners do not perform enough projects or expend the effort to create such databases. Owners who have performed many projects but have not developed usable historical project databases have an opportu-
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
nity to improve their competence in project and program management by organizing their own data. Computational methods such as resampling and bootstrapping are also used when data are insufficient for direct statistical methods.
The bootstrap method is a widely used computer-based statistical process originally developed by Efron and Tibshirani (1993) to create a proxy universe through replications of sampling with replacement of the original sample. Bootstrapping is used to estimate confidence levels from limited samples but is not applicable for developing point estimates.
Event trees, also known as fault trees or probability trees, are commonly used in reliability studies, probabilistic risk assessments (for example, for nuclear power plants and NASA space probes), and failure modes and effects analyses. The results of the evaluations are the probabilities of various outcomes from given faults or failures. Each event tree shows a particular event at the top and the conditions causing that event, leading to the determination of the likelihood of these events. These methods can be adapted to project cost, schedule, and performance risk assessments.
System Dynamics Models
Projects with tightly coupled activities are not well described by conventional project network models (which prohibit iteration and feedback). Efforts to apply conventional methods to these projects can lead to incorrect conclusions, counterproductive decisions, and project failures. In contrast, system dynamics models (Forrester, 1969) describe and explain how project behavior and performance are driven by the feedback loops, delays, and nonlinear relationships in processes, resources, and management. System dynamics models can be used to clarify and test project participants’ assumptions as well as to design and test proposed project improvements and managerial policies. Because system dynamics models are based on dynamic feedback the models can also be used to evaluate the impacts of various failure modes or root causes, particularly in cases where the root causes can be identified but the ripple effect of their impacts is difficult to estimate with any confidence.
System dynamics models have been effectively used for project evaluation, planning, and risk assessment (Cooper, 1980; Lyneis, Cooper, and Els, 2001; Ford and Sterman, 2003). Although the use of these models is not standard practice for project planning and risk management, they can significantly help owners to improve their understanding of project risks.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Sensitivity analysis of the results of any quantitative risk analysis is highly desirable. A sensitivity coefficient is a derivative: the change in some outcome with respect to a change in some input. Even if the probability of a particular risk cannot be determined precisely, sensitivity analysis can be used to determine which variables have the greatest influence on the risk. Because a primary function of risk analysis is to break down the problem into essential elements that can be addressed by management, sensitivity analysis can be very useful in determining what decisions the manager should make to get the desired results—or to avoid undesired results. In the absence of hard data, sensitivity analysis can be very useful in assessing the validity of risk models.
Project Simulations
Project simulations are group enactments or simulations of operations, in which managers and other project participants perform the project activities in a virtual environment before undertaking them on the project. This type of simulation may or may not be supported by computers; the emphasis is not on the computer models but rather on the interactions of the participants and the effects of these interactions on project outcomes. For this reason, project simulations are very good for team building before a project actually starts up. They are not inexpensive, but the cost is generally comparable to the costs of the other techniques cited here, and they can be very cost-effective in the long run, compared to the typical approach of jumping into major projects with little or no preparation of the personnel and their working relationships. Engineering and construction contractors have developed project simulation methods (Halpin and Martinez, 1999), and owners can develop their own or specify that their contractors should perform such simulations before a project starts, in conjunction with the other preproject planning efforts.
Stochastic simulation models are computerized probabilistic simulations that, for computational solution, typically use random number generators to draw variates from probability distributions. Because the computer simulation is performed with random numbers, these methods are also called Monte Carlo simulations. The objective of the simulation is to find the uncertainties (empirical probability distributions) of some dependent variables based on the assumed uncertainties (subjective probability distributions) of a set of independent variables, when the relation-
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
ships between the dependent and independent variables are too complex for an analytical solution. Thus each iteration (random simulation) may be considered an experiment, and a large number of these experiments gives insights into the probabilities of various outcomes. Monte Carlo simulation is typically used to combine the risks from multiple risk factors and as such is useful to determine whether the total risk of a project is too great to allow it to proceed or to determine the appropriate amount of contingency. This technique is the second of the two methods explicitly cited in OMB Circular No. A-94 (OMB, 1992).
Stochastic simulations differ from multivariate statistical models because they are typically not based on hard data. They can be useful in the absence of real data in that they are based on subjective assessments of the probability distributions that do not require large databases of previous project information. An often-cited weakness of this method is that subjective assessments of probability distributions often lack credibility, because they may be influenced by bias. This can be overcome to some degree by a carefully structured application of expert judgment (Keemey and von Winterfeldt, 1991).
As is the case with all the other computer methods for quantitative risk analysis discussed here, the validity of the method lies entirely in the validity of the probabilistic models. Monte Carlo simulation is very versatile because it can be applied to virtually any probabilistic model. However, the validity of the results may sometimes be suspect, due to the following factors:
- The independent variables may not actually be independent;
- The number of iterations in the simulation may be insufficient to produce statistically valid results; or
- The probability distributions assumed for the independent variables are subjective and may be biased if they are provided by project proponents.
It is certainly possible to develop project-specific cost models, for example, by using causal parameters that are totally independent. However, many risk analyses are not based on project-specific models but simply adopt the standard engineering additive cost models, in which the total cost is the sum of work package costs. The simulations simply add up the uncertainties associated with work packages, but they may be inaccurate because these work packages are not necessarily independent. It is computationally much easier to perform Monte Carlo simulation if the analyst avoids the need to consider interactions between variables by simply assuming that all variables are independent; however, an analysis without consideration of common mode failure can lead to an under-
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
estimation of total project risk. In project risk assessment, a common mode could be an event or environmental condition that would cause many cost variables to tend to increase (or decrease) simultaneously. It is widely recognized that a single event can cause effects on a number of systems (i.e., the ripple effect). If the event occurs, the costs of these systems will all increase, whereas if it does not occur, they will remain within the budget. Thus these affected costs are definitely not statistically independent.
Collaboration between people who are very conversant with the specific risks of the project and those who are familiar with probabilistic methods is typically required to reduce bias and to produce realistic quantification of project risks. Project owners should ensure that the probabilistic inputs are as objective and unbiased as possible and that the reasons for choosing specific probability distributions are adequately documented.
As with any method, the use of stochastic simulation requires quality control. The owner’s policies and procedures on Monte Carlo simulation should include cautions to project directors and managers about the limitations of this method as it is commonly applied. The project director is generally not a specialist in Monte Carlo simulation, and does not need to be, but should understand the advantages and limitations of this approach. This is particularly true now that Monte Carlo simulation is readily available through common spreadsheet software and so can be used by people with little knowledge of statistics. A project director should know enough to be able to critically evaluate the stochastic simulation results for plausibility and should not accept the results just because they come from a computer.
It is common for Monte Carlo simulations to use far fewer iterations than the minimum normally required to get statistically valid answers. But simulations with insufficient iterations may underestimate the probability in the tails of the distributions, which is where the risks are. (See, for example, Alder, Feldman, and Taggo, 1998.) Therefore, a simulation with fewer random samples may indicate more or less risk than one with more iterations. There are mathematical formulas (Breyfogle, 1999) that can be used to compute the minimum number of iterations for acceptable confidence limits on the means or the values in the tails of the distribution. If a consultant or contractor is performing Monte Carlo simulations for risk assessments, it would be prudent for the owner’s project director to review the confidence limits on all values computed using Monte Carlo simulation, to ensure that a sufficient number of iterations has been performed.
The use of Monte Carlo and other techniques for mathematically combining the risks of individual work packages into a single project risk number should not obscure the fact that the objective is to manage the risks.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
As typically used, Monte Carlo simulations tend to be focused on total risk probabilities, not on sensitivity analysis, risk prioritization, or assessing possible outcomes from different proposed risk management policies.
Additive models, as the name implies, are those in which the combination of risk factors is based on simple addition. An example is the summation of cost elements to generate the total project cost, or the summation of activity durations to generate the total project duration. These models are relatively simple programs based on the summation of moments, derived from probability theory, to combine risks for dependent as well as independent variables. If the objective is simply to find the probability distribution of the project cost estimate as the sum of a number of work packages or activities, stochastic simulation is unnecessary. One advantage of simple additive models is that they are easily understood, and it is usually obvious which activities contribute the most to the total project uncertainty and which do not. This method is the basis for the program evaluation and review technique (PERT) for determining uncertainty in project completion times.
In bottom-up project cost estimating, the total cost is simply the sum of the costs in the WBS work packages. This is a purely linear relationship. Therefore, estimating the uncertainty in the total cost requires only summing the uncertainties in the individual cost accounts, modified by the dependencies between them. Probability theory tells us that we can compute the moments of the probability distribution of the total project cost by summing the moments of the uncertainties in all the individual cost accounts (Burlington and May, 1953; Hald, 1952). The number of moments can be approximated to some finite number. (This is a very common method of approximation in engineering—for example, the truncation of a Taylor Series after one term in order to gain a linear equation.) The second-moment approach (Benjamin and Cornell, 1970) uses the first two moments, i.e., the mean and the variance, and neglects the third (skewness) and higher. The second-moment approach does not deal with full probability distributions but uses only the means, variances, and covariances (the first two moments) to characterize uncertainties.
This approximation is justified because it is very difficult or even impossible to estimate higher moments (skewness, kurtosis, etc.) with any accuracy, even if one were in possession of large historical databases. In most cases of risk assessment, the probability distributions are largely subjective and based on judgment and experience rather than hard data. There is little point in making highly precise computer calculations on numbers that cannot be estimated accurately anyway.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
There are some additional advantages of the second-moment approach:
- Priorities for risk mitigation can be obtained from a Pareto analysis using just the uncertainty in each individual risk factor and the correlations between risk factors.
- Sensitivity analyses are easily performed.
- As a project progresses, the estimates of the uncertainties in future cost accounts or activities can readily be revised, based on the past performance of the project itself. This is one of this method’s most useful properties. By comparing the actual performance on completed work packages, activities, or milestones with the prior estimated uncertainties, one obtains revised estimates of the work packages, activities, or milestones yet to come.
Through second-moment analysis, project directors can use the information and experience on the actual project to revise the estimates of the work to go. This approach can be a valuable tool for program managers, if each project director is required to report the updated, revised cost at completion, including the confidence bounds on this estimate, for every reporting period. Because this method looks forward instead of backward, as most other project management methods do (including earned value analysis), unfavorable revisions to either the expected cost at completion or the uncertainty in the cost at completion should trigger management action. Conversely, favorable revisions to either the expected cost at completion or the uncertainty in the cost could allow management reserves to be reallocated to other projects with greater needs. (See Chapter 8 for a discussion of managing risks of project portfolios.)
The second-moment method provides a simple, convenient method for the adjustment of risks, and hence the adjustment of the required contingencies, as a project proceeds and data are obtained on how well or badly it is performing. The objective of this approach is to react as soon as possible to information on recent project performance that tends to confirm or to refute the current estimates. The key control parameter is the estimated cost (or time) at completion. For example, if the best estimate of the cost at completion, updated with the most recent progress information, is higher than the original estimate, then, assuming no scope changes, either the risk of overrunning the budget is greater than originally estimated, or program management corrective action may be needed to bring the project back on target. Conversely, if the updated best estimate of the cost at completion is the same as or lower than the original estimate, then the required contingency can be decreased. In this approach, the estimates of all future work packages are updated as the actual costs for each completed work package become available through the cost reporting system.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
TABLE 4-2 Summary of Risk Analysis Tools
Tool | Characteristics |
Two-dimensional impact/ probability | Qualitative, simple to use and most frequently used, can be expanded to three or more dimensions, and can be combined with FMEA |
Pareto diagram | Simple qualitative method for prioritizing risk elements |
Failure modes and effects analysis (FMEA) | Qualitative, used for initial screening only, effective in a team environment |
Project Definition Rating Index | Qualitative, used in front-end project planning, effective in a team environment |
Multivariate statistical model | Quantitative, requires historical database |
Event tree | Quantitative, rarely used for risk analysis |
System dynamics model | Both qualitative and quantitative, rarely used but effective, requires skilled modelers |
Sensitivity analysis | Quantitative, useful regardless of which other process used, useful in absence of hard data |
Project simulation | Both qualitative and quantitative, useful for team building, expensive to implement |
Stochastic simulation | Quantitative, frequently used, often misused, so limitations must be made clear |
Additive model | Quantitative, can be adjusted as project progresses |
Table 4-2 provides a summary of the qualitative and quantitative methods of risk analysis reviewed in this section.
Although additive, second-moment models lack the computational complexity of stochastic risk assessment techniques, for most practical applications they are more than adequate. From the standpoint of the owner, the purpose of project risk assessment is to minimize the impact of uncertainty on the project. How this is best accomplished will vary with circumstances, but, in general, simple direct methods have proven them-
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
selves in practice. This does not discount the value of stochastic models, but their application needs to be considered in terms of their contribution to risk management. Probabilistic simulations are of particular value when data are sparse and the full range of possible adverse events cannot be easily inferred. Provided that a sufficient number of simulations are performed, boundaries for total project risk can be established. However, for the vast majority of projects, it is the committee’s collective experience that the simpler models are more useful for generating risk estimates that can be used in day-to-day project management.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Suggested Citation:'4 Risk Identification and Analysis.' National Research Council. 2005. The Owner's Role in Project Risk Management. Washington, DC: The National Academies Press. doi: 10.17226/11183.
Next: 5 Risk Mitigation »