Mastering Opportunities and Risks in IT Projects (Book Excerpt)

Identifying, anticipating and controlling opportunities and risks: A model for effective management in IT development and operation.

Management systems show the management objectives (in relation to a specific purpose), proven methods for achieving the objectives and the associated control and monitoring mechanisms. The purpose can be the management of a company, an IT project or just compliance with a quality, environmental or information security standard. This book uses a model to describe how the fundamentally necessary core process of risk management works within such a management system. Its main feature is the cyclical repetition of the identification and evaluation of opportunities and risks, which leads to necessary control measures influencing them and to the subsequent implementation of measures to improve the effectiveness and efficiency of the management system.

Each cycle begins with the definition of key goals for the purpose intended and a review of the decisions already made. This is followed by the identification of influencing factors that threaten or even favor the achievement of goals - and their assessment. As the basis for focusing on key goals and acute threats, this step is an important prerequisite for economic risk management. 

In order to find influencing factors that favor or threaten the achievement of the selected goals, the combination of a methodical cause-and-effect analysis with own empirical values is recommended. The risk portfolio comprises the set of all influencing factors. After an assessment of their probability of occurrence as well as of the amount and type of potential damage that may occur if a goal is not achieved, this enables the focus to be placed on a subset of the threats, which in the model described is referred to as the risk profile.

Model for managing opportunities and risks

The risk profile reassessed in the respective cycle forms the basis for management decisions on the treatment of risks. This treatment can, for example, consist of taking out insurance (risk transfer), outsourcing risky areas from the IT project under consideration to other parts of the organization (restructuring) or implementing and following up measures to actively reduce risk. If, in individual cases, the risk owners do not consider it necessary to reduce the risks by means of a transfer, restructuring or measures, this must be documented as acceptance of the residual risks.

In the final step of each cycle, the measures that can be taken to improve the effectiveness of risk management and its cost/benefit ratio should be examined. This includes a review of all components of the model such as processes, methods, metrics and scales for evaluation, documentation, etc.

Considerable potential for improving the management model lies in making knowledge about general or organization-specific threats to the goal types explicit - and in permanently adapting this explicit knowledge to changes in the threat situation, for example. One way of implementing this explicit knowledge is the creation and maintenance of sets of rules that submit questionnaires to the people responsible for risk analysis and from whose responses corresponding entries in the risk profile result. However, these suggested values must be validated and possibly corrected on the basis of own human judgment and intuition. A further improvement is foreseeable as soon as machine learning systems can be used for risk analysis purposes.

It is possible to carry out risk management both effectively and efficiently: With good methodology, threats can be noticeably mitigated. Thereby risk management becomes effective. The avoided damage results in a benefit that can be calculated and placed in relation to the required effort. This results in a measure of efficiency. As with any other management process, both effectiveness and calculability of efficiency are an essential basis for optimization.

The book is available as e-book, paperback and hardcover (directly in the Tredition book shop incl. reading sample - or in bookstores).  

Economical Measurements (The Limits of Automation)

The functional size of an application should be measured regularly in order to determine the growth and hence the development productivity of each release or increment. Only by comparing cyclic measurements, improving or worsening effects can be identified. Furthermore the organisation gets empirical productivity values which are the key for cost estimations of new development projects (see: Measuring Productivity in Software Development). 

Experience shows that cyclic measurements will be neglected if the accociated effort is too high. Measuring techniques based on counting rules allow a high degree of automation by mapping the objects to be counted to design features of an existing application and implementing the according scripts or programs for counting. The investment in implementing these script or programs quickly pays for itself when repeated measurements do not require anymore manual effort. Possible targets for automated counting are:

• Dialog models represented by XML or XHTML files as being used by many GUI frameworks

• XML schemas defining interface or message structures

• Meta data of a DBMS including information about tables and attributes

When applying automated measurement processes which have a stronger focus on design features than on the use cases of a system there is a risk of inaccurateness if counting objects have been used multiple times by the application design. An example of this is the comparison of two applications A and B, which are different implementations of the same use cases:

• At application A similiar dialogs have been implemented separately and partially redundant.

• At application B a generic dialog has been implemented which automatically adapts according to the use case respectively invocation. 

An automated size measurement based on design features will result in a lower value for application B, which is wrong because both applications are implementations of the same use cases. In the worst case this will promote bad development practice (copy & paste) and poor design, if developers try to increase the measured productivity by creating redundant counting objects. Actually this is fatal in several ways, because the reuse of components and services is one of the most important keys to sustainably increase productivity and improve maintainability.

Conclusion: Automated measuring techniques based on design features are necessary to make size measurements practicable at all. However a side effect is the limited comparability of applications with different degrees of reutilization.

The COSMIC Method

COSMIC is a size metric (see Methods for measuring the Size of Software) defined by the standard ISO/IEC 19761. The method is conformal to the standard for Functional Size Measurement (ISO/IEC 14143) and thereby it considers the use cases of a system (similar to the Function Point Analysis).

Objects to be counted by the COSMIC method

COSMIC uses an own definition of actors, the so-called "Functional Users", which can be - additionally to human users and external systems - any hardware or software components sending data to the software to be measured across the system boundaries and thereby triggering a functional process, or receiving and processing data from a functional process of the software. Depending on the direction these data movements are called "entries" or "exits". Additionally, the method counts accesses to data elements which are stored in the database ("read" or "write" operations) and related to the functional processes.

An advantage of the COSMIC method is that it counts data elements directly instead of larger data structures or elementary processes as the Function Point Analysis does (see also Is the Function Point Method still up-to-date?). Another advantage is that regarding the database it considers data movements performed by the functional processes, i.e. read and write operations, and not the static size of the database. These characteristics along with the broad definition of "Functional Users" makes COSMIC suitable for measuring the functional size of real-time systems.

A disadvantage can be that all entries, exits, reads and writes have the same weight and therefore differences in the complexity of their pre- or post-processing can not be considered.      

The Impact of neglected Quality Assurance on Productivity

The productivity of a software development project can be raised by neglecting quality assurance and starting production with an inadequately tested application. One reason for this can be time pressure. An apparent positive effect: The new-development productivity is higher than in projects with more test efforts. 
 
But: Beyond the project duration the effects are:

• Higher defect rate compared with other systems
• Lower productivity when developing future releases or increments, due to the high effort for error analyses and corrections

In most cases these effects are not of lasting nature because the defect rate can be reduced to a "normal" level by high efforts required after production start. However this approach is short-sighted because the effort for tests, error analyses and corrections is simply being shifted from the new-development project to those who are responsible for the subsequent further development. And test effort is being shifted to the first users of a new system. 

The diagram below shows a sample course of defect density and productivity after production start (Rel. 1.0) which is typical in such cases. Contrary to the Impact of Technical Debt on Productivity these projects are mostly able to recover after a longer period, i.e. the defect density decreases due to the meanwhile found and fixed errors and the productivity increases due to the reduced efforts for error correction.

Course of Defect Density and Productivity in a case of neglected qa (example)

An indicator for neglected quality assurance is a low test coverage.

My post Measures for optimizing Productivity shows better measures for increasing the productivity.

Software Development with Methods of the industrial Production

Methods of industrial production may be a response to 

• increasing complexity of requirements,
• shorter development cycles and
• growing cost pressure.

This has nothing to do with the series production of the same product. It more concerns the following aspects:

• Standardization of system and application architecture 
• Reusability of components
• Automation  of development and quality assurance processes
• Measurability of productivity and quality

While today a lot of standards, concepts for reusability and tools for automation are available and established in software development, many companies neglect measurements or use estimations instead. However regular and accurate measurements of productivity and quality are an important basis for the reliable determination of the effort for upcoming development projects (see also Measuring Productivity in Software Development).

The Impact of Technical Debt on Productivity

Some measures for raising the productivity are not very recommendable:

• Incomplete implementation of requirements
• Postponed corrections of known defects
• Disregard of commentation and technical design
• Insufficient or missing documentation
• Insufficient test coverage
• Build and deploy processes too complex and not automatable

A development project increasing the productivity by such measures takes on technical debt which must be paid back including interest after this project is completed - while the further development. Interest payments are:

• increasing effort for further development, which results in decreasing productivity,
• above-average effort for defect analyses and corrections.

The impact of technical debt may look similar to the diagram below. After completion of a development project and production start of an application (Rel. 1.0 in this example) the productivity measured for the further development (Releases 1.1, 1.2, etc.) is decreasing continuously (see: Measuring Productivity in Software Development), provided that the input considers the total expenses including these for defect analyses and corrections. 

Course of Defect Density and Productivity in case of technical Debt (Sample)

Steady code analyses are suitable to detect symptoms of technical debt almost early, already while the initial development project (see: Using Sonar for steady Analyses of the Code Quality). 

My post Measures for optimizing Productivity shows better measures for increasing the productivity.

The PASS Data Interaction Point Method (DIP Method)

The Data Interaction Point method of the PASS Consulting Group is a function-oriented size metric (see: Methods for measuring the Size of Software). It counts data elements which are involved in interactions of the considered system with the actors of its use cases, which can be human users as well as other systems. Complexity is considered by different weights of the counts, e.g. when input elements of a dialog have a higher weight than display elements.

Interactions to be counted by the
Data Interaction Point Method

The basic principles, i.e. the orientation on use cases and the counting of data objects from the external view of the actors are the same on which the Function Point Analysis (FPA) and the COSMIC method are based on. However, a major difference to the FPA is the level of detail for counting. FPA counts e.g. data structures, where the score depends of their number of data element types and record element types defined by open-ended interval scales with only three steps. The DIP and the COSMIC method directly count data elements. While COSMIC does not distinguish score values the DIP method derives them by the usage of data elements, e.g. whether they are for read-only or for read-write access, and thus oriented towards the interactional complexity.

The DIP method is better suited for benchmarking size-related KPIs, e.g. the productivity (see: KPIs for Controlling in Software Maintenance and Development).

Also Measurements can help creating Quality

Additionally to root cause analyses of defects (see: Defects can help creating quality) steady measurements and benchmarks are an important basis for continuous improvements. The following three metrics can already cover the most important aspects:

• Productivity (Cost Efficiency): The performance of the development process until the achievement of a predefined (outer and inner) quality ready for delivery (see: Measuring Productivity in Software Development).

• Defect Rate (Defect Density): The quotient from the number of production defects per time unit and the size of the system. It must be defined which types of defects are relevant for the defect rate.

• Maintenance Effort/Costs: The quotient from the effort for maintenance tasks per time unit and the size of the system. Ideally differentiated according to task types as defect analysis, defect correction, advisory services, etc.

For each of these aspects target values resp. thresholds (baselines) are important, where exceeding indicates a need for clarification or, in the next step, a need for immediate action. Finding these thresholds will be facilitated through benchmarks of different systems, projects or shops, where measurements will be performed by using exactly the same methods.

The easiest way is using the mean value or, if extreme values shall not be considered, the median as a threshold. In cases where a measured value is worse than the mean value or the median of all values an analysis or assessment helps finding the root causes. Mostly root causes lead to improvement measures. Their effectiveness becomes transparent by the course of subsequent measurements.

Defects can help creating Quality

Even if software defects are not considered as ‘normal’ (see: Is Software with Zero-Defect Quality possible?), every defect is a chance for improvement.

For each defect a root cause analysis should be performed. This does not mean superficial causes like ‘programming error’. A proven method for the root cause analysis is ‘5 Why’, where you ask multiple (not necessarily five) times for the reason. Only when you reached the beginning of the cause-and-effect chain that there is no practical answer to the ‘why?’ question, the root cause most likely has been found.

At this point it is important that the root cause of the defect matches with a defect category. The list of defect categories should be standardized company-wide. Technically these categories can be implemented as a pick-list in the ticketing or defect tracking system. The standardization is the key for evaluations like categories which are defect hotspots, which is important to see where improvement measures are most effective.

With knowledge of the root causes improvement measures should be found which prevent the recurrence of such defects. This can be improvements of the constructive quality assurance, e.g. development methods or tools, possibly changes of the team, or regarding other areas. Of course there may be some defects which can not be prevented in the future by any measures.

A continuous improvement process on basis of defect analyses will only work if anybody is responsible for the communication of the found measures, plans their implementation, keeps track of it and checks the effectiveness after implementation.

This process, which similarly is in use in the car industry since the 70s, helps preventing defects in the future, which is a better investment than in analyses, tracking and corrections of defects. In other words: It creates quality.

Using Sonar for steady Analyses of the Code Quality

The platform Sonar allows static code analyses of different projects and presents the results in a standardised form. Sonar applies some own rules enhanced by the rules of integrated tools as e.g. FindBugs, Checkstyle or PMD (for the Java world).

The experiences with Sonar used in a huge portfolio of applications are good. For Java there are more than 800 rules available, which are well described, can be activated and deactivated and can be configured regarding the severity of violations (e.g. Blocker, Critical, etc.) or parameters. Experience has shown that it is better not to use the standard configuration but to discus an own configuration with the most experienced developers and to rate less rather than more rules as blockers. Further it is important that the process model respectively the quality gates clearly define that e.g. it is forbidden to use a release as long as there are blockers, which at least are potential production defects.

Sonar Dashboard showing the results of a Code Analysis (Sample)

Additionally to these rules Sonar measures the comment density, the rate of commented or duplicated code, the cyclomatic complexity, the lack of cohesion in methods (LCOM4) and the class complexity (response for a class, RFC), which are proven indicators for the code quality. Based on an own coding styleguide, which should clearly define threshold values for these metrics, a Sonar profile can be used to set these thresholds for an indication of warnings and errors for each metric. This makes sure that each application being analyzed will be measured and rated by the same standard.

Best practise is to integrate the Sonar code analysis in the build processes of the different applications. Then, after each new build, i.e. each night or continuously - according to the configuration, a new analysis result is available.

The Sonar web client presents the analysis results in dashboards and even in case of a huge portfolio of applications clearly structured. There are tools for e.g. the analysis of time courses or hotspots. The authorized user can drill-down for each violation into the code and, if required, declare it as false-positive.

The experience is that only with a tool such as Sonar the definition of a coding styleguide is really worthwhile, because the compliance can be checked without extensive time and in short intervals.

Information about Sonar can be found at http://www.sonarqube.org/features/.

KPIs for Controlling in Software Maintenance and Development

The following indicators allow controlling of the essential key performance areas of software maintenance and development:

• Productivity (Cost Efficiency): The quotient from the size of produced or changed software and the effort required (see: Measuring Productivity in Software Development).
• Defect Rate (Defect Density): The quotient from the number of production defects per time unit and the size of the system. It must be defined which types of defects are relevant for the defect rate.
• Effort/Costs for correcting Defects: The quotient from the effort for correcting defects (only types which are relevant for the defect rate) per time unit and the size of the system.

These measurements are completely independent of any process model or methodology, i.e. they are useful for heavyweighted processes as well as for agile development. Even agile development requires KPIs in-time for justification.

The benefits of KPIs in software maintenance and development are:

• Performance and quality become measurable and comparable.
• KPIs allow the determination of quantitative success criteria and a precise verification of their achievement.
• Trends become transparent. KPIs serve as indicators for a need for action.
• Performance and quality become predictable (e.g. effort estimations on basis of experiences and empirical data regarding the productivity of similar projects).