Principle: Ambiguity in interpreting outcomes or performance measures/indicators of regulatory intervention when also seeking to prevent breaches (Also known as the Chameleon Regulatory Intervention Indicator Principle) The number of regulatory interventions are often used as outcomes or performance measures/indicators for organizations. This usually occurs in the public sector, but it can also occur in private sector settings where […]

Introduction [Note: This article is still being developed. Please post any comments for improving it at the end of the article]. This article works through a simple illustrative example to show the range of possible impact/outcome evaluation designs and techniques for improving the similarity between comparison and intervention groups in impact/outcome evaluation. Impact/outcome evaluation is one […]

Principle: Providing the evidential basis for all estimates The evidential basis for estimates of any sort should always be provided when estimates are given. This is so that anyone using such estimates can assess their likely accuracy. Failing to give such estimates exposes decision-makers to the risk of thinking that the estimates they have been […]

Duignan, P. (2012). Anyone else think the way we do our M&E work is too cumbersome and painful? Using DoView Visual Strategic Planning & Success Tracking M&E Software: Simplifying, streamlining and speeding up planning, monitoring and evaluation. The 1st Pan Asia-Africa M&E Forum RBM&E and Beyond: Increasing M&E Effectiveness. Bangkok, 26-28 2012. This virtual conference […]

A name for the outcomes theory principle: Unequal inputs principle (‘level playing field’).

The Canterbury region in New Zealand is currently being reconstructed following two major earthquakes in 2010 and 2011. In response a Canterbury Earthquake Recovery Authority (CERA) has been set up by the New Zealand government. This provides an example of a community needing to be reconstucted on a number of levels. In such instances of social reconstruction, which arise from natural disasters and other causes, it is important that a productive discussions are facilitated at various levels about the goals and coordination of the reconstruction. The key issues which need to be addressed are: 1. the involvement of stakeholders in major decisions regarding the reconstruction rather than it being dominated by national authorities; 2. determining the best practical tool/process for underpinning the high-level strategic direction discussions at various levels; and, 3. how gaps and overlaps between the multiple projects being undertaken by multiple parties (both from within the location being reconstructed and from outside of it) can be identified either for the reconstruction as a whole, or for certain sectors and sub-areas within the reconstruction. The DoView (R) Visual Planning (TM) process is a visually based tool/process which could potentially be used to ensure that discussions about the direction and priorities for reconstruction are undertaken in a way that facilitates clear strategic thinking. It could be considered for use in reconstruction processes at various levels, for instance at a high community-wide level, or more specifically in regard to service provision for particular sectors within the community which is being reconstructed. More information about DoView Visual Planning at http://outcomescentral.org.

A set of types of economic evaluation analysis can be identified and categorized in a new way based on whether or not effect-size estimates are available for changes to high-level outcomes brought about by the program or intervention being examined. Such an approach makes it easier to make decisions regarding what is the most appropriate type of economic analysis depending on the type of information available from impact evaluation about attribution of change in outcomes to the intervention being examined. The three major types of economic analysis (cost of intervention analysis, cost-effectiveness analysis and cost-benefit analysis) are further sub-divided on the basis of three levels of information being available about effect sizes. These levels are: 1) no attributable effect-size information available apart from the cost of the intervention; 2) attributable effect-size information available on mid-level outcomes; 3) and attributable effect-size information available on high-level outcomes.

Impact evaluation should always be considered in evaluation design but it should not be assumed that impact evaluation (also known as high-level outcome/impact attribution evaluation) should always be attempted. Impact evaluation attempts to prove that changes in high-level outcomes can be attributed to a particular intervention. The appropriateness, feasibility and affordability of doing impact evaluation for any intervention should always be carefully assessed. It is is often be better to save precious evaluation resources for use only on selected high priority impact evaluations or for non-impact evaluation (e.g. implementation/formative evaluation). Attempting impact evaluation where it is not appropriate, feasible or affordable can lead to pseudo-impact evaluations which appear to be impact evaluations, but do not provide robust information sufficient to satisfy key stakeholders that it has been established that changes in outcomes can be actually attributed to the particular program.

People use many different terms when working with outcomes systems (results, monitoring, performance management, evaluation, evidence-based pratice and strategic planning systems) and building outcomes models (logic models, results chains, strategy maps, intervention logics). This is partially a result of the range of different disciplines involved. Outcomes theory attempts to identify the smallest number of terms essential to do what needs to be done when doing outcomes-related work. One of outcomes theory’s insights is that the purpose of a number of terminological distinctions (such as vision / mission, final outcomes / intermediate outcomes, process / outcomes distinctions, outcomes / impacts) can be better achieved by working directly with a visual outcomes model and showing the causal position of boxes visually within the model. This approach avoids having to insist that stakeholders use specific terms (e.g. the outcome / impact distinction) in very specific ways. The diversity of the disciplines and settings in which people work with outcomes, plus the widespread continued common sense interpretation of a term such as an outcome, makes tight language control a somewhat futile strategy at the current time. Given that the same results can be achieved by just using a visual model, it is suggested that little energy should be put into arguing about terminological distinctions at the moment. (The substance of this article formed the basis for: Duignan, P. (2009) Rejecting the traditional outputs, intermediate and final outcomes logic modeling approach and building more stakeholder-friendly visual outcomes models. American Evaluation Association Conference, Orlando, Florida, 11-14 November 2009.)

There is a set of building-blocks which underlie all outcomes systems. Outcomes systems are any systems which attempt to specify, measure, attribute, or hold parties to account for changes in outcomes. These systems are called by a range of names such as: results management systems, performance management systems, evaluation systems etc. The building-blocks are: 1) an outcomes model/intervention logic; 2) not-necessarily controllable indicators; 3) controllable indicators; 4) high-level impact/outcome evaluation; 5) implementation evaluation; and 6) economic and comparative evaluation. This conceptual model can be used to clarity, critique and improve outcomes systems of any type in any sector.