What are outcomes systems?

A topic article within the Outcomes Theory Knowledge Base

Print Article
Citation
, XML
Authors
  • Duignan Paul
  • Paul Duignan Website: http://outcomestheor… Dr Paul Duignan is an expert outcomes specialist and program evaluator. He has worked in many sectors and with many organizations advising them on outcomes, evaluation, strategy, stakeholder processes, evidence-based practice an outcome-focused contracting. In particular he has extensive experience in the public and community health sectors, in all types of social programs and he has had some experience in the environmental, sustainability and related sectors. He has worked on outcomes and evaluation issues for organizations ranging from the IMF on the international scene through to many government ministries, departments and agencies, and with small community providers within local communities. He has taught evaluation at postgraduate level at several universities. His PhD developed an integrated framework for program evaluation. He presents at workshops, develops evaluation plans and develops outcomes models for a wide variety of organisations in many sectors. He has won international evaluation awards for his work, was the sole New Zealand Fulbright Senior Scholar in 2005 (working at the Urban Institute in Washing D.C. on outcomes issues). He has developed the area of outcomes theory (www.outcomestheory.org) integrated user-friendly approach to outcomes and evaluation called DoView Planning and been involved in the development of DoView Outcomes and Evaluation software (http://www.doview.com). His outcomes and evaluation blog can be found at http://www.outcomesblog.org. A more detailed Curriculum Vitae can be found here. The type of workshop and seminars Dr Duignan runs are described here. You can comment on and discuss his work on the DoView Visual Outcomes Planning group on Linkedin.

Abstract

This article argues that it is useful to view a range of different systems used in various sectors, disciplines and professions as all being types of ‘outcomes systems’. Such systems are attempts to deal with specifying, measuring, attributing and holding players to account for changes in outcomes of various types (hence the name outcomes systems). They include results management, performance, monitoring, evaluation, evidence-based practice, contracting and strategy. This article is part of the interlinked set of articles on various topics which make up the Outcomes Theory Knowledge Base.

Introduction to outcomes theory

Subscribe to author’s e-newsletter

Author’s Outcomes Blog

Author on Twitter

Introduction [1]

A common problem of major importance faced in all sectors, disciplines and professions is how outcomes of various types should be specified, measured, attributed, contracted/delegated and who should be held to account for changes in outcomes. Before the development of outcomes theory as a framework for conceptualizing the common principles which underlie all such systems, discussion of these types of systems has taken place using a diverse set of theoretical languages from a range of disciplines (e.g. economics, policy analysis, program evaluation, organization development, Human Resources theory, strategic planning theory etc.). Outcomes theory is an attempt to provide a common conceptual language consisting of a robust set of definitions and principles which can be used to analyze, describe, critique and improve such systems regardless of the sector in which they are being used; the disciplinary language being used to discuss them; or the profession which deals with them in day to day practice. At the heart of outcomes theory is the concept of an outcomes system.

An outcomes system is any system which attempts to deal with specifying, prioritizing, justifying, measuring, attributing and/or holding parties to account for changes in outcomes of any type and the steps which lead up to those outcomes. Such system go by a variety of names, such as: results management systems, strategic planning systems, performance management systems, performance measurement systems, program evaluation, evidence-based practice systems, investment strategies, value-for-money exercises, benchmarking exercises, contracting for outcomes systems,pay for performance systems, etc.

Figure 1: ‘Outcomes systems’ are operating in areas of activity which have traditionally been seen as somewhat distinct

In order to assist thinking about such systems, outcomes theory uses the concept of an outcomes model underlying all outcomes systems of any type. Such an outcomes model, which can be highly complex in the case of some outcomes systems, is conceptualized within outcomes theory as a visualized outcomes structure presented as a vertical visual hierarchy. At the bottom of the hierarchy are the lowest-level actions undertaken by players, these then lead up through a set of steps which are linked by causal links to the highest-level outcomes at the top of the visual model. It is these highest-level outcomes in the outside world which it is hoped a program or intervention will improved.

Needs met by outcomes systems

Outcomes systems conceptualized in this way are systems which attempt to meet a set of needs. The needs that outcomes systems of different types attempt to meet (not all systems attempt to meet all these needs) are set out below:

    1. Identifying and specifying outcomes (the things in the outside world which players are trying to ultimately change through their actions (interventions))
    2. Measuring either quantitatively or qualitatively the levels of such outcomes (indicators)
    3. Setting levels on indicators which parties are attempting to have have occur (targets)
    4. Entering into contracts for, and holding particular parties (providers) to account for, 1-4
    5. Attempting to attribute changes in particular indicators to the actions of particular parties (i.e. proving that it was the actions of a particular party or a particular group of parties which determined a specific change in an indicator)
    6. Identifying and specifying some, or all, of the relevant lower-level steps in the causal chain which leads from parties’ actions right up to high-level outcomes
    7. Identifying priorities for action in regard to lower-level steps within the causal chain in 7
    8. Doing 1-5 in regard to one or more of the lower-level steps within an outcomes model
    9. Providing analysis regarding the plausibility of the claimed causal link between lower-level steps and other lower-level steps, or lower-level steps and high-level outcomes
    10. Providing evidence from previous research supporting the claimed causal link between lower-level steps and other lower-level steps, or lower-level steps and high-level outcomes
    11. Using various research designs (called outcome/impact evaluation designs ) to attempt to attribute changes in high-level outcomes to an intervention
    12. In some instances, in the case of 10, attempting to estimate an effect-size for an intervention (the amount of change in a high-level outcome which results from a particular intervention of a particular size)
    13. Undertaking activity to improve the veracity of the outcomes model (e.g. talking to key informants, undertaking research of various sorts (process evaluation)
    14. Improving the implementation of interventions through undertaking research and related activities (often called formative or developmental evaluation)
    15. Attempting to provide cost estimates for undertaking various interventions
    16. Attempting to provide benefit estimates for various positive high-level outcomes and cost estimates for various negative high-level outcomes
    17. Combining 13 and 14 into cost benefit and cost-effectiveness analyzes.

Benefits of thinking in terms of outcomes systems

There is a common set of principles which underlie efforts to achieve the eighteen items listed here. It is the task of outcomes theory to clearly articulate these principles and their implications and to provide a robust set of definitions and conceptual models which can improve the way we talk about such systems and facilitate the sharing of knowledge across the many disciplines which currently deal with such systems. If successful, outcomes theory will lead to an effective way of comparing outcomes systems across sectors, communicating the best way such systems should be constructed, and assisting in the improvement of existing systems and the design of better outcomes systems in the future. How a robust outcomes system can be set up to underpin strategic planning, prioritization, performance management, evaluation, evidence-based practice and accountability specification in contracting is set out in Duignan’s Outcomes-Focused Visual Strategic Planning approach . A more technical example of how one can critique a particular outcomes system using outcomes theory principles is illustrated in the United Nations Results-Based Management System using the Outcomes Systems Checklist (a checklist of the features of a technically well-constructed outcomes system).

One specific area where a good understanding of outcomes theory adds value is in regard to misguided attempts to improve outcomes systems then tend to other problems within the system which were not foreseen because those doing the system redesign do not have an overall cohesive theory of the basic building blocks of a well constructed outcomes system. This leads to repeated swings in the reform of outcomes systems which tend to emphasis one or another of the basic building-blocks of outcomes systems at the expense of other building blocks. An example of such vacillation is between systems which emphasize measuring high-level outcomes (which ultimately creates frustration because such outcomes cannot be attributed to particular programs); and those which just measure lower-level steps because these can be attributed to particular programs and used to hold them to account (however, these systems, in turn, create frustration because they are seen as being just based on low level outputs. Outcomes theory is designed to help prevent such vacillation which, in the case of large scale public sector systems, for example, can be costly and cause considerable stress for those involved – particularly those at the lower levels within such systems who have to meet a constantly changing set of reporting demands from those above them.

Conclusion

The concept of an outcomes system is a new way of conceptualizing a range of different systems such as performance management, monitoring, results-based, evaluation, evidence-based practice, strategic planning and contracting systems. Outcomes theory lets us conceptualize the principles which determine whether or not such such systems are well constructed and provides specific recommendations for improving existing systems and building better systems in the future.

Please comment on this article

This article is based on the developing area of outcomes theory which is still in a relatively early stage of development. Please critique any of the argument laid out in this article so that they can be improved through critical examination and reflection.

Citing this article

Duignan, P. (2009).What are outcomes systems? Outcomes Theory Knowledge Base Article No. 216.(https://outcomestheory.wordpress.com/article/what-are-outcomes-systems-2m7zd68aaz774-15/).

Links

List of topic articles in the Outcomes Theory Knowledge Base Applied system using the outcomes theory approach to develop visual outcomes models and evaluation plans for projects and interventions of any type –Easy Outcomes

[If you are reading this in a PDF or printed copy, the web page version may have been updated].

[Outcomes Theory Article #216]

HS ED 21 Sep 2012

References

  • Some of this work was developed when the author was the 2005 New Zealand Fulbright Senior Scholar working at the Urban Institute in Washington D.C
  1. The building-blocks/types of evidence used in outcomes systems (Redirect)
  2. Types of claims able to be made regarding outcomes models (intervention logics/theories of change) (Redirect)
  3. Reconstructing a Community – How the DoView Visual Planning methodology could be used (Redirect)
  4. Simplifying terms used when working with outcomes (Redirect)
  5. Impact evaluation – where it should and should not be used (Redirect)
  6. Types of economic evaluation analysis (Redirect)
  7. Unequal inputs principle (‘level playing field’) principle
  8. Welcome to the Outcomes Theory Knowledge Base
  9. Organizational Requirements When Implementing the Duignan Approach Using DoView Within an Organization
  10. M & E systems – How to build an affordable simple monitoring and evaluation system using a visual approach
  11. Evaluation of Healthcare Information for All 2015 (HIFA2015) using a DoView visual evaluation plan and Duignan’s Visual Evaluation Planning Method
  12. DoView Results Roadmap Methodology
  13. Problems faced when monitoring and evaluating programs which are themselves assessment systems
  14. Reviewing a list of performance indicators
  15. Using visual DoView Results Roadmaps™ when working with individuals and families
  16. Proving that preventive public health works – using a visual results planning approach to communicate the benefits of investing in preventive public health
  17. Where outcomes theory is being used
  18. How a not-for-profit community organization can transition to being outcomes-focused and results-based – A case study
  19. Duignan’s Outcomes-Focused Visual Strategic Planning for Public and Third Sector Organizations
  20. Impact/outcome evaluation design types
  21. Introduction to outcomes theory
  22. Contracting for outcomes
  23. How a Sector can Assist Multiple Organizations to Implement the Duignan Outcomes-Focused Visual Strategic Planning, Monitoring and Evaluation Approach
  24. How community-based mental health organizations can become results-based and outcomes-focused
  25. Paul Duignan PhD Curriculum Vitae
  26. Integrating government organization statutory performance reporting with demands for evaluation of outcomes and ‘impacts’
  27. Non-output attributable intermediate outcome paradox
  28. Features of steps and outcomes appearing within outcomes models
  29. Principle: Three options for specifying accountability (contracting/delegation) when controllable indicators do not reach a long way up the outcomes model
  30. Outcomes theory diagrams
  31. Indicators – why they should be mapped onto a visual outcomes model
  32. What are Outcomes Models (Program logic models)?
  33. Methods and analysis techniques for information collection
  34. What are outcomes systems?
  35. The problem with SMART objectives – Why you have to consider unmeasurable outcomes
  36. Encouraging better evaluation design and use through a standardized approach to evaluation planning and implementation – Easy Outcomes
  37. New Zealand public sector management system – an analysis
  38. Using Duignan’s outcomes-focused visual strategic planning as a basis for Performance Improvement Framework (PIF) assessments in the New Zealand public sector
  39. Working with outcomes structures and outcomes models
  40. Using the ‘Promoting the Use of Evaluation Within a Country DoView Outcomes Model’
  41. What added value can evaluators bring to governance, development and progress through policy-making? The role of large visualized outcomes models in policy making
  42. Real world examples of how to use seriously large outcomes models (logic models) in evaluation, public sector strategic planning and shared outcomes work
  43. Monitoring, accountability and evaluation of welfare and social sector policy and reform
  44. Results-based management using the Systematic Outcomes Management / Easy Outcomes Process
  45. The evolution of logic models (theories of change) as used within evaluation
  46. Trade-off between demonstrating attribution and encouraging collaboration
  47. Impact/outcome evaluation designs and techniques illustrated with a simple example
  48. Implications of an exclusive focus on impact evaluation in ‘what works’ evidence-based practice systems
  49. Single list of indicators problem
  50. Outcomes theory: A list of outcomes theory articles
  51. Standards for drawing outcomes models
  52. Causal models – how to structure, represent and communicate them
  53. Conventions for visualizing outcomes models (program logic models)
  54. Using a generic outcomes model to implement similar programs in a number of countries, districts, organizational or sector units
  55. Using outcomes theory to solve important conceptual and practical problems in evaluation, monitoring and performance management
  56. Free-form visual outcomes models versus output, intermediate and final outcome ‘layered’ models
  57. Key outcomes, results management and evaluation resources
  58. Outcomes systems – checklist for analysis
  59. Having a common outcomes model underpinning multiple organizational activities
  60. What is best practice?
  61. Best practice representation and dissemination using visual outcomes models
  62. Action research: Using an outcomes modeling approach
  63. Evaluation questions – why they should be mapped onto a visual outcomes model
  64. Overly-simplistic approaches to outcomes, monitoring and evaluation work
  65. Evaluation types: Formative/developmental, process and impact/outcome
  66. Terminology in evaluation: Approaches, types (purposes), methods, analysis techniques and designs
  67. United Nations Results-Based Management System – An analysis
  68. Selecting impact/outcome evaluation designs: a decision-making table and checklist approach
  69. Definitions used in outcomes theory
  70. Balanced Scorecard and Strategy Maps – an analysis
  71. The error of limiting focus to only the attributable
  72. Reframing program evaluation as part of collecting strategic information for sector decision-making
  73. Distinguishing evaluation from other processes (e.g. monitoring, performance management, assessment, quality assurance)
  74. Full roll-out impact/outcome evaluation versus piloting impact/outcome evaluation plus best practice monitoring
  75. References to outcomes theory
  76. Techniques for improving constructed matched comparison group impact/outcome evaluation designs

Meta