Principle: Ambiguity in interpreting outcomes or performance measures/indicators of regulatory intervention when also seeking to prevent breaches (Also known as the Chameleon Regulatory Intervention Indicator Principle)
The number of regulatory interventions are often used as outcomes or performance measures/indicators for organizations. This usually occurs in the public sector, but it can also occur in private sector settings where the behavior of individuals is being regulated – for instance, HR issues within a large corporation. In cases where regulatory interventions are being used as outcomes or performance measures/indicators, there can be some ambiguity when attempting to interpret the meaning of an increase or decrease in the number of regulatory interventions being undertaken by an organization or department. On the one hand, an increase can be interpreted as, either: 1) showing that the organization is doing its job well (hence achieving its outcomes); or, 2) showing that the organization is failing to do its job because there is an increase in the behavior it is trying to regulate. Conversely, a decrease in regulatory interventions can be interpreted as, either: 1) showing that the organization is doing its job well (by using other means to prevent the bad behavior which is being regulated – i.e. ‘the regulatory intervention is only a last resort, we are focused on preventing it having to be used’); or, 2) showing that the organization is failing to achieve its mandate of effectively regulating the behavior. This ambiguity will increase to the degree that the organization or department includes within its outcomes the ‘prevention’ of the behavior it is regulating, in addition to it just having a role making straight regulatory interventions.
Because of this ambiguity principle, such changed in such indicators when presented on their own are not interpretable. In order to interpret them, they need to be put in the context of the visual outcomes model showing the steps in the intervention, what other indicators are being measured and what evaluation questions are being answered in regard to the outcomes model.
The Courts’ System is an example where an immediate (as opposed to a long term) preventive outcome is not usually seen as within the Courts’ mandate. Therefore the above principle would suggest that that there is likely to be little ambiguity in interpretation of regulatory interventions going up or down (other things being equal) in the case of Courts.
In contrast, a national department of conservation reported a reduction in the number of times it involved itself in a regulatory process. In this instance, it was the number of times it made legal representations to Conservation Resource Management Consent Hearings. At the time, the department was in the media spotlight for staff and budget cuts.
A media interviewer, interpreting the reduction in regulatory interventions as a failure of the department to effective pursue one of its outcomes, asked the department’s head: ‘…on the face of it, is it a lower priority? [the regulatory intervention – the department getting involved in Conservation Resource Management Consent Hearings]. The department’s chief (interpreting the drop in the measure in the opposite way) replied:
‘What you are falling into is the trap of judging and measuring our success by the number of cases we take regardless of the outcome. We see [Conservation Resource Management Hearings] [the regulatory intervention] as a last resort. We would rather sit down without spending money on lawyers and work out issues if we can and confine the [Conservation Resource Management Hearings] issues to ones that we really can’t reach agreement on’.*
Both sides in this argument are making reasonable ‘face value’ interpretations of the change in the indicator. In order to actually interpret what’s going on with this indicator, further information would be required. For instance, whether there has been an increase in departmental activity focused on getting the parties together prior to potential Resource Management Consent Hearings. The outcomes theory approach to making Chameleon Regulatory Intervention Indicators interpretable is to insist that they are always discussed against a visual outcomes model (e.g. a DoView). When viewed as in the DoView below, a clearer picture emerges of what is may be going on.
Information about the indicator in red would be required in order to be able to interpret the regulatory intervention indicator in black. Even then it could not be absolutely determined, just from the indicator in red, that the department had been successful in reducing the number of contentious issues going to hearings (which is the desired flow within this outcomes model). So one would really need to answer the evaluation question which also appears in the outcome model.
*’Can DoC Achieve its Core Role on its Current Funding and After its Third Round of Restructuring?’ Radio New Zealand Nine to Noon. 27 March 2013 9:29am http://radionz.co.nz.
First posted: 27 March 2013