“When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of the meagre and unsatisfactory kind.” - Lord Kelvin

A metric is a purpose-specific expression of the quantity and/or quality of goods and services. By measuring and summarising a set of environmental goods and services into a standard metric, we can make comparisons between different sets of goods and services for ranking priorities or making investment decisions. Market Based Instruments use metrics as a currency of what is being bought and sold in the market. Just as we can determine the dollar values of two bags of carrots of different quality or the dollar value of a bag of carrots compared to a bag of potatoes, we can use an appropriate biodiversity metric to determine the biodiversity values of fencing grazing animals out of two different areas of native vegetation. Converting the information we have about natural resource goods or services into units of a metric allows us to make comparisons between different amounts, types and locations of goods and services and their relative contributions to NRM outcomes.

The metric used in an MBI will be determined by the objectives of the MBI, the type of goods and services being measured and the data available to construct the metric. A metric can be as simple as the number of hectares of native vegetation, or as complex as a mathematical model of multiple environmental outcomes from a set of management services.

A metric can be constructed from:

• existing data
• new measurements
• models
• expert knowledge
• community values, or
• a combination of any or all of the above

Metrics are used in many ways to define and measure qualities and quantities of interest for the purpose of comparison between places or over time. Two metrics which may be familiar to users of this document are the Consumer Price Index (CPI) and the Dry Stock Equivalent (DSE) used to measure inflation and stock carrying capacity, respectively.

Metrics can be designed to describe a range of natural resource goods and services and are used in most MBIs. Metrics can take many forms and contain a range of attributes.

Achieving natural resource management (NRM) outcomes usually requires a mix of policy instruments and MBIs are part of that mix. The effectiveness of different approaches and combinations of approaches should be considered before an MBI is adopted. For example, the effective implementation of an MBI may require supporting regulation and provision of extension and information. If an MBI is to be used it must be designed to make effective use of existing and past policy interventions.

An MBI can use a number of mechanisms to define and select the goods and services desired from the market. These include an assessment framework and selection (or exclusion) rules or criteria and trading rules which restrict trades leading to undesirable outcomes (e.g. problem hotspots, off-target impacts, perverse or inequitable outcomes). Metrics are one of the components of MBIs and are used to quantify the goods and services which are being sought and traded. For example, in a cap-and-trade mechanism for water the metric can be as simple as volume of water in megalitres. For a conservation tender or offset scheme the metric may combine attributes representing conservation significance and management services to determine the relative benefit of taking different actions in different parts of the landscape. The metric provides a relative measure of quantity and/or quality of alternative bundles of environmental services, which can be used to inform investment decisions.

Using a well-designed metric in an NRM MBI supports the:

• measurement of indicators of relative or absolute change
• management of risk (due to scientific uncertainty)
• definition and measurement of (some of) the rights which form the basis of contracts
• allocation of scarce funds between alternative uses
• evaluation of NRM outcomes.

A well-designed metric used in an MBI can also:

• encourage focus on clear, defined objectives
• ensure that all participants in the ‘market’ are treated with a level of objective fairness based on a previously agreed set of criteria for valuing goods and services (probity)
• facilitate the collection of data with potential for other uses in NRM capacity building, planning and evaluation
• improve natural resource management through knowledge and communication by creating evidence based management and
• incorporate community values, expert knowledge and biophysical data into NRM decision making
• support the setting, meeting and evaluation of regional targets.

Step 1 and step 2 of the decision support tool outline the key questions to be answered before selecting and designing an MBI. A number of these questions are also relevant to designing a metric to work in an MBI because the metric needs to be designed to support the MBI to achieve the policy and program objectives.

What is measured by a metric?

Metrics support MBIs by defining and measuring the state and change in environmental goods and services. It is most desirable to measure the outcomes from NRM to design the metric to most closely reflect the cause and effect relationships from management and increase the effectiveness of the metric. Measuring potential outcomes also allows for greater flexibility in the approaches taken to achieve the natural resource condition change; land managers may change inputs or practices with consideration of the cost implications and outcome benefits for individual profitability and NRM. However, outcomes are not always known or easily measured (see figure 1) and it may be necessary to make assumptions about how activities lead to outcomes and measure the inputs or outputs as indirect measures of the outcomes (figure 1). Issues related to measuring change are discussed further in later sections of this document.

Fig 1 Cause and effect and measurability of change from NRM.

Other potential benefits of metrics

A well-designed metric which is cost-effective to measure can be used for purposes other than those described for MBIs. If the metric and its components describe the quantity and quality of goods and services, this information can be used to: focus capacity building programs; improve knowledge of natural resource condition and change; prioritise between the types and location of action in non-market-based incentive programs or even on publicly owned and managed land; and monitor natural resource condition changes and evaluate policies or intervention programs.

For example, a metric designed for use in a conservation tender could be used to prioritise sites for:

• targeted land-manager awareness, education or skill development programs
• preparation of management plans
• implementing a comprehensive monitoring program
• evaluating a past incentive scheme (compared to unfunded control sites)
• building and reporting on the knowledge base of natural resource distribution and status
• a fixed-price incentive payments scheme.

In simple terms, a metric defines what is bought and sold in the market. Many existing markets and institutions do not adequately value environmental goods and services and the resulting market failure results in misaligned incentives between private providers and society (Stoneham et al, 2003). Market failure can occur because of asymmetries in information between landholders and environmental agencies. In simple terms, the purchaser (ie. environmental agency) knows best what they want to purchase, the seller (ie. landholder) knows best what the value of the goods and services or property rights are, the MBI operates to efficiently reveal the information held be each party to the other to achieve cost-effective management of the natural resource. The metric is the quantitative expression of what is being bought and sold.

Eigenraam et al (2007) suggest that an environmental agency, not landholders, knows its preferences/priorities regarding different environmental assets and an agency may be in a better position than landholders to predict how management actions will enhance environmental assets. The environmental agency needs a ‘metric’ to describe the good/service that would come from improved environmental management by the landholder. The metric then becomes a tool for discriminating between alternative investments in the market of landholder offers.

Other benefits of metrics

The main use of a metric is the quantification and combination of multiple attributes into a single weighted score. This score can then be used as the basis for a range of decisions, including decisions about allocation of funding through an MBI. The metric score can also be used for evidence-based prioritisation of different sites or projects for purposes other than implementing an MBI. Connor et al (in press) have shown that much of the efficiency gained by using a conservation tender instead of a uniform payment incentive scheme (in the Catchment Care Program) came from improved measurement of the relative values of different actions (or similar actions in different parts of the landscape) and the subsequent ability of the program to prioritise and fund the highest value for money projects. Using the metric to rank priorities for investment has benefits which may persist regardless of the mechanism of incentive distribution.

A well designed metric which provides an evidence base for prioritisation could be used to prioritise other intervention programs, not only incentive payments. Depending on the cost of developing and collecting data for the metric, the metric may be an efficient tool for determining the priority sites for preparing management plans or even targeting priority landholders for capacity building. Conner et al (in press) were able to use the metric developed for the Catchment Care Program to evaluate the cost-effectiveness of the pre-existing fixed price incentive scheme. Having collected data for the metric, decisions could also be made about priority sites for inclusion in a program monitoring the environmental assets and threats.

Connor J.D., Ward J. and Bryan B.A. (in press). How Cost Effective are Conservation Auctions? The Australian Journal of Agricultural and Resource Economics.

Eigenraam M., Strappazzon L., Lansdel N., Beverly C. and Stoneham G. (2007). Designing frameworks to deliver unknown information to support market-based instruments Agricultural Economics 37:261-269

Stoneham G., Chaudhri V., Ha A., and Strappazzon L. (2003). Auctions for Conservation Contracts: An Empirical Examination of Victoria's Bush Tender Trial. Australian Journal of Agricultural and Resource Economics 47: 477–500

A metric must be appropriate for the purpose, place, scale and time of use. Some existing metrics may be modified for use in new projects or locations dealing with similar natural resource mangement (NRM) issues, while other circumstances will require that a new metric be developed. To effectively support an MBI a metric needs to:

• focus on program objectives and priorities
• combine quality and quantity assessments where appropriate
• provide an objective, reliable and repeatable measure of goods and services
• be simple to understand and explain to participants, and transparent enough to demonstrate fair decision making
• enable manageable calculation of net change or outcomes
• ensure design and implementation are cost-effective
• allow comparisons and discrimination between a range of realistic scenarios or groups of goods and services
• be defensible if data is limited or uncertainty high
• consider the reversibility of impacts, side effects, market interactions and chance of success
• take into account trigger thresholds that would have a major impact on the desired outcomes
• consider time lags for outcomes to be realised
• be clear about what it does not measure.

To develop a metric with these characteristics it is necessary to draw on expertise from a range of intersecting disciplines including policy and planning, economics, science and the delivery team (figure 2). The community representing users of the ecosystem services being sought should also be consulted to ensure the MBI is appropriate and the metric captures the full suite of potential values.

Fig 2 Metric design is a multi-disciplinary activity which incorporates objectives, knowledge and skill from policy, economics, science and the MBI delivery.

Focus on program objectives and priorities

Metrics should be designed to support the achievement of the objectives of the policy and program. Metrics can be designed to incorporate and weight criteria with respect to priorities, or the metric may be relatively simple and the MBI designed with decision criteria about preferences and priorities for NRM outcomes. The metric should work as part of the MBI to achieve overall objectives and avoid perverse outcomes. A program designed to reduce nutrient run-off from land used for cropping can incorporate decision criteria in the MBI design or directly in the metric by appropriately weighting indicators of reduced nutrient loss related to management practice change. The program may not want to encourage broadscale land-use change and should ensure the metric and MBI designs do not promote this type of perverse outcome. This will require an adequate regulatory framework underpinning the MBI, appropriate rules in the MBI, and weighting in the metric to prevent or deter undesirable outcomes. A good understanding of the market and market behaviour will also assist the prevention of perverse outcomes.

If the program aims to achieve multiple NRM outcomes (e.g. water quality and biodiversity conservation) the MBI needs to be designed with this focus. One way to do this is to develop a metric which measures and combines attributes from the multiple natural resource outcomes sought.

Combine quality and quantity assessments where appropriate

Simple quantity metrics may be appropriate for MBIs such as water cap-and-trade schemes. A conservation auction or offset scheme may be interested in the extent, type, condition and conservation significance of vegetation or other biodiversity indicators. Indicators that accurately represent the ‘quality’ of vegetation are incorporated and usually multiplied by some function of the change due to management and the spatial extent of management to produce the metric.

The spatial arrangement of goods and services in the landscape is an important component of quality and has an influence on their value in the market. The location where NRM outcomes are achieved will have different benefits in terms of ecosystem services provided. Metric design should account for the location of goods and services to ensure appropriate benefits are attributed where: a biophysical threshold is reached or impacted (more benefit may be gained in some locations); the ecosystem service benefits are remote from the site of action and the path between action and impact is important, and; some groups of goods and services provide greater benefits if other groups are also in the market (e.g. if neighbours both protect and manage adjacent lots of native vegetation).

Provide an objective, reliable and repeatable measure of goods and services

Simple quantity metrics may be appropriate for MBIs such as water cap-and-trade schemes. A conservation auction or offset scheme may be interested in the extent, type, condition and conservation significance of vegetation or other biodiversity indicators. Indicators that accurately represent the ‘quality’ of vegetation are incorporated and usually multiplied by some function of the change due to management and the spatial extent of management to produce the metric .

When goods and services are traded through an MBI, both the buyer and seller are interested in how observable the changed management and/or outcomes are. The quantity and quality of natural resources are usually variable in space and time and measurement needs to account for this variability. A metric designed to measure the quantity and/or quality of a natural resource or likely changes from management needs to be reliable and repeatable enough to detect expected or important changes despite the variability. To do this we must use objective measures of the indicators in the metric and have reasonable confidence of the accuracy and precision of our measurement. If our measurement methods are subject to error or dispute we may not be able to verify the quality of goods and services; select between alternative offers in the market; provide confidence to market participants that the MBI is fair or that the environmental assets described by the metric will hold their value; or observe (measure) outcomes even if they occur.

Metrics often need to: use available data even when it is limited; collect data on surrogates when direct measurements are impossible or too costly; weight different attributes according to priorities and best available knowledge of the system; and estimate the outcomes from practice change even when empirical data are lacking or limited. If we are clear about our process and invest wisely in new data collection we can overcome many of these constraints. For the rest we need to tailor the metric to ensure it is defensible. 

Metrics should be designed to allow for improvement when new information, scientific understanding or analysis techniques become available. This can be facilitated if metrics are constructed as small modular units which can be upgraded without the need to overhaul the entire metric or MBI design. A good example of this is the Victorian EcoTender metric which has four component metrics for change in four environmental services. If one of the component metrics is improved with new information or science, it can be substituted for the old component metric without disturbing the other component metrics.

Defensible if data is limited or uncertainty high

The task of designing a metric is made more difficult when data are limited or there is high uncertainty about the appropriateness of surrogates or the causal relationships between management actions and natural resource outcomes. There may also be uncertainties from external factors such as climate variability, commodity prices and farm profitability, or even labour markets and the availability of labour or technical assistance. Designing a metric is an exercise in producing the ‘best’ metric for the MBI being used, though it may not be ‘perfect’, or perfectly technically or scientifically robust. The use of an MBI (and its associated metric) is a choice between policy instruments (e.g. regulation, information provision), all of which have imperfections and may have inefficiencies and uncertainties which are not explicit.

Some remedy to the problem of producing a good metric when information is inadequate can be found by reducing uncertainty through appropriate use of expert knowledge , data analysis and modelling. Collection of new data may be required to support the design of a new metric and opportunities may be available to collect data with uses in other programs and purposes. Data collection can be one of the most expensive components of metric design and use and decisions about the type of data necessary should consider the likelihood of new data reducing uncertainty to an acceptable level.

Other solutions include modifying a metric which has been developed and tested in a data-rich environment, reduce the relative weighting of metric attributes with high uncertainty, or seek cheaper surrogates to measure and thereby trade off some metric fidelity for greater certainty.

Simple to understand and explain to applicants and transparent enough to allow fairness

To provide guidance and confidence to participants in an MBI it is necessary to be able to explain what the program aims to achieve, what is preferred and what the priorities are. If a metric is complex it may make communication more difficult. The aim is to ensure the metric is easily communicated by keeping it simple (without limiting their power to evaluate goods and services and discriminate between offers in the market) and/or concentrating on communicating how different environmental goods and services are included in the metric. If the metric is complex it should still be possible to explain the components of the metric and the relative contribution of each component.

Metrics do not need to be completely transparent to all stakeholders; there is some evidence that revealing all information about how the metric scores and weighs attributes may reduce the cost-effectiveness and efficiency of an MBI program. However, participants in the program have a right to expect that their goods and services will be fairly appraised and that they are not unfairly disadvantaged in the market.

Communications about the metric and how it works should be informed by program objectives, the design of the MBI and metric, and a good understanding of the market.

Enable manageable calculation of net change or outcomes

MBIs for NRM are interested in maintaining and/or improving the extent and/or condition of natural resources by encouraging appropriate management actions. These mechanisms evaluate the expected gain in quantity and/or quality of the resource and ascribe greater benefit to greater gain. The metrics used to do this usually include a component of current benefit or significance of the natural resource goods and services (the baseline), ensuring that value is determined based on the relative change from this position (maintaining or improving the quantity or quality of goods and services). To calculate net change or outcomes from management actions we need to calculate the likely outcomes from different suites and combinations of eligible actions and attach weights to these outcomes.

We calculate outcomes from management actions by analysing or modelling existing data, drawing on expert knowledge and opinion, and by collecting new data to demonstrate the causal links between actions and outcomes . When metrics are complex or rely on complex models, we need to test the metric for sensitivity to different values for the attributes included. This can be done by using computer-sensitivity analysis or by testing a number of realistic scenarios and seeing how the results match our expectations. If computational requirements or expertise are significant for the metric, capacity should be available to test and use the model when required.

When metrics predict the outcome from management actions it is difficult to design contracts for the delivery of the goods and services because certainty and observability may be low. Outcomes may also not be evident within the timeframe for the program. These issues mean that we may not be able to contract market participants to outcomes (e.g. recharge reduction) but may use outputs (e.g. area fenced, percentage of groundcover maintained) instead. Expected outcomes used to construct the metric for change need to strongly reflect the outputs used for contracts. In this way contracts will be fair, protected from dispute and enforceable.

Ensure design and implementation are cost-effective

The costs associated with metric design and implementation can account for a large proportion of the program costs for an MBI. These costs are incurred from: data collection (including expert input), analysis and modelling to parameterise the metric; data collection and management to populate the metric during implementation; and metric use. While it is desirable to have low costs for these activities, it is more important for the implementation to be cost-effective, that is low cost relative to the scale and budget of the program and relative to alternate approaches (i.e. non-MBI approaches such as regulation or extension). The costs of metric design and implementation may be justified if subsequent programs will use the metric and operate more cost-effectively as a consequence of the earlier work. Where metric development costs include the cost of additional data collection, there may be additional benefits for other planning and action programs.

Costs can be minimised by: using an existing appropriate metric  including training and other supporting materials; minimising field data collection or designing data collection protocols which can be rapidly and cost-effectively applied by people with a wide range of NRM expertise; adapting existing data management systems and databases; and making decisions about the level and detail of data required to achieve an objective, reliable and repeatable metric.

Allow comparisons and discrimination between a range of realistic scenarios or groups of goods and services

When a metric is calculated for different goods and services it is desirable that different groups of goods and services should be able to be discriminated. Different suites and combinations of goods and services may achieve the same metric scores if they represent the same level of overall outcome. If the outcomes are different, the metric should be sensitive enough to detect the difference. Sensitivity can be tested by repeated calculation of metrics with different combinations of scores for the different components of the metric. This can be done using sensitivity-analysis software tools which allow many calculations of many combinations of the different attribute scores. Sensitivity analysis can also be done by describing scenarios that should produce similar scores or very different scores and calculating the metric scores. In this way the metric calculations can be examined against an intuitive understanding of the priorities and weights.

A useful exercise is to consider two participants in the market (perhaps neighbours) who have almost the same goods and services to offer but perhaps with small differences. The metric scores should be quite similar and make some intuitive sense as to which participant offers the higher benefit. Sensitivity analysis effectively tests hundreds or thousands of scenarios like this and where the differences extend to their maxima.

Consider the reversibility of impacts, side effects, market interactions and chance of success

The reversibility of impacts, potential for side effects and the chance of success are all related to the level of risk involved in achieving the desired outcomes from an MBI program, without associated perverse outcomes. Risk can be considered the product of the likelihood of something happening and the consequence if it happens. Events with high likelihood and/or significant consequence are considered to be high risk. For example, if a species is threatened with imminent extinction, it is probably a riskier strategy to undertake a revegetation program for habitat restoration than to translocate some of the species to more secure habitat or start a captive breeding program. MBIs are not the solution to all problems and should take account of the likelihood of success from the actions implemented. Some metrics have been specifically designed using a risk analysis framework.

A risk of negative side effects may occur, for example if the metric focuses on achieving gains from a limited set of ecosystem services, which in themselves may cause other natural resources to be impacted. For example, an MBI which promotes revegetation and alters catchment hydrology may compromise the ecosystem services provided by catchment flows. More recently, metrics and MBIs have been considering a broader range of ecosystem services and using multi-metrics to incorporate scores for the range of services of interest.

An MBI may seek to value and purchase environmental benefits for which there are already whole or partial markets. MBIs focused on revegetation and habitat restoration will lead to sequestered carbon, an environmental service for which there is emerging markets. Metric design should take account of whether components of the goods and services are already valued in existing markets and how the MBI interacts with those markets to maximise the benefits.

Take into account trigger thresholds or synergies that would have a major impact on the desired outcomes

Ecosystem services and the ecosystems which provide them do not always operate in continuous or linear ways. For example, a stream needs a certain amount of flow before all the pools will be joined up. Once the ‘threshold’ flow is reached, a whole suite of ecosystem services are produced which were not being produced before; for example fish populations can mix and breed, waterborne seeds can be dispersed, and billabongs can be flushed with fresher water.

An MBI can be designed to take account of known thresholds and optimise the benefit from them (or avoid them if they are negative). This can be achieved by creating an assessment framework or rules in the MBI which account for trigger values or thresholds. Alternatively, an MBI can use the components of a metric in a dynamic interaction with the market to benefit from thresholds through ‘complementarity’. Complementarity describes the interdependencies between the groups of goods and services available, that is the value of a group of goods and services may depend on which other groups of goods and services are available in the market. In practice this means that funding one project in an MBI changes the relative value of other projects. As groups of goods and services are taken out of the market, the value of other groups changes and the weighting of attributes in the metric may change to reflect this. A number of MBIs have incorporated complementarity. For example, an MBI focussed on maintaining and improving the ecosystem services from intact native vegetation may include an attribute for location of sites with respect to other native vegetation (some measure of landscape context). The benefit of investing in managing a site near to other patches of native vegetation may be considered higher than investing in isolated patches. However, the benefit of investing in a managed site may be even greater if it is near other managed sites (management funded by the MBI program). The landscape context score for a site may depend on which other nearby sites are being funded. There is complementarity between the sites and a fixed value in the metric for landscape context would not recognise benefits which depend on prior selection of other sites for funding.

Consider time lags for outcomes to be realised

Timing of outcomes from a program has an influence on metric design because it affects prioritisation (earlier outcomes are usually preferred over later ones) and because it impacts on the design of contracts. There may be uncertainty about the timing of outcomes from management and some may be likely to be produced after the program or contract period has ended. For this reason, many metrics estimate the likely outcomes within the life of the program or a specified period after the program has officially ended, but contract on the basis of outputs. By contracting on the basis of outputs it is possible to have more certainty about progress in the program implementation but perhaps less certainty about program achievements. Contracting for outputs requires the development of strong causal relationships between outputs and outcomes.

Be clear about what it does not measure

One way to look at what the metric does measure is to define what it cannot measure or should not be used to measure. Most metrics focus on a defined set of indicators to be measured and often measure surrogates for complex system characteristics (e.g. surrogates for species-level biodiversity status, which would be difficult and expensive to measure). A metric may not: measure the impact of management actions below a given spatial resolution; may not be appropriate for some sites; or may not measure attributes of interest even about a defined quality (e.g. metrics assessing vegetation condition sometimes include measures of tree habitat—hollows—and sometimes do not). If substantial investment is made in design and implementation of a metric there may be interest in using it for other purposes. An explanation of what the metric can and cannot be reliably used for can prevent misuse.

A number of authors (Parkes et al., 2003; Oliver et al., 2005; Whitten, 2005b; and Rolfe, 2007; Whitten et al., 2007) have described frameworks for metric design which combined incorporate the principles:

• focus on program objectives and priorities
• combine quality and quantity assessments where appropriate
• provide an objective, reliable and repeatable measure of goods and services
• be simple to understand and explain to participants, and transparent enough to demonstrate fair decision making
• enable manageable calculation of net change or outcomes
• ensure design and implementation are cost-effective
• allow comparisons and discrimination between a range of realistic scenarios or groups of goods and services
• be defensible if data is limited or uncertainty high
• consider the reversibility of impacts, side effects, market interactions and chance of success
• take into account trigger thresholds that would have a major impact on the desired outcomes
• consider time lags for outcomes to be realised.

An additional consideration for metrics is to be clear about what is not measured. Once investment has been made in the framework, data collection, storage and analysis for a metric there can be temptation to use the metric for other purposes, some of which may stretch the logic used to develop the metric in the first place. McCarthy et al. (2004) suggest a number of qualifications to the use of the ‘habitat hectares’ metric (Parkes et al., 2003) for purposes outside the original use. These include questions about the use of the metric as a tool for setting region-wide or statewide objectives, limitations for assessing sites with natural disturbance regimes, usefulness of comparing quality between different vegetation types, and use of the metric to evaluate conservation values in intensively cleared landscapes. If a metric is designed to support the MBI in achieving the defined objectives of the program, it may not be appropriate for other uses, or may require modification for additional use.

A common misuse of metrics for MBIs is the use of metrics designed for decision making (e.g. as the basis for discriminating between bids in a conservation auction) as the basis of monitoring programs. Monitoring may be best achieved using a subset of attributes of the metric, a combination of attributes not included in the metric or other indicators which are more sensitive or more cost-effectively measured and reported than the metric. The power of a monitoring design is critical to detecting the change of interest (Field et al., 2007) and may be higher if alternate indicators are used to those used in the MBI. The monitoring design should also balance power and cost-effectiveness (Field et al., 2004) and the appropriateness of the metric for monitoring should be considered in light of this constraint.

Table 1 illustrates how the basic principles of metric design were dealt with in a case study price-based MBI focused on reducing recharge to saline aquifers over a 10-30 year period.

Focus on program objectives and priorities

The attributes in a metric, how they are combined and weighted and the data available for use in the metric determine what the metric measures. To be effective in achieving policy and program objectives, a metric needs to be constructed to measure the right natural resource properties, processes and management outcomes to meet stated objectives. Where the endpoint of the program is not well-defined it is too easy to develop a metric based on available data or theory at the expense of targeting the key outcomes required (Failing and Gregory, 2003).

Oliver et al. (2005) describe the policy and program objectives which drove the development of a biodiversity metric for the NSW Environmental Services Scheme (ESS). The policy objective was to promote land-use or land-management change that improved the provision of a range of environmental services (Oliver et al., 2005). The types of environmental services that would be examined were determined first, and then the best ways to measure a range of environmental services were developed (Grieve and Uebel, 2003). Metrics also needed to provide a measure for use at the property, catchment and regional scales.  

The BushTender trial (Stoneham et al., 2003) had a policy objective of establishing contracts for nature conservation with private landholders. The metric used in BushTender (including the Habitat Hectares component (Parkes et al., 2003) focussed only on measurement of the biodiversity significance, management improvements and cost of individual bids in a conservation auction. The same metric has subsequently been combined with metrics for other environmental services to achieve a broader set of policy and program objectives from the EcoTender trial (Eigenraam et al., 2008). EcoTender has the extended policy objective of cost-effectively establishing contracts for provision of several jointly supplied environmental services (for improvements in terrestrial biodiversity, aquatic function, saline land area and carbon sequestration). The policy driver for EcoTender extends that of BushTender to capture a greater range of environmental services but also to avoid perverse outcomes where improvement of provision of one environmental service has a negative impact on the provision of other services, especially where these services are undervalued by existing markets. While there are many advantages from designing and implementing metrics and MBIs covering multiple environmental services is not always easy and programs and policies need to be guided by these challenges (e.g. Gole et al., 2005).

Combine quality and quantity assessments where appropriate 

Metrics for MBIs need to provide an objective, reliable and repeatable measure of the goods or services. Different NRM issues require different metrics because the goods and services differ and may be simple quantities (e.g. megalitres of water) or complex qualities (e.g. condition of native vegetation) of the natural resource. Measures of quality often combine a number of attributes which represent the composition or function of the natural resource.

While quantity metrics may be simple, they will not always be simple to calculate. Whitten et al. (2005a) used complex biophysical modelling to measure recharge (megalitres) at different scales (from the paddock scale to the region scale). Measurement of recharge required a number of estimations and judgements including the estimation of diffuse source net recharge to shared irrigation aquifers and the identification of net recharge impact boundaries (Whitten et al., 2005a).

Gibbons and Freudenberger (2006) reviewed indices of vegetation condition and reported that there is no standard definition of the ‘quality’ of vegetation but that quality had been defined in different ways according to use. The assessment of quality is often a matter of reliance on theory, expert knowledge and the measurement of surrogates. The River Murray Forest Project in South Australia derived a metric for a trial single-sealed bid reverse auction for revegetation with both carbon sequestration and biodiversity conservation values. This simple metric converted continuous data to categories for the diversity of species to be planted, the distance to and size of nearby remnant vegetation, the total and individual patch size of plantings, and measures of the carbon sequestration potential and security (O’Connor and Collard, 2007). More detailed datasets and more sophisticated models of the system could include additional attributes and combine them in different ways. However, the cost-effective solution of using simple attributes for surrogates of biodiversity and carbon-sequestration values enabled the market for revegetation to be tested ahead of more detailed metric availability.

The assessment of quantity and quality may rely on surrogates, estimates or models and the assumptions underlying these should be documented along with the metric. An example of this is the use of spatial data either for quantity (e.g. hectares of native vegetation) or quality (e.g. percent remnant native vegetation—a potential component of a landscape context metric). The requirements for standard but sophisticated spatial analyses of attributes such as landscape context should not be underestimated (Gole et al., 2005). The use of models such as those underlying geographic information systems (GIS) should also be understood and the accuracy of models should be tested and reported.

Provide an objective, reliable and repeatable measure of goods and services

Failing and Gregory (2003) highlight the problem of developing lists instead of indicators when trying to describe the condition of ecosystems. There is a tendency to use indicators for which there is already data or well-developed understanding of a process rather than high representation of the core attributes of the system which are important in the specific management context (Failing and Gregory, 2003). Indicators (or attributes in a metric) will be most useful when they are selected as appropriate measures or surrogates for key properties of the system, after the program objectives have been defined.

A large number of metrics measuring environmental assets, processes and outcomes have been developed for different purposes. However, not all of these metrics have been tested for reliability and where they have been tested some inconsistency has been found. Cushman et al. (2008) examined a large set of metrics to determine if there is a parsimonious set of metrics which reliably measure landscape structure. The study showed that few metrics were universally appropriate for all use in all locations (only eight out of 54 metrics measured attributes present in all the locations tested), and only seven out of 49 components of landscape structure explained more than 10% of the variance of the metrics tested (i.e. many of the metric attributes were not very important in many of the locations tested). This study demonstrates that metrics may be robust in one location or for one set of examples but should be tested for new use in new applications and locations. Metrics need to be tested for attribute redundancy to ensure they are reliable and consistent (Cushman et al., 2008).

Andreasen et al. (2001) provide an example of the issues which need to be addressed in developing an objective, reliable and repeatable metric. They describe considerations for developing a metric to measure ecological integrity in terrestrial systems. Key considerations include the need for the metric to be meaningful at different spatial scales, grounded in natural history, relevant and helpful, flexible, measurable and comprehensive enough to measure composition, structure and function. Andreasen et al. (2001) also suggest the information useful for evaluating a candidate metric, combining attributes into a metric, and testing a metric.

Attributes in a metric also need to be weighted to reflect their relative importance. The weightings need to take account of the relationship of attributes to each other and to the end-point or objective of the metric/MBI. Weighting is a complex task and must be done using reliable and defensible methods. Some of these methods are discussed later in this summary (e.g. see the section on dealing with limited data and uncertainty below) and others can be reviewed in texts such as Clemens (1996) and von Winterfeldt and Edwards (1996).

The methods used to combine attributes in a metric are also important and need to be considered to avoid redundancy (where different attributes measure part or all of that measured by another attribute) and compensation (increase in one attribute masks the decrease in a related attribute). Caution is recommended when adding attributes together in a quality metric due to the risk of compensation of one attribute for another. The example given from McCarthy et al. (2004) illustrates that the loss of trees may be compensated by the increase in coarse woody debris in a metric which combines the two (the example discussed is from the Habitat Hectares metric (Parkes et al., 2003). These two attributes are not equivalent or interchangeable but an additive metric may not distinguish between two cases where the decreasing value of one attribute is compensated by the increase in the other. Weighting attributes or multiplying attributes together can help overcome the problem of compensation between attributes. The problem of arriving at a zero value for the metric, if only one attribute has a zero value but the other attributes do have meaningful values, is avoided if the minimum value of each attribute is set above zero (McCarthy et al., 2004). In practice, a mixture of addition and multiplication is usually appropriate (Gibbons et al, 2005).

Be simple to understand and explain to applicants and transparent enough to allow fairness

MBIs aim to establish contracts between parties which optimise the environmental benefit from investment. A number of authors have highlighted that market efficiency is affected by information asymmetry (Laffront, 1990; Latacz-Lohmann and Van der Hamsvoort, 1997) and that land managers may not have all the relevant information about government priorities in an MBI or how information about priorities would influence contracts established through the MBI. One mechanism for defining and quantifying priorities is the MBI metric. Communication about the program objectives and priorities must go beyond explanation of the metric; however, a metric which is easily explained will assist the communication. Parkes et al. (2003) and Oliver et al. (2005) both highlight the need for the metrics developed in their programs to be simple to understand. This does not mean that all attributes and their scoring will necessarily be understood by all participants, but that land managers can be provided with a clear message about the important components of the environmental services which are sought through the MBI (and metric).

Overly simple metrics can be criticised as not representing all possible contributing attributes of the system or of not being ‘scientifically’ defensible. Scientifically defensible in its simplest form means that the logic and data used to construct the metric come from a systematic and falsifiable approach to decision making. Where judgements need to be made to accommodate uncertainty and knowledge gaps, the judgements should be made on the best available evidence and programs implemented to examine outcomes and improve the metric (Failing and Gregory, 2003).

While providing clear information about metric components can improve the efficiency of the market, it is not necessary to reveal all the weighting and scoring of a metric. Cason et al. (2003) have shown that landholders may misrepresent their costs if they know they are offering high-quality benefits and the total environmental benefit from a fixed MBI budget may be lower. Metrics and MBIs can still be designed to be simple to understand and explain while absolute scoring and weighting processes may remain hidden if it is though market efficiency will be improved (Stoneham et al., 2003).

Gole et al. (2005) report that landholders involved in the Auction for Landscape Recovery (WA) valued contact with community support officers who were able to explain the MBI program to them. They also report that some landholders experienced initial difficulty working with the design of the program which may have reflected difficulties in defining and communicating the objectives of the program and how multiple benefits would be assessed.

Enable manageable calculation of net change or outcomes

Stoneham et al. (2003) and Oliver et al. (2005) describe metrics designed to incorporate the current status of environmental services and the predicted change following management change. Both these metrics require some estimation of a transformation function—the estimated change from current value to future value following intervention (this could include prevention or slowing of decline as well as improvement in the condition of the natural resource of interest). Reliable scientific information on the nature of the relationships between land-use change and ecosystem impacts is critical for the functioning of environmental markets (Whitten et al., 2005a). Stoneham et al. (2003) recognise that the transformation function for the BushTender trials was not known with certainty and that extreme or unexpected events are difficult to include in estimated functions.

The EcoTender program has taken the calculation of change another step by implementing a catchment modelling framework (CMF) to estimate multiple environmental benefits from revegetation and remnant native vegetation maintenance and improvement (Eigenraam et al., 2007). The CMF was able to measure a change in the level of ‘service’ provided by the landholder for the multiple environmental benefits sought by the program. The program recognised that some on-farm management actions could be incorporated in the future but further research is required to determine appropriate monitoring and enforcement strategies.

A key impediment for the Auction for Landscape Recovery (ALR) was the inability to develop or use effective estimates of future management benefit of tendered projects and threat/risk analysis (Gole et al., 2005). Methods already used in other projects were considered of questionable value to the ALR, because they did not transfer well to the new environment. Gole et al. (2005) recommended a dedicated research program to develop transformation functions and provide workable and meaningful methodologies.

Most work in this area has relied on biophysical models (e.g. Whitten et al., 2005a; Connor et al., 2006) or expert knowledge, with or without the use of MCA (e.g. Stoneham et al., 2003; Oliver et al., 2005). Efforts to document changes resulting from large agri-environment schemes which have spent billions of dollars in Europe and North America have been patchy and produced mixed results (e.g. Kleijn et al., 2006). MBIs with a priori assessment of natural resource condition, description of changes sought through management (i.e. management plans and contracts) and adequate monitoring programs (see Field et al., 2007, for a discussion of key issues for designing meaningful monitoring) offer an opportunity to evaluate and improve transformation functions and metrics.

Because outcomes may not be realised for a significant amount of time and can be difficult to monitor or observe (e.g. status and resilience of plants and animals) many MBIs use metrics which determine expected gain from intervention but establish contacts on the basis of outputs. Stoneham et al. (2003) and Windle and Rolfe (in press) discuss the issue of contracting on the basis of outputs but estimate outcomes in the metric to discriminate between bids in the respective conservation tenders. Another example is the land management changes required by the Wimmera Steep Hill Country (SHC) case study to reduce recharge to a saline aquifer over 10–30 years require expensive upfront investment in order to produce ecosystem services in the longer term (Whitten and Shelton, 2005). However, the success of these actions can only be measured at a much later date, thus compliance is difficult to measure. The relatively large upfront capital investment required is likely to be beyond the capacity of most land managers and so a payment schedule based on significant upfront support and ongoing performance-based payments based on management inputs is likely to be needed. In this example the metric may be accurate in predicting the change expected from land-management change but the program proponent may still bear the risk of failure and it will be difficult and potentially costly to monitor compliance.

Ensure design and implementation are cost-effective

There are few reports of the total or itemised costs of metric design and implementation. Costs can also be difficult to calculate when metric design and implementation costs are not easily separated from other costs associated with MBI design and implementation and when institutional or in-kind support is provided but not accounted for in metric development. However, metric design and MBIs are not unique in this regard as cost accounting for non-MBI approaches to NRM can also be limited by similar problems. Windle and Rolfe (in press) demonstrated that while the initial design and development costs of a conservation tender in the Australian rangelands were higher than a grant scheme, there was no evidence of any difference in operating costs between the grant and tender-based schemes in their case study.

Both the design and implementation of metrics need to be taken into account when estimating cost-effectiveness as metrics with low data collection and analysis costs will be more cost-effective. Administrative costs can account for 30%–80% of the total cost of an agri-environmental scheme (Falconer and Whitby, 1999) and metric costs are can be a significant component of the administrative cost. Data collection is often expensive and some programs have tried to limit these costs by designing metrics which require lower levels of expertise and less time to collect the data (e.g. Oliver et al., 2007). Rapid approaches can be very cost-effective if the methods remain objective, reliable and repeatable. Beverly et al. (2005) compared detailed and rapid assessment methods for predicting salinity impacts from land-use change and found that only the complex models could identify the changes of interest at the paddock–farm scale required by the program. 

Lowell et al. (2007) report that model-calibration costs for an MBI focussed on the establishment of forest plantations were approximately nine times usage costs, though subsequent calibration costs in the same catchments would be lower as a consequence of the initial work. These authors also report that the model calibration and usage costs constituted around 30% of the total operational cost (i.e. administration, modelling, on-ground activities) of the MBI and 20% of the total cost of the program. In the case of the ALR (Gole et al., 2005), the estimate of administrative costs, including all research and operational costs, was around $500,000 and with an expenditure on transfer payments to landholders of around $200,000 administrative costs account for 70% of the total (although this project was a fixed-budget MBI pilot project).

Metric design and implementation will be more cost-effective if subsequent programs can use the metric and operate more cost-effectively as a consequence of the earlier work. There will usually be additional benefits for planning from the availability of additional data and this will improve the overall cost-effectiveness of metric implementation.

Allow comparisons and discrimination between a range of realistic scenarios or groups of goods and services

A number of MBI projects have conducted pilot programs or case studies as part of the testing the MBI design and metric. For example, Eisner et al. (2007) conducted a pilot program of using 10 applications in a competitive-tender MBI aimed at changing agricultural management practices in the Great Barrier Reef catchments to reduce sediment and nutrient loads impacting on the reefs. The metric was able to distinguish between different management-practice changes within a single agriculture type and between different sites within a sub-catchment.

The Catchment Care competitive auction for biodiversity conservation and water quality gains tested the metric under the full range of auction conditions before applying it in the trial, and made adjustments to the metric to ensure it was selecting bids consistent with the goals of the program (Bryan et al., 2005). The tests used created 1000 bids with a complete mix of plausible scores for all metric attributes (and groups of attributes). Stepwise multiple linear regression was used to identify attributes that most strongly influenced the metric score and the metric was refined after simulations to better reflect the goals and priorities of the program (Bryan et al., 2005).

O’Connor et al. (2007) and O’Connor and Collard (2007) also conducted sensitivity analysis of metrics designed to discriminate between bids in a conservation tender and a tender for revegetation contracts, respectively. These sensitivity analyses allowed examination of the relative importance of attributes in the different metrics and assessment of how different potential bids would rank at the end of a competitive tender. Both projects used simulated datasets to examine the ability of the metrics to compare different groups of goods and services.

Be defensible if data is limited or uncertainty high

Data for decision making is often lacking and decisions often need to be made in an environment of uncertainty. Uncertainty in metric construction and use can arise from many sources but commonly arises from uncertainty about measurement error, model and model input parameter errors, spatial variability, errors in spatial data, the effects of aggregation of spatial data, temporal variability, the inherent variability in natural processes, model and model parameter uncertainty (Nguyen et al., 2006).

There is an increasing body of literature addressing decision making in uncertainty which is relevant to metric design for MBIs. A number of tools including decision tables and decision trees have been proposed for formal decision making that involve identifying three main components: acts, states, and outcomes (Resnik, 1987). The acts refer to the decision alternatives, the states refer to the relevant possible states of the system, and the outcomes refer to what will occur if an act is implemented in a given state. Ben-Haim (2001) advises that key to decision making in uncertainty is judicious use of available information, calculation of the consequences of incorrect decisions (both undesirable outcomes occurring or desirable outcomes being missed), the incorporation of value judgements and the preparedness of the decision-maker for risk-taking.

Eisner et al. (2007) note uncertainty in their metric measuring benefits from changing agricultural management practices in the Great Barrier Reef catchments to reduce sediment and nutrient loads impacting on the reefs. They note the uncertainty comes from both limitations and assumptions in the underpinning conceptual framework of the metric and from shortage of appropriate data to populate the metric. Eisner et al. (2007) used an expert panel to estimate the nutrient and sediment load reductions from different combinations of sugarcane management practice. They recognised that the disadvantages of using the expert panel instead of a model were potential lack of transparency of decisions and bias in the expertise available. However, benefits included some reduction in uncertainty by producing a highly flexible assessment system in which any combination of practices used by a farmer could be assessed (this capability was not available from existing models). Components of the system which had uncertainty that was considered too high were omitted from consideration in the metric (i.e. estimation of loss of nitrogen to the atmosphere). Eisner et al. (2007) also dealt with uncertainty in some metric attributes by expressing values for those attributes as ranges rather than as means.

Another approach to dealing with uncertainty is preparedness for trial and error and communication and discussion of findings. Oliver et al. (2005) recognised that data collection for their metric estimating biodiversity benefits from changed land use and management introduced some uncertainty through inter-operator variability (observer bias). These authors have designed their metric to enable scrutiny of the expert opinions on which they are based and the authors invite feedback on the method.

Multi-criteria analysis (MCA) has been used in a number of projects to combine and weigh components of a metric. Oliver (2002) and Oliver et al. (2007) facilitated expert panels to develop a set of indicators of vegetation condition based on their importance and feasibility using an analytic hierarchy process (AHP), a form of MCA. Hajkowicz (2006) also presents a framework suitable for using expert and other input to produce metrics for a range of purposes. These examples deal with uncertainty by allocating weighting to attributes using available data, knowledge and opinion in a transparent and defensible way. When this approach is combined with thorough uncertainty analysis (see section above that discusses discrimination between a range of realistic scenarios), uncertainty can be better understood and accounted for. Reckhow (1994) suggest the use of simple models with thorough uncertainty analysis where uncertainty is high, because complex models may be difficult to scrutinise (the Information Paradox of Rowe (1977).

Not all reports on metric design and MBI implementation report enough information about the construction of the metric, weighting of attributes, and approaches to dealing with uncertainty to permit scrutiny and learning by others. This is an issue for future learning, and MBI practitioners should be encouraged to document and report the assumptions and limitations of the metrics they design and use to allow further enquiry and development around problems of uncertainty.    

Consider the reversibility of impacts, side effects, market interactions and chance of success

Where there is a risk of irreversibility there is a case for favouring less risky actions that achieve the change sooner (Whitten and Shelton, 2005). Metrics can be designed to take account of irreversible impacts, though this is often better dealt with through the rules established in the MBI design. Oliver et al. (2005) showed how land-management changes which led to difficult-to-reverse impacts on derived native grassland condition produced very low (negative) metric scores (Land Use Change Impact Score). Depending on the regulatory framework underpinning the MBI, the same irreversible impacts could have dealt with by not allowing the proposed negative land management change at all (regulation), or ruling land managers undertaking that management change out of the MBI. Metric design should ensure that unwanted irreversible impacts are not promoted or rewarded in an MBI.

Environmental benefits from natural systems are usually linked and cannot always be managed separately. Wu and Skelton-Groth (2002) demonstrated that the failure to recognise multiple benefits from funding for conservation programs could result in the loss of some of the conservation benefits. Strappazzon et al. (2003) argued that where multiple environmental goods were produced from an action, funded programs should concentrate on those goods for which there are not existing trading systems. This has been demonstrated in practice by Eigenraam et al. (2007), who conducted the EcoTender in Victoria to achieve multiple benefits for terrestrial biodiversity, aquatic function, saline land and carbon sequestration from revegetation and remnant vegetation management. The tender-based MBI calculated environmental benefits for the first three of these outcomes, with carbon sequestration benefits treated as though there were an existing market for that service. EcoTender considered how the multiple benefits from supported actions could be measured, but took the approach that where the program interacted with an existing (or likely) market for carbon sequestration, it may best to allow that market to access the carbon sequestered through the program (landholder were also able to sell their carbon sequestration units to the program at a fixed price, as though that was the ‘market price’ in an established market). EcoTender demonstrates the importance of purchasing multiple environmental outcomes where a focus on only one of the outcomes could produce perverse results with respect to another; for example, purchasing only the saline land benefits through revegetation may result in declines in aquatic functions if water tables fell too far.

Take into account trigger thresholds that would have a major impact on the desired outcomes

Trigger thresholds are quantity or quality states that, if reached, will result in non-linear improvement or decline in a defined value. In simple terms this could equate to ‘the straw that breaks the camels back’ or saving enough money for an overseas trip—all amounts less than the threshold do not trigger the outcome. In natural resource management it may be possible to change management on a number of properties but it will only be after a specific property joins the program that the results are really noticeable. An example of this would be conservation works on remnant vegetation patches that are widely separated in a fragmented landscape. While each patch may improve due to management, potential gains from species movements between sites (dispersal) may not occur until an important connecting site in the landscape is managed for conservation. An example of this is the MBI designed by Rolfe et al. (2005) to improve landscape linkage for biodiversity in the southern Desert Uplands of Queensland. The MBI design recognised the different values of investment in different parts of the landscape and introduced a ‘limited cooperation’ model to improve landholder involvement in determining locations for remnant vegetation corridors and so achieve higher order outcomes than would have been achieved if individual properties submitted bids in a conservation tender without awareness of the greater gains from conservation on contiguous patches of remnant vegetation.

The potential to benefits from optimising contract selection in MBIs has been taken further by Gole et al. (2005) in the Auction for Landscape Recovery in Western Australia. This project recognised that there could be strong interdependencies between the potential contracts for conservation work being funded. Bid interdependencies mean that the decision to fund any one contract changes the benefits score for other projects. For example, when the goal is regional biodiversity conservation, the biodiversity contribution of a contract reflects only those components of biodiversity not yet captured by other funded projects (the principle of ‘complementarity’, see Margules and Pressey, 2000). Hajkowicz et al. (2007a) re-analysed the data from the Auction for Landscape Recovery using different models of complementarity and showed that the optimal outcome for conservation was not selected by simple bid ranking but depended on the rules used to determine complementarity. Despite the additional benefits of more sophisticated methods for selections contracts to capitalise on complementarity, Gole et al. (2005) report that the challenges of communicating the process and its conceptual and computational complexity was a key limitation of the project.

Consider time lags for outcomes to be realized

Transition function must reflect the change within the time specified in the contract or the expected gains may be reduced if land management or use change after the contract period finishes and before the expected gains are realised. For some MBIs the time lag may be simple (e.g. the benefits from a cap and trade on water should be realised within a short period). For other MBIs the time lag may be difficult to determine or not occur for many years (e.g. carbon sequestration will increase until trees reach maturity). Published reports have not clearly specified the expected timing of outcome achievement or the types of contracts and whether contracts remain in force until the expected time of outcome achievement. For example, the BushTender metric includes a services score attribute for retaining dead trees and logs (not removing firewood) and contracts in the BushTender trial were established for actions over three years. It is not reported whether the outcomes from management services such as retaining dead trees and logs are expected to be realised in three years (Stoneham et al., 2003). The time lag to expected outcomes also means that services scores contracts of different length (within the same MBI) should be weighted according to the benefits likely after each time period. Benefits will rarely accrue linearly with time. Because the type, amount and timing of changes from management intervention are difficult to predict and assess, many MBIs have established contracts on the basis of outputs rather than outcomes (e.g. Stoneham et al., 2003; Rolfe et al., 2005; Connor et al., 2006).

Andreasen J.K., O’Neill R.V., Noss R. and Slosser N.C. (2001). Considerations for the development of a terrestrial index of ecological integrity. Ecological Indicators 1 21–35

Ben-Haim, Y. (2001). Information-gap decision theory. Academic Press, San Diego, California, USA

Beverly C., Bari M., Christy B., Hocking M. and Smettem K. (2005). Predicted salinity impacts from land use change: comparison between rapid assessment approaches and a detailed modelling framework. Australian Journal of Experimental Agriculture 45: 1453-1469

Bryan B.A., Gatti S., Connor J., Garrod M. and King D., (2005). Catchment Care – Developing an auction process for biodiversity and water quality gains. A NAP market-based instrument pilot project. CSIRO Land and Water and the Onkaparinga Catchment Water Management Board.

Clemen R. (1996). Making hard Decisions: An introduction to decision analysis, Duxbury Press, Pacific Grove, California.

Connor J.D., Clifton C., Ward J., and Cornow P. (2006). Commonwealth MBI Pilot Project: Dryland Salinity Credit Trade. National Market-based instrument Pilot Project (accessed at http://www.napswq.gov.au/publications/books/mbi/pubs/round1-project57.pdf)

Cushman S.A., McGaringal K. and Neel M.C. (2008). Parsimony in landscape metrics: Strength, universality, and consistency, Ecological Indicators (in press) doi:10.1016/j.ecolind.2007.12.002

Eigenraam, M., P. Barker, M. Brown, A. Knight, and S. Whitten. 2006. Forest Conservation Fund Conservation Value Index Technical Report. Assessment Methodology Advisory Panel, Tasmania

Eigenraam M., Strappazzon L., Lansdel N., Beverly C. and Stoneham G. (2007). Designing frameworks to deliver unknown information to support market-based instruments Agricultural Economics 37:261-269

Eisner R., Le Grand J., and Norman P. (2007). A water quality metric for the Great Barrier Reef catchments. Department of Natural Resources and Water, Queensland

Failing L. and Gregory R. (2003)..Ten common mistakes in designing biodiversity indicators for forest policy. Journal of Environmental Management 68: 121–132

Falconer, K. and Whitby, M. (1999). The invisible costs of scheme implementation and administration. In: Countryside Stewardship: Farmers, Policies and Markets, Van Huylenbroeck, G. and Whitby, M., eds., pp. 67-88. Pergamon, Amsterdam.

Field, S. A., Tyre, A. J., Rhodes, J. M., Jonzen, N. & Possingham, H. P. (2004) Minimizing the cost of environmental management decisions by optimizing statistical thresholds. Ecology Letters 7, 669-675.

Field S. A., P.J. O’Connor, A. J. Tyre, and H.P. Possingham (2007) Making monitoring meaningful. Austral Ecology 32; 485–491

Gibbons, P., Ayers, D., Seddon, J., Doyle, S. and Briggs, S. (2005) BioMetric Operational Manual Version 1.8: A Terrestrial Biodiversity Assessment Tool for the NSW Property Vegetation Plan Developer. NSW Department of Environment and Conservation

Gibbons P. and Freudenberger D. (2006) An overview of methods used to assess vegetation condition at the scale of the site. Ecological Management and Restoration. 7(S1):S10-S17

Gole C, Burton M, Williams KJ, Clayton H, Faith DP, White B, Huggett A, and Margules C (2005) Auction for Landscape Recovery FINAL REPORT. WWF-Australia. (accessed at http://www.napswq.gov.au/publications/books/mbi/pubs/round1-project21.pdf

Grieve A. and Uebel K. (2003). Developing New Income Streams for Farmers: NSW Environmental Services Scheme. (accessed at http://www.forest.nsw.gov.au/env_services/ess/files/essREPORT.pdf

Hajkowicz S. (2006) Multi-attributed environmental index construction. Ecological Economics 57: 122-139

Hajkowicz S., Higgins A., Williams K., Faith D. and Burton M. (2007a) Optimisation and the selection of conservation contracts. Australian Journal of Agricultural and Resource Economics. 51: 39-56

Kleijn, D., Baquero R. A., Clough Y., Diaz M., De Esteban J., Fernandez F., Gabriel D., Herzog F., Holzschuh A., Johl R., Knop E., Kruess A., Marshall E. J. P., Steffan-Dewenter I., Tscharntke T., Verhulst J., West T. M. and Yela J. L.. (2006). Mixed biodiversity benefits of agri-environment schemes in five European countries. Ecology Letters 9:243-254.

Laffont, J. (1990). The Economics of Uncertainty and Information, MIT Press, Cambridge.

Latacz-Lohmann, U. and Van der Hamsvoort C. (1997). Auctioning conservation contracts, a theoretical analysis and an application. American Journal of Agricultural Economics. 79: 407–418

Lowell K., Drohan J., Hajek C., Beverly C. and Lee M. (2007). A science-driven market-based instrument for determining the cost of environmental services: A comparison of two catchments in Australia. Ecological Economics 64: 61-69

Margules C.R. and Pressey R.L. (2003) Systematic conservation planning. Nature 405: 243-253

McCarthy M.A., Parris K.M., van der Ree R., McDonnell M.J., Burgman M.A., Williams N.S.G., McLean N., Harper M.J., Meyer R., Hahs A. and Coates T. (2004). The habitat hectares approach to vegetation assessment: An evaluation and suggestions for improvement. Ecological Management & Restoration 5:1, 24–27

Nguyen N., Woodward R.T., Matlock M.D., Denzer A. and Selman M. (2006) A guide to market-based approaches to water quality. World Resources Institute.

O’Connor P.J and Collard S. (2007). The River Murray Forest Score: a metric for determining value in bids for revegetation contracts under the River Murray Forest Project. Department of Water, Land and Biodiversity Conservation, South Australia

O’Connor P.J, Morgan A., and Bond A. (2007) BushBids: Eastern Mount Lofty Ranges Biodiversity Stewardship Initiative. South Australian Murray-Darling Basin Natural Resource Management Board

Oliver I. (2002) An expert panel-based approach to the assessment of vegetation condition within the context of biodiversity conservation: Stage 1: the identification of condition indicators
Ecological Indicators 1:223-237

Oliver I, Ede A, Hawes W and Grieve A (2005) The NSW Environmental Services Scheme: Results fot he biodiversity benefits index, lessons learned, and the way forward. Ecological Management and Restoration 6:197-205

Oliver I., Jones H., Schmoldt D.L  (2007). Expert panel assessment of attributes for natural variability benchmarks for biodiversity. Austral Ecology 32: 453-475

Parkes D., Newell G. and Cheal D. (2003). Assessing the quality of native vegetation: The 'habitat hectares' approach. Ecological Management & Restoration 4 (s1): S29–S38

Reckhow, K.H. (1994). Water quality simulation modeling and uncertainty analysis for risk assessment and decision making. Ecological Modeling, 72, 1-20.

Resnik M.D. (1987). Choices: an introduction to decision theory. University of Minnesota Press, Minneapolis, Minnesota, USA.

Rolfe J., McCosker J. and Windle J. (2005) Establishing east-west landscape linkage in the Southern Desert Uplands Research Reports (Research Report No. 6). National Market-based instrument Pilot Project (accessed at http://www.napswq.gov.au/publications/books/mbi/pubs/round1-project18.pdf)

Rowe, W. D. (1977). The Anatomy of Risk. New York: John Wiley and Sons.

Stoneham G., Chaudhri V., Ha A., and Strappazzon L. (2003). Auctions for Conservation Contracts: An Empirical Examination of Victoria's Bush Tender Trial. Australian Journal of Agricultural and Resource Economics 47: 477–500

Strappazzon L., Ha A., Eigenraam M., Duke C. and Stoneham G.(2003) Efficiency of alternative property right allocations when farmers produce multiple environmental goods under the condition of economies of scope The Australian Journal of Agricultural and Resource Economics, Volume 47: 1-27

Whitten SM, Coggan A, Reeson A, Gorddard R. (2007). Putting theory into practice: market failure and market based instruments (MBIs). Working Paper 2 in the Socio-Economics and the Environment in Discussion CSIRO Working Paper Series Number 2007-02. May 2007. ISSN 1834-5638. 46 pp.(accessed at http://www.csiro.au/resources/SEEDPaper2.html)

Whitten S.M., Khan S., Collins D., Robinson D., Ward J. and Rana T.. (2005a). Tradeable recharge credits in Coleambally Irrigation Area: Report 7 Experiences, lessons and findings. CSIRO & BDA Group (accessed at http://www.napswq.gov.au/publications/books/mbi/pubs/round1-project33.pdf)

Whitten S.M. and Shelton D. (2005b). Market for Ecosystem Services in Australia: practical design and case studies. CSIRO Sustainable Ecosystems. (accessed at http://www.cifor.cgiar.org/pes/publications/pdf_files/Whitten-Australia.pdf)

Windle J. and Rolfe J. (in press). Exploring the efficiencies of using competitive tenders over fixed price grants to protect biodiversity in Australian rangelands. Land Use Policy (in press) doi:10.1016/j.landusepol.2007.09.005

von Winterfeldt, D. and Edwards, W. (1986). Decision Analysis and Behavioral Research, Cambridge University Press, New York.

Wu J-J. and Skelton-Groth K. (2002). Targeting conservation efforts in the presence of threshold effects and ecosystem linkages Ecological Economics 42: 313-331

Metrics define and measure goods and services and a given metric should be able to be used in a range of different MBIs. Once environmental benefits can be defined and measured in a metric, the metric should be able to be used in the most effective MBI available. However, different metrics for the same goods and services have been designed for use in individual projects, locations and MBIs because each project has a different starting point, different requirements for metric design and cost-effectiveness, and different theories of how actions link to outcomes. Some metrics have been used in more than one MBI type; for example, the Victorian BushBroker scheme (an offset credit scheme) uses a consistent assessment method for scoring vegetation quality and gain through management actions designed for the Victorian BushTender metric (a conservation tender). Other MBI projects have tried a number of different types of metrics to find the most suitable type. Table 2 shows the metric types typically used for different types of MBIs (Appendix 1 shows the metric type used for example MBIs).

Using the same metric for different MBIs has multiple benefits for cost-effectiveness and learning. Different MBIs using the same metric can be compared for cost-effectiveness (i.e. environmental benefit per dollar), leading to improved understanding of the market and subsequent improvement of the MBIs used.

Table 2. Metric types commonly used by different types of MBI

Many natural resource management (NRM) issues have common characteristics though they may relate to different natural resource problems. Common characteristics include:

• Allocation of a limited resource or defined impact on a resource (e.g. cap-and-trade for water or salinity credits)
• Promotion of changed management or land use (e.g. conservation, pollution or recharge tender schemes)

The components (attributes) of metrics may be different for different NRM issues but the metric type and logic may be similar. (Appendix 1 shows the metric type used for some example NRM issues.) One of the ways that metrics have been designed for different NRM issues is the conversion of some NRM issues into a ‘point source’ problem. This is particularly useful when the impact of land management practice occurs at some distance from the site (e.g. impact of sugar cane farming on the quality of the Great Barrier Reef). By taking the sum of the change in land management for a site and treating it as though it was a point source, metrics available for point-source problems can be used in an MBI.

Using or adapting an existing metric

An existing metric may have much to recommend it, not least of which is the thought and expense that has already gone into developing and testing it. There may also be requirements or standards for using a metric for projects within an NRM body, jurisdiction or state. The benefits from consistency should not be underestimated and existing standards and jurisdictional requirements should be examined before deciding to design a new metric or substantially adapt an existing one. As more metrics are developed, tested and documented, it will be easier to evaluate existing metrics for suitability for new programs or use in new locations. There are some advantages and disadvantages to consider when reviewing existing metrics for modification and use (see table 3).

Table 3 Advantages and disadvantages of using or modifying an existing metric 

The following questions are recommended when reviewing an existing metric for use in a different MBI project or location.

1. Is the metric design consistent with the objectives of the new program?
2. Are their jurisdictional requirements or standards which direct the choice of metric?
3. Are there any operational constraints to adoption of the metric (e.g. intellectual property rights, copyright, metric support requirements, data management platform limitations)? 
4. Can attributes specific or unique to the new program or use be included (substituted or added) in the existing metric?
5. Can attribute weightings be adjusted to meet the objectives of the new program?
6. Can new inputs/management actions be incorporated into the metric if required? 
7. Are any methods used to predict change appropriate for the new program?
8. Is data available at an appropriate scale to calibrate and populate the metric for the new program, use or location? 
9. Will adoption of the metric assist similar projects or adjacent regions to develop cost-effective metrics and/or MBIs?
10. Can any data collected for the metric be used to improve the metric and/or value-add to the program (e.g. new data on species distributions, report on current habitat condition, form a baseline for monitoring)?
11. Are the modifications required for the metric to meet the program requirements a cost-effective alternative to constructing a new metric?

A good metric combines policy, economic, operational and scientific considerations to support the MBI project. The scientific considerations are discussed here to frame metric design as a deliberate and systematic process of building on a defensible logic and the analysis of an evidence base to produce a testable tool for measuring goods and services. A scientific approach supports the objective of producing an evidence-based, repeatable and defensible metric. There are six elements in metric design and use which require a scientific approach and support from scientific investigation (refer to table 4). The overall design of the metric should be governed by the policy and program objectives and any operational constraints. 

Table 4 - six elements in metric design

Metric logic

Different MBIs have designed metrics with different logic or structure. The main metric types are quantity, quantity and quality, model (process model), and risk analysis. These metric types usually combine a measure of the current state of the resource and the expected change or outcome from the delivery of management services. Attributes have been combined in different ways with quality and quantity (sometimes a function of quantity e.g. log10area) almost always multiplied together, as increases in quality are calculated for each unit of natural resource good or service. The conceptual framework can be built on scientific theory, results from previous studies, analysis and modelling of existing and new data, multi-criteria analysis (MCA, a form of statistical analysis of the preferences and weightings of expert informants), expert knowledge, or a combination of all of these.

Metric attributes

Metric attributes need to represent components of the goods and services being described. Attributes can range from simple measures (e.g. hectares of land) to complex surrogates for processes (e.g. extent of soil modification as an indicator of ecological resilience). The combination of attributes should be supported by scientific theory or previous research findings.

Selecting attributes can be achieved in numerous ways including:

• regression analysis – can predict how one response variable is predicted by other variables (possibly easier or cheaper to collect).
• pattern-based methods – can compare a selection of sites with different attributes and calculate a score representing the ‘difference’ between the sites.
• expert opinion – can be gathered informally through discussion and workshops or brought together through processes such as a Delphi technique (structured group communication) and followed by MCA (eg. analytical Hierarchy Process, or AHP,  a form of MCA used to arrive at weightings).
  
  

Attributes will usually be calculated for existing goods and services and change or outcomes.

Each attribute can be assigned a maximum and minimum to represent the meaningful range of possible values. Depending on the logic and attributes a metric can use:

• raw attribute values
• values transformed to a standard distribution (e.g. between 0-1)
• benchmark values to rate new values against an agreed standard.
• continuous or categorical values
• weighted or unweighted values
  
  

Metric evidence base

Evidence to test and compare attributes may come from theory, previous studies, existing or especially collected data, dummy or simulated data generated for the purpose, or expert knowledge. Many metrics require some spatial data for attributes but site level data is also usually required as spatial datasets are commonly too coarse to be solely relied on at the property, paddock or site scale.

Where there are data gaps it may be necessary to commission additional data collection, use expert panels to derive adequate surrogates or modify the metric logic to provide more representative measurement. Data will most often be lacking about the probability of achieving management benefits. Dedicated research programs may be required to develop or refine the accuracy and reliability of prediction over time. Implementation of a metric within an MBI is an opportunity for adaptive learning about the metric and instrument and future programs can benefit from some well-designed studies as part of implementation. A metric will only be as good as the data used in it and data gaps should be acknowledged and dealt with to avoid perverse outcomes.

To allow monitoring and enforcement of contracts, metrics and data used need to produce a sufficient indication of the outcome at the expected time to allow detection against background variation in the system. This problem is solved to some degree by designing contracts around outputs and not outcomes, but program evaluations will benefit from the use of metrics which have adequate ‘power’ (i.e. ability to detect a change when one occurs) to detect outcomes.

Metric validation

Sensitivity analysis and scenario testing have already been discussed above . The purpose of validation is to ensure that the metric accurately represents the goods and services (and property rights) of interest, is comprehensive of key components of the system, does not over or underweight important attributes and can effectively discriminate between alternative groups of goods and services. Validation also serves to test any field data collection protocols, data entry and management, and data retrieval and enquiry. Once tested, the metric should be ready to use.

Metric documentation and communication

To assure quality in keeping information about the metric and to share learning, metric documentation should encompass:

• the final metric and its attributes
• rejected attributes and justifications
• data sources
• metric specifications, including specifications for data management, metadata for datasets used, specifications for data collection protocols
• process of metric development
• process of metric testing
• personnel involved in metric development
• lessons learned
  
  

Communicating the metric and the process for developing the metric is a critical step in learning and sharing the learning. Writing and reporting on the process of development allows reflection on the challenges and limitations faced, especially if the process has occurred over an extended timeframe. Some of the excellent work of early metric designers has been critical to the ongoing development and success of metrics and MBIs in Australia .

Metrics for use in MBIs are necessarily purpose-specific and universal rules for development are difficult to describe. The questions described in the following guidelines are provided as a guide to key issues in metric design and not as a substitute for rigorous discussion about the specifics of the best policy approach, MBI, or metric required for a given project. Working through the questions users should be able to document key issues for metric design and use, and develop or refine a plan for metric development.

Increasing experience and documentation of MBIs will help new players to access and understand the skills, resources and time required to adapt or develop an appropriate metric for a new MBI. Documentation of metrics, metric development processes and lessons learned will be critical to sharing experience and expertise.

Metric design need not be the exclusive territory of any group including scientists, economists, policy advisors, technical experts or community representatives. Metrics are designed to represent goods and services. How goods and services are represented and prioritised within a metric requires input from a range of sources. Equally, it is unlikely that only one expert from a discipline has complete and relevant knowledge for all metrics and locations, and discussion and inclusion of a group of experts may be beneficial. Access to appropriate expertise in MBI design and social, economic and scientific researchers can be difficult. It is desirable to not only use appropriate and local knowledge, but also to enlist help from other places and perspectives. This section provides recommendations on how to access, make links with and different experts for metric design, and involve experts in metric development.

Access to expertise

The members area of this website provides a place to network with MBI practitioners, designers, implementers and others with expertise in MBIs. The website hosts a ‘Facebook’-style community of MBI practitioners and experts called the Little Orange Book. The Little Orange Book is an electronic forum for policy officers, regional NRM group staff, social and economic researchers and other people who are interested in or working on market-based instruments for natural resource management. It promotes interaction with people interested in MBIs, and discussion and information sharing. The website also hosts discussion forums where users can post a question or share stories, access a library of MBI publications, and an events calendar where users can submit MBI-related events. Use of these resources will improve access to assist in linking different experts. Practitioners and experts may already have links with other people working in the area and may be able to enlist the help of other support.


Working with experts

Most metrics described in the literature have been developed by project staff with input from expert panels via meetings and workshops. On a few occasions, analytical tools have been used to record and combine opinions and knowledge and subject it to statistical analysis. Other approaches have used Delphi techniques to generate potential metric attributes. With an appropriate list of experts, electronic mail can be used to receive contributions from experts to allow the program to access a large expertise set and minimise any time or practical constraints for experts to contribute (e.g. compared to face-to-face workshops). This approach can help overcome ‘group think’ in workshops.

Using experts to weight attributes in a metric is also useful. This can be done informally or using techniques such as Multi-Criteria Analysis (MCA) which enables the knowledge and opinions of experts to be statistically analysed to form an ‘arithmetic consensus by contribution’. Studies have also highlighted that experts have biases and knowledge gaps (not necessarily opinion gaps). It can be useful to use processes which minimise or at least recognise the bias of expert input.

Setting priorities, selecting attributes and determining weightings requires both technical and value judgements. It is important to realise that different people may be required to make technical judgements and value judgements and the processes used to combine these expertise sets need to reflect the capabilities and contributions of the different people (Failing and Gregory, 2003).

Oliver (2002) and Oliver et al. (2007) used several processes to facilitate expert panels to develop a set of indicators of vegetation condition based on their importance and feasibility. The approach included use of a Delphi technique to generate potential indicators without initial evaluation. The process used electronic mail to contact and receive contributions from experts, and asked an initial set of experts to nominate three other experts to contribute to the program. This approach not only allowed the program to access a large expertise set, it facilitated access to experts who were not initially known or identified by project staff and minimised any time or practical constraints for experts to contribute (e.g. compared to face-to-face workshops). The second phase of the program used a version of MCA, the analytic hierarchy process (AHP). This tool enabled analysis of the knowledge and opinions of experts about attributes which are most important surrogates of biodiversity and those which could be most feasibly assessed. Results indicate that an ‘arithmetic consensus by contribution’ can be reached and attributes categorised according to importance and feasibility. The study also highlighted that experts have some biases, in this case a correlation between importance weights for attributes and the spatial scales at which the experts work. Gole et al. (2005) report that the use of an expert reference group to undertake the main work of determining outcomes from management actions was a key success factor for the program.

Failing L. and Gregory R. (2003)..Ten common mistakes in designing biodiversity indicators for forest policy. Journal of Environmental Management 68: 121–132

Gole C, Burton M, Williams KJ, Clayton H, Faith DP, White B, Huggett A, and Margules C (2005) Auction for Landscape Recovery FINAL REPORT. WWF-Australia. (accessed at http://www.napswq.gov.au/publications/books/mbi/pubs/round1-project21.pdf)

Oliver I. (2002) An expert panel-based approach to the assessment of vegetation condition within the context of biodiversity conservation: Stage 1: the identification of condition indicators
Ecological Indicators 1:223-237

Oliver I., Jones H., Schmoldt D.L  (2007). Expert panel assessment of attributes for natural variability benchmarks for biodiversity. Austral Ecology 32: 453-475