Value of Science 108: The VALUABLES Impact Assessment Framework, Step 2: Identifying How New Information Influences Decisions

A core principle in assessing the value of scientific information is that the new information yields societal benefits only when it influences a decision. In the previous explainer, we learned how to complete the first step in the VALUABLES impact assessment framework (Figure 1) by clearly defining the improved information to be evaluated and the alternative sources of information that would be used to make a decision in its absence. This first step establishes the difference in sources of information available to the decisionmaker between the reference case and the counterfactual case.

The second step—represented in row two of the framework—is to describe how the difference in information might influence decisions. We do this by identifying the actions taken by the decisionmaker in these two different scenarios.

Often, when a decisionmaker uses improved information to make her decision, she will choose an action that is different from the action she chooses when she uses the alternative source(s) of information. Think back to the example of the value of GPS with the live traffic information, from the Value of Science 107 explainer. Suppose that you need to drive downtown for an errand, and your decision involves choosing the route you will take—either the highway that links your home to downtown, or side roads. GPS with live traffic information (the reference case, in blue) tells you that traffic has created a huge backup on the highway that links your home to downtown. In this case, you are likely to want to avoid this traffic jam and take side roads all the way to downtown.

Suppose your alternative source of information is GPS with no live traffic information (the counterfactual case, in red). This information would not capture the traffic jam on the highway. In this case, you are likely to choose to take the highway to downtown, as the highway usually is faster than side roads in the absence of a traffic jam.

We use the second row of the VALUABLES impact assessment framework to establish the differences between the actions chosen in two cases—either a world in which the improved information is available, or one in which the information is not available. So, returning to the GPS example: in the blue column, you would write the chosen action with the improved information, “Take side roads.” In the red column, you would write the chosen action with the alternative information, “Take the highway.”

For a researcher who is using the framework to complete an impact assessment, filling in the second row likely will require some qualitative research to learn how different kinds of information lead to different actions by decisionmakers. Usually, the quickest way to understand how information is linked to actions is to interact with the decisionmakers themselves. For example, a researcher might try to have some casual conversations with decisionmakers to gain a better understanding of the decision context in which the information is used. These initial conversations might be followed by formal interviews or focus group sessions in which questions are structured to elicit these details from decisionmakers.

Another approach relies on written documents to fill in the second row of the framework. In some decision contexts, the way in which information is used to select from a set of available actions is formalized in regulations, manuals, or guidance documents. In these decision contexts, the researcher can use these documents as evidence to support her claim that improved and alternative information leads to specific decisionmaker actions.

In the Value of Science 107 explainer, we used the example of Stroming et al.’s retrospective impact assessment that examines the impact of satellite information on the management of harmful algal bloom advisories around Utah Lake. Recall that the researchers learned from direct conversations with the lake managers that managers use satellite information on algal blooms together with monthly field tests, lab tests, and visitor reports (as described in the first row of Figure 2, in the blue column). The researchers also learned that, before incorporating satellite information into the advisory process, lake managers had used just the latter three sources of information (indicated in the red column).

The next step was to identify how the difference in information in these two scenarios leads to different actions by lake managers. The researchers examined several guidance documents, as described in Section 3 of their journal article, to understand how water quality information is used to issue algal bloom warnings in Utah and other states. The researchers also conducted interviews with managers from the Utah Department of Environmental Quality to develop a more detailed characterization of the decision context in which satellite information was used in a particular algal bloom event in the summer of 2017. The researchers found that, depending on the severity of the algal bloom, lake managers can either issue warnings and public advisories, or prohibit access to the lake. These two options comprise the suite of decisionmaker actions, as seen in the second row of Figure 2. But while the actions are identical in both the reference and counterfactual cases, the satellite data are available on a weekly basis (blue column), and field tests are conducted on a monthly basis or as reported (red column). This key difference—the “delta” between the two scenarios—means that decisionmakers might take action more quickly with the availability of satellite data.

In the next explainer, we will follow this logic through to show how information used in decisionmaking processes can lead to beneficial outcomes for people and the environment—and how researchers can use the VALUABLES framework to quantify these outcomes.