Decisionmaking for Demonstration Projects

On March 30th, 2023, Resources for the Future (RFF) held a workshop on decisionmaking for demonstration funding, exploring how the Department of Energy (DOE) can make decisions funding demonstration projects from the many programs included in the Bipartisan Infrastructure Law and the Inflation Reduction Act. Demonstration funding focuses on late-stage energy technologies, which is a departure from DOE’s typically earlier-stage research and development (R&D) funding programs. Our workshop covered three issues that largely set demonstration funding apart from earlier-stage R&D: (i) Type of Metrics, (ii) Risk and Portfolio Analysis, and (iii) Community Benefit Plans. The first session discussed metrics, which describe the criteria used to judge individual demonstration projects and capture not only the direct costs and benefits, but also spillover benefits such as knowledge spillover and network effects. These individual demonstration projects make up a portfolio of selected projects, and the second session discussed how to maximize benefits in a portfolio while accounting for risk and uncertainty. Finally, the third session dealt with community benefits plans, a requirement in recent demonstration funding opportunities accounting for 20 percent of the overall technical merit review of a proposal. In this paper, we summarize the results of the three sessions and expand on the findings in each session with additional research and commentary.

Under the Inflation Reduction Act and the Infrastructure Investment and Jobs Act, the Department of Energy (DOE) has received tens of billions of dollars for demonstration funding. This funding focuses on late-stage energy technologies that are ready to be deployed at commercial scale. While the department has funded some demonstration projects in the past, most DOE funding has traditionally been focused on earlier-stage research and development (R&D). Because of both the scale of the money involved and the stage of technological development, demonstration project funding presents new challenges for DOE as it determines how to disburse the funds, incentivize success, and be a careful steward of taxpayer dollars.

This paper reviews and expands on a workshop held by Resources for the Future (with funding from Breakthrough Energy Foundation), providing lessons learned and guidance for policymakers in charge of distributing this funding. The workshop involved three sessions, focusing on metrics, risks and portfolio analysis, and community benefits. Though these are not all the relevant areas and are not unique to demonstration funding, the issues considered here are qualitatively different from those encountered in funding earlier-stage, lower-cost R&D.

The first session was devoted to metrics, the criteria by which to judge individual projects. From an economics point of view, one wishes to maximize net benefits, so the two most prominent metrics are the costs and benefits of the projects. However, the primary reason to do demonstration projects is not for the direct benefits of an individual project but for the knowledge spillovers and network benefits it induces. These benefits are difficult to quantify; however, they can be significantly larger than the direct costs and benefits of the demonstration project. In this paper, we discuss some of the challenges in generating appropriate metrics and the potential to use proxy metrics to get at these questions.

Although metrics are assigned to individual projects, ultimately the goal is to develop a successful portfolio of projects. An essential part of developing such a portfolio is understanding risk and the probabilistic nature of each project’s benefits. This was the subject of the second session of the workshop. We discuss some aspects of risk and how one should develop a portfolio that maximizes expected benefits while incorporating risk.

The final session of the workshop considered community benefits. In service of major Biden administration initiatives on easing community energy transitions, creating jobs, and improving environmental justice, DOE requires project proposers to document how they will benefit the surrounding and other affected communities in what are formally known as community benefits plans. We discuss these requirements with an eye toward the guidance given to applicants on how to develop these plans and how that guidance helps DOE decisionmakers evaluate the quality of the plans.

1.1. Comparison with Current Project Evaluation

This list of topics does not directly align with the evaluation criteria used in many of DOE’s released funding opportunity announcements (FOAs) for demonstration projects. In fact, these recent FOAs involve two levels of review. The first is a set of scored criteria. For example, in the FOA for the hydrogen hubs (H2Hubs), the criteria are technical merit and impact, financial and market viability, workplan, management team and partners, and community benefits plan. In addition, a set of program policy factors can be used to select the final portfolio of projects to fund. These include considerations related to diversification, environmental justice, and domestic content.

Many of these scored criteria address the quality of the project and its risk of failure rather than the costs and benefits. The emissions benefits of the project are mentioned in the first scored criterion, however, and spillover benefits appear implicitly in some of the program policy factors. The idea of a diverse portfolio appears in a number of the program policy factors, but the responsibility for such diversity lies with the program leaders, not the applicant. Community benefits are both directly scored as one of the main criteria and considered as part of the program policy factors.

1.2. Transparency and Disclosure

While it was not a primary subject of the workshop, transparency is a vital issue for DOE’s demonstration project funding. With billions of dollars on the line, taxpayers and the general public have a right and need to know how these demonstration programs are being run, how projects are being selected, and how successful they have been.

So far, DOE’s record is not encouraging. As an example, the agency has not released information on most steps of the process for the $8 billion H2Hubs program. The original request for information issued by the Office of Clean Energy Demonstrations (OCED) generated over 100 comments, but none were released, although we obtained them through a Freedom of Information Act request. The FOA to start the application process was released (all FOAs are as a matter of course), but the individual concept notes submitted as part of the application process were not. After the agency made “encourage” or “discourage” decisions based on these notes, it released summary statistics but no additional information on the decisionmaking process. Two applicants that we are aware of out of the 79 released their concept notes. The full applications were received by DOE in April, but we still await further information on the applicants and their proposals.

Given the amount of money on the line, OCED and other offices issuing demonstration grants do not want to risk disclosing confidential business information (CBI), and the applicant teams are in competition with one another and thus are disinclined to be transparent. We have heard that some confidentiality is being maintained even within a team. Nevertheless, in our view, the public need to know should be the default position, with the burden of and justification for secrecy resting with the agency and the applicant, as appropriate.

1.3. Lessons Learned

This workshop led us to develop a number of detailed recommendations for DOE, included at the end of each section below. We highlight some important ones here:

• Benefits from future deployment induced by learning and derisking likely dwarf direct benefits, so learning potential, market creation, and total potential abatement are vital metrics to consider.
• Diversification across project designs is important both to hedge risk and to maximize potential benefits, but there is insufficient information to use sophisticated diversification strategies.
• Mechanisms that institutionalize continuous two-way engagement between the private sector implementing the hub and the community groups are vital to community benefit plans.

Sections 2 through 4 roughly follow the structure of the workshop and go into greater detail on each of the three sessions. Each section discusses the key themes, issues, and considerations raised by invited speakers and workshop attendees in that session. We augment several of these thematic areas with additional literature that either was referenced by speakers or is directly relevant. Finally, each section expands on these lessons learned and provides a more detailed list of recommendations based on the themes discussed.

To compare applications for a given funding opportunity, it is important to develop metrics that capture the benefits and costs of a project. However, not everything can be quantified with a numerical metric. Therefore, identifying proxy metrics that capture these considerations is an important part of developing a framework for demonstration-funding decisions.

In addition to metrics that directly address the costs and benefits, other metrics address the level of risk associated with projects. The higher the risk and, hence, the probability of failure, the lower the expected benefits, all other things equal. At the same time, riskier projects may have an extraordinarily long tail, meaning benefits could be very large with a low probability. Having such projects in the portfolio should be encouraged. The incorporation of risk into portfolio funding decisions is the subject of Section 3.

2.1. Costs and Benefits

The cost metrics start with government expenditures but could include total expenditures (private and public) and any environmental damage associated with the project. On the benefits side, the most obvious is reduction in emissions. This includes greenhouse gases and other air pollutants. Changes in emissions can be converted into dollar figures through the social cost of carbon and the social cost of other pollutants such as SO2 and NO2, as well as PM2.5, of which SO2 and NOx are precursors.

However, one of the major takeaways of the workshop is that these direct costs and benefits of the project should not be the primary focus for decisionmakers. Instead, one of the motivations for demonstration projects is that there are spillover benefits from demonstrating the technology, and these spillover benefits can dwarf the direct benefits of an individual project by enabling emissions reductions across broad sectors of the economy. Two things to consider are knowledge spillovers and network effects, in which a user of a technology gains a benefit from others using it. These effects mean that the demonstration of the technology can lead to increased deployment, leading to further induced benefits. Most of this session focused on the potential spillover benefits from demonstration funding and associated metrics.

While they were not a major subject of this session, jobs are also often used as a metric when evaluating a project and have a role in community benefits plans. Yet jobs are a confusing metric from an economics point of view. For any given project, labor represents a cost, not a benefit. In addition, it can be difficult to know when a new job is created versus resulting from people moving from one job to another. Nonetheless, if under- or unemployment exists in a given community, increasing local employment may have benefits. Similarly, benefits may also result from increased tax revenue, which we discuss in Section 4.

2.2. “The Technology Pork Barrel”

Demonstration projects face the “valley of death” between R&D and deployment. This includes high capital requirements that exceed the resources available to companies in the sector, high technology risk for projects that have not been proven at scale and uncertain demand for the technology, counterbalanced by potential spillover benefits. The Technology Pork Barrel describes the systematic biases to R&D programs introduced by political institutions, making them “susceptible to performance underruns and cost overruns” (Cohen and Noll 1991, 53).

There are several sources of government failures when selecting financed projects (“picking winners”) that may lead to the issues identified by Cohen and Noll (1991). First, information asymmetry, or poor access to information, may lead to bad allocation decisions. Governments typically do not participate in the marketplace, meaning they might have a limited understanding of pricing, competing technologies, and end user preferences. Second, incentives in bureaucracies are likely to cause excessive risk aversion relative to incentives in private markets or even the socially optimal level. The implication is that technology investments by the government can crowd out low-risk private sector investments instead of stimulating new investment. Finally, concentrated interest groups and lobbying groups, as well as electoral politics, may weigh policy decisions toward benefiting specific groups or political jurisdictions (Nemet et al. 2018).

2.3. Upscaling and Timelines

Upscaling takes a significant amount of time and is a nontrivial endeavor. At the workshop, Gregory Nemet presented an analysis of 511 demonstration projects (see Nemet et al. 2018). Figure 1, a graph of nuclear plant construction from 1950 to 2020, shows the time necessary for plants to ramp up in size. There were many small nuclear energy demonstration projects at the outset of these demonstration programs, and 15 years of demonstrations were needed to build up to small commercial-scale plants. It took another 10 years to advance to large commercial-scale plants.

The long timeline necessary to bring demonstrations to commercial scale makes them well suited for iterative upscaling, which means that rather than building large-scale plants from the outset, demonstrations could benefit from gradual increases in size. In fact, data from carbon capture, utilization, and sequestration (CCUS) pilot and demonstration projects indicate that “larger plant sizes increase the risk of CCUS projects being terminated or put on hold” (Wang et al. 2021). Increasing capacity by 1 megaton of carbon dioxide (Mt CO2) per year increased the risk of failure by nearly half.

This finding suggests that size may be a useful metric for demonstration funding, with the perhaps counterintuitive idea that larger increases in size should be disfavored, at least in earlier demonstrations. However, economies of scale are often present, leading to favoring larger projects on cost grounds, and thus the initiatives laid out in the Bipartisan Infrastructure Law, which seek to accelerate progress in key technologies toward commercialization and appear to prioritize scale, work in the opposite direction.

2.4. Market Conditions

In his workshop presentation, Nemet also identified market conditions as an important influence in the success of demonstration projects. Market prices for carbon typically do not follow the theoretical increasing Hotelling price path, which predicts that the price of an exhaustible resource rises “at a constant pure rate of time preference for the time of project duration” (Nemet et al. 2018) Rather, demonstrations have sometimes begun at the same time as markets are heading the wrong way. For example, carbon capture and sequestration projects in Europe have come online just as the carbon price in the European Union crashed or as oil prices decreased in the United States, making the use of the captured carbon dioxide for enhanced oil recovery uneconomical. With a wide range of potential outcomes and uncertainty regarding carbon prices in 2030, some projects may need to survive multiple years selling into a market with depressed prices.

Workshop participants noted the importance of demonstration projects in identifying and building end-use markets for the technologies. In many cases, these markets may not currently exist or have not considered the cleaner production method or product being demonstrated. For instance, green and blue hydrogen technologies motivated by the hydrogen hub demonstrations will likely have price and logistical considerations that differ from those of the incumbent gray hydrogen industry. Demonstrations should prove that these emerging technologies have a place within the market. This means that an important qualitative metric is identifying a pathway toward full commercialization after the initial benefit of federal support in the form of grants or incentives is gone. Additionally, policymakers can consider using quantitative metrics, such as the existing and projected green premium of the technology—that is, the additional cost of a product made with low-emitting technologies instead of traditional high-emitting technologies—to assess end users’ willingness to pay. Other metrics could be the number of transactions taking place in the market (which may be hard to obtain) and the stability of the premium across users and over time. Finally, in the longer term, one could look to the number of different markets (or proto-markets) operating. Consolidation into bigger and more efficient markets would be another sign of market creation success.

2.5. Metrics to Capture Spillover Benefits

Several ideas were suggested for metrics that can capture the spillover benefits from demonstrations. Primarily, this means policymakers should prioritize not just the direct costs and benefits of the projects but also the learnings from projects. In fact, for many of the demonstrations under OCED’s purview, the direct emissions outcomes (or the per-unit cost of emissions reduction) of the selected projects are less important than the knowledge gained from their deployment. Workshop participants stressed the importance of failure as a tool for learning how to choose future demonstration projects in improving program design.

In the face of uncertain profitability and potential failure, projects can still accelerate learning, lowering information costs and risks, while also demonstrating the value or technical feasibility of the technology to others within the market. The key value at the demonstration phase is from either “revealing technology performance relative to expectations or other technologies” or demonstrating the potential cost reductions in later stages (Reiner 2016). The latter, also known as learning from replication, has historically been essential for other clean technologies, such as solar photovoltaics and wind. However, the former, also known as learning by diversity, is necessary to avoid technology lock-out or lock-in before replication is considered.

Bossink (2020) breaks down the idea of learning by diversity further and describes the first demonstrations for a particular technology as exploring the technical possibilities of the technology. In this phase, project developers focus on the capture of intellectual property and further iterative technical improvements of initial prototypes to develop a “plausible promise prototype.” Later demonstration projects in the replication phase focus more on scaling up and taking advantage of learning curve benefits, which can lead to improved cost-efficiency and effectiveness (Bossink 2020). Thus, the appropriate metrics for learning depend on the maturity of a particular technology being demonstrated and the number of demonstrations for that technology. In the earlier stages, the degree of novelty of the demonstration is a qualitative metric that

can demonstrate the potential for intellectual property (IP) capture, whereas in the later stages, metrics should focus on the potential for improved cost efficiency and how easily these efficiencies can be replicated across projects.

One way to capture the potential learning from a demonstration is to examine the total amount of abatement that a new technology can enable. For example, in demonstrating a technology to decarbonize steel manufacturing, the total potential abatement would be the emissions from steel manufacturing -- the part decarbonized by the demonstrated technology. This metric places an upper bound on the spillover benefits from the demonstration project. A similar metric used in the private sector is the total addressable market, a top-down estimation of the total potential market for a product or service (i.e., if a company were to achieve 100 percent market share) (Hariharan et al. 2015). This metric is already widely used among earlier-stage venture capital firms.

Demonstrations can expand the size and reach of a market, allowing for greater network opportunities. These networks cover both the technical and social aspects of a technology, from the physical distribution of the product to the complex groupings of “organizations, professionals, users, customers, and regulators” (Bossink 2020). Network benefits may be ambiguous and difficult to measure, but it is important to account for contributions of the deployed technology to networks and vice versa. For example, demonstrations may accelerate or enable the expansion of a market, allowing other players in the market to grow quickly or provide value to small or independent users. Through demonstrations, a shared language and practice can be formed among participants, alongside supply-demand networks, value networks, and communities of interest and practice (Heiskanen et al. 2017). These networks can further propagate learning, and perhaps as a proxy for measuring their development, metrics can look at new partnerships and agreements between actors in these spaces. Additionally, metrics could also attempt to measure whether systemic sharing or aggregation is planned.

Demonstrations can also coordinate the activities of complementary goods and services in the market. A successful demonstration should account for whether such coordination—such as with suppliers or end users—is necessary and determine how this can be accomplished. Thus, an important metric is the dependence of the demonstrated technology’s deployment on the cost and development of complementary goods, as well as the potential increased value of complementary technologies as a result of the new technology.

However, many workshop participants noted the potential hurdles involved with information sharing and the diffusion of practices among developers of innovative technologies. Developers would likely prefer to make innovations as proprietary as possible, at least in part to retain a market advantage. Some participants, citing past experiences working with technology developers, believe that many developers place little emphasis on knowledge transfer, instead going only as far as knowledge management.

2.6. Ex Ante versus Ex Post Criteria

While much of the academic literature evaluating demonstration projects is ex post (i.e., with hindsight), the immediate challenge DOE faces is determining adequate criteria ex ante (i.e., before making the decisions on winners). Participants noted that several ex ante considerations, such as the appropriate private sector cost share and selection of projects that will maximize knowledge spillover, are subject to uncertainty.

Furthermore, once projects are selected, the relationship between ex ante and ex post metrics becomes more important. Participants noted that metrics established on the ex ante side should map to the metrics that will be used as the basis of evaluation ex post. Therefore, data collection becomes an important issue, including for metrics measured over the course of the demonstration and ex post, and even among projects within the sector that do not ultimately receive federal funding. Additionally, the quality of these data can vary; for example, the quality may differ based on how easy it is to quantify the data. While some metrics may be quantifiable and others may require a proxy to adequately measure quantitatively, still others may be measured only qualitatively.

2.7. Complementary Policies

Innovative technologies may also require complementary or related policies to drive adoption, such as renewable portfolio standards or tax credits. Given the volatility of market conditions for many historical demonstration projects, as outlined in Section 2.4, these related policies may be essential for ensuring robust demand for the technology. To understand how the private sector sees existing or proposed related policies as interacting with new innovations and adoption, policymakers should engage with this sector. In many cases, investments in new technologies may be long-lived, meaning that early adopters may depend on the stability and certainty of related policies for the success of their demonstration projects. Policymakers should keep this in mind when selecting metrics.

2.8. Recommendations

DOE should use metrics that maximize the spillover benefits of a project, which may be much more significant than the direct costs and benefits.

While direct costs and benefits of demonstration projects can and should be tracked, they should not be the sole or primary focus of the selected metrics. Individual demonstrations alone may have limited impact on reducing carbon emissions, but the technology could have a large impact as it reaches full commercialization. Demonstrations can help derisk the technology and, through spillovers, accelerate the rate of future deployments. The direct benefits may be dwarfed by the level of deployment induced by the spillover benefits.

DOE should focus on the degree of learning from a project, even if it fails.

A common theme across this session was the importance of learning, even from potential failures. Both failed and successful projects can provide invaluable learning opportunities to other project developers, suppliers, and end users in the market. While these knowledge spillovers may be difficult to quantify, metrics should attempt to account for them. Additionally, metrics should also identify the common barriers to information sharing, such as commercial sensitivity and proprietary technologies, and seek solutions to overcome them.

Possible metrics for knowledge spillovers include the novelty of the demonstrated technology and the total abatement it potentially enables.

More novel technologies likely have higher spillover benefits. The total potential future abatement that the technology can provide sets an upper bound on the spillover benefits for climate. In particular, if widespread deployment of a technology is unlikely to lead to large benefits, the spillover environmental benefits will be limited.

Metrics should account for the effect that a demonstration will have on the full market, including complementary goods and services.

Demonstrations may have an important impact on suppliers and end users, helping coordinate their activities or increase their value. Furthermore, demonstrations should attempt to prove that adequate market demand exists for the demonstrated technology. One metric for this is the green premium for the ultimate product. Metrics should measure network benefits as well as the impact that the demonstration has on complementary goods and services. These impacts could be measured quantitatively by the number of agreements and partnerships that market participants enter into or qualitatively by assessing methods of systemic sharing or aggregation in the application.

Larger sized projects should not be favored by default.

The literature suggests a balance is needed between large sized projects to take advantage of economies of scale and smaller sized projects that have less inherent risks of failure.

The second session of the workshop was devoted to uncertainty and methods of selecting a portfolio of projects, which incorporates risk and builds on the metrics of the individual projects. It is important to carefully consider a portfolio for three reasons: First, the spillover benefits of multiple projects may be duplicative, reducing the ultimate impact. Second, and conversely, the spillover benefits of a given project may be synergistic with those of other projects, making the whole greater than the sum of its parts. Third, in the presence of uncertainty, a portfolio of diversified projects can hedge the risk and maximize expected benefits.

3.1. Types of Uncertainty

In this session, Erin Baker, a professor at the University of Massachusetts–Amherst, outlined three fundamental types of uncertainty and their risk and portfolio implications. The first type of uncertainty is technological, and because private sector entities are risk-averse, government funding provides a source of capital. However, when policymakers are choosing the types of technologies to invest in, Baker suggested a focus on projects with a higher probability of failure but a “long tail” of potentially transformative long-term effects. As discussed in the first session, government entities may tend to prefer safer, risk-averse projects, but they are uniquely positioned to carry demonstration-stage technological risk, which can translate into eventual commercialization by the private sector. The portfolio implication is that the more long-tail projects DOE invests in, the more likely one of these projects can eventually reach commercialization. This closely follows the general venture capital philosophy, which Jim Cabot, managing director at Breakthrough Energy Ventures, described in his presentation. According to the “power law” in venture investing, typically 20 percent of a venture fund accounts for the majority of the fund’s returns. In the case of government demonstration funding, this percentage will likely be much higher, given the level of risk tolerance, but DOE should keep an open mind about riskier projects. Additionally, it is important to consider diversification across sectors and technological approaches.

The second type of uncertainty Baker noted is related to the design and implementation of the technology in a commercial setting. Demonstrations will likely require many subprocesses and smaller individual design decisions. As discussed in the first session, the most important aspect of this uncertainty and its associated risk is the learning from the project. Demonstrations should follow a learning-by-doing philosophy, with knowledge spillover to other entities within the market and the wider economy, rather than being held by a single firm. Selected projects should have a high option value, with the willingness to fund a project derived not only from the success of the project but also from the value of learnings, even in the case of failure. Workshop attendees suggested diversification both across technologies and across approaches within those technological areas. This can improve knowledge spillovers and the probability of learning from selected projects. This diversification does not need to happen concurrently but can be sequential over time.

Finally, the third type of uncertainty described is with policy—including policies that currently do not exist or are temporary. Because of existing policy uncertainty in several technology areas, private entities in industry face compounding uncertainty. If the government can fund technologies and prove feasibility or cost-competitiveness while also ensuring policy support, industry would likely be more willing to invest. Additionally, while deployment may depend on policy support, the development of technologies can also drive shifts in policy. For example, policymakers can consider policy changes that can support technologies that are not yet profitable under current policies. Ultimately, the portfolio implication is still to consider diversification across technologies, as this can provide value even in the presence of currently lower-cost technologies. As an example, modeling of offshore wind under a wide range of climate policy scenarios shows that it provides substantial climate value, even in the presence of low-carbon technologies that are cheaper in the near term, such as onshore wind, solar, and nuclear energy (Cranmer and Baker 2020).

3.2. Selecting a Portfolio

Selecting projects in light of these considerations can be complicated. However, some workshop participants said that one should simply pick the “best” projects. Projects with high expected returns (including spillovers) and relatively low downside risks are desirable in most frameworks, although those with upside risks—the possibility of very high returns—should not be ignored. Selecting projects based on their overall quality along with an intuitive notion of diversification may be sufficient for these demonstration-funding programs, obviating the need for overly complex analysis.

Discussion of portfolio analysis raises the question of the appropriate level of risk tolerance for the government. From a political point of view, it might be desirable to minimize any chance of failure so as not to give opponents of the program anything to inveigh against, with the failure of Solyndra being a prime, albeit trite, example. However, deploying only well-understood projects that are sure to succeed would lead to few, if any, spillover benefits, negating a major motivation for these programs. Importantly, the government’s risk tolerance should be at least as high as that of the private sector.

One way to select a portfolio, which has been discussed at prior DOE workshops on R&D portfolio analysis (Bush et al. 2020), is the use of expert elicitation to quantify uncertainty, allowing one to calculate an optimal R&D portfolio through stochastic optimization. One can also make use of ideas from finance, such as modern portfolio theory, in which variance in outcomes is a measure of risk. Other, more modern approaches exist as well (Salo et al. 2011). Given the paucity of data, the fact that these are not tradable financial instruments, and the relatively small number of potential investments involved, a naive approach to diversification (picking the “best” projects and ignoring their mix) should be sufficient.

A final note on portfolio selection is that diversification should not be a consideration for just the projects funded in a given FOA. Projects in different programs can have correlated risks of failure if they involve the same technologies and sectors, and they also can have duplicative spillover benefits. Therefore, diversification should apply across the entire government portfolio of investment. The broader the scope of projects included in a portfolio analysis, the better, other things being equal.

3.3. Diversity in Leadership

While not directly related to portfolio diversification in a traditional sense, the need for gender and racial diversity across the portfolio of selected projects was an area of particular interest to workshop attendees. Traditionally, leadership in earlier-stage start-ups is primarily white, male, and educated at elite institutions. However, companies with greater gender and ethnic diversity are likely to outperform their peers and experience above-average profitability (Hunt et al. 2015). Baker suggested the need for more inclusive research in energy transitions and cited a recent paper providing action items for government agencies and philanthropic institutions (Ravikumar et al. 2022). Perhaps the most important related consideration mentioned by attendees was how to ensure that decisionmakers’ and evaluators’ implicit and explicit biases do not influence the decisionmaking process.

3.4. Trade-off between Deployment and Capacity Building

An important topic during the discussion portion of this session was the trade-offs between a deployment-first mentality and capacity building for all the associated processes surrounding a core technology. Proponents of the former argued that the purpose of demonstration projects is to derisk the technology and demonstrate commercial viability so that institutional investors can eventually fund the technology once it reaches commercialization. In their view, the best way to do this is to deploy the projects as quickly as possible to attract entrepreneur and investor interest, accelerating the pace of the next handful of demonstration projects.

On the other hand, part of the challenge is not just to demonstrate whether the technology will work in a commercial setting but also to reduce the associated green premium. In most cases, products such as clean hydrogen are more expensive than their carbon-emitting counterparts (including so-called grey hydrogen). To be competitive in the market, the costs of the two (including subsidies and fees) must be relatively close to parity. These costs are influenced by both the core technology and the underlying scaling efficiencies associated with the cost of capital, labor, and supply chains. Because some of these efficiencies take time to develop, some participants argued that demonstration funding should be granted with an eye toward the longer term and not just the deployment of the core technology.

Ultimately, both sides agreed with one of the key issues outlined in the first session—that upscaling can take a significant amount of time beyond the first handful of demonstrations. Even with the deployment of the first project (and the handful of projects that follow), the private sector may still be unwilling to participate until further projects have been deployed.

3.5. Recommendations

DOE should adopt a level of risk tolerance at least as high as that of the private sector.

Taking risks is fundamental to the spillover benefits that motivate government funding of demonstration projects. Simply demonstrating well-understood technologies will lead to few, if any, additional benefits. DOE should select projects with the understanding that a certain number of projects may fail. A nonzero failure rate should be considered a success of the program (assuming the failure arose from high risk projects rather than mismanagement). We endorse DOE’s phased-funding approach used for project awards to mitigate the impact of project failures.

DOE should work to ensure that duplication of demonstrated technologies is minimized and that riskier demonstrations are spread across projects to try to make project failure as uncorrelated as possible.

Avoiding duplication and correlation of risks is fundamental to maximizing benefits and hedging risks, although this guidance should be taken as qualitative, not quantitative. In addition, DOE should tilt toward funding projects with long tails, those with low probability of very large payouts.

DOE should select a diverse portfolio of “best” projects both within a particular FOA and across multiple FOAs and programs rather than attempt complicated portfolio analysis.

Neither the data nor the numbers exist to support a complicated analysis of diversification for DOE demonstration funding. Nonetheless, DOE should ensure that the portfolio of projects is broad to maximize diversity and hedge the risk.

DOE should work to have a diverse set of project leaders, project reviewers, and staff members.

In line with the administration’s focus on equity and inclusion, DOE should recognize that the differing perspectives brought by people with different backgrounds can both serve a role in achieving the Biden administration’s diversity, equity, inclusion, and accessibility goals and lead to better and more diverse projects.

Recent FOAs from DOE have required applicants to submit an initial community benefits plan (CBP) covering four main goals: community and labor engagement; investing in the American workforce; advancing diversity, equity, inclusion, and accessibility; and contributing to the Justice40 Initiative. CBPs are weighted at 20 percent of the overall technical merit review of proposals—a significant percentage, illustrating the importance and value of these plans for DOE’s project evaluation. CBPs are required to describe prior community engagement efforts, the current state of affairs, the potential impact of the project on stakeholders, and future plans to help reach the four goals. In addition, CBPs should also describe the resources (e.g., staff, budget) dedicated to implementing the plan.

For each of the goals, DOE requires that applicants’ CBPs describe milestones, roles, responsibilities, timelines, dedicated resources, and mechanisms for tracking progress and ensuring accountability. Applicants are expected to define their own metrics of success for the different aspects of their plans. The plans also include a temporal dimension through the milestones integrated with the go/no-go decisions between the different project phases. Importantly, DOE encourages the use of community benefits agreements, community workforce agreements, and other legally binding contracts between stakeholders to ensure that promised benefits are realized. Alternatively, applicants can collect letters of support from local stakeholders.

4.1. Elements of a Good CBP

According to OCED’s guidelines, a good community benefits plan identifies the key stakeholder organizations and shows understanding of their concerns. Most importantly, the first goal of the CBP, community and labor engagement, is defined as ensuring that community and labor have opportunities to influence project decisions and especially the siting of the facilities and infrastructure. In their study of carbon capture and sequestration projects, Ziegler and Forbes (2010) state that understanding local community context is a key principle of effective two-way community engagement. However, they note that the funder should be able to evaluate the effectiveness of the plan to reach communities affected by the project. Moreover, if applicants are not familiar with a community, they might do a poor job of characterizing its history and dynamics, which could make it difficult for DOE to evaluate this part of the CBP.

Good workforce plans show the potential to create and support quality jobs and, in particular, identify partners to establish training or apprenticeship programs. To meet the goal of diversity, equity, inclusion, and accessibility, DOE is looking for high-quality partnerships that will ensure disadvantaged and underrepresented workers have access to the economic opportunities generated by the project. In this session, Daniel Raimi emphasized that developing a local workforce might prove harder in rural and disadvantaged communities, which are likely to lack robust training programs, housing, and education institutions.

The Justice40 part of the plan should show an understanding of the project’s positive and negative impacts on disadvantaged communities. Even though a project may not be located in the neighborhood of a disadvantaged community, applicants are encouraged to think about potential secondary benefits flowing to such communities farther down the supply chain or farther away (OCED 2022).

Finally, the FOAs and DOE’s guidelines encourage applicants to support their CBPs by including enforceable agreements such as project labor agreements (PLAs), community workforce agreements (CWAs), and community benefits agreements (CBAs). DOE considers these as ways to guarantee that developers deliver promised benefits. PLAs are collective-bargaining agreements for the construction industry, negotiated between developers and construction trade unions. These are meant to reduce delays and decrease the cost of project administration by having one global agreement for all trade unions rather than multiple bilateral agreements. PLAs can include local hiring clauses, equity plans, and strategic recruitments in underserved communities. They are enforced by the National Labor Relations Board. CWAs are specific PLAs that include community-oriented commitments for equitable workforce development, social justice, and small business support (DOL n.d.).

CBAs are agreements signed between project developers and community groups, which can be neighborhood associations, faith-based organizations, unions, environmental groups, or other grassroots organizations that represent the interests of the residents who will be affected. Similarly to PLAs and CWAs, negotiations often include local hiring agreements, living wages, and procurement for women- and minority-owned enterprises. As of 2018, fewer than 30 CBAs have been adopted in the United States since the 1990s, mostly in large cities such as Los Angeles and New York City in the context of local economic development (Belongie and Silverman 2018).

4.2. Community Representation

While the goal of getting community representation for the demonstration projects is laudable, it may be difficult in certain cases. The most serious issue is intransigence. Some groups may be opposed to any industrial development in their community. This could be because they fear the disruption and environmental impacts more than they welcome the possibility of jobs and economic development or because the community has already experienced significant development, among other reasons. If such groups refuse to come to the table or prove intractable, they not only could set back the CBP process but also might jeopardize the entire project.

From a broader perspective, it may be difficult to identify all the possible groups and determine which are relevant for engagement. This is particularly challenging in larger, more diverse communities with groups that may want different things. Further, some groups may lack the capacity for deep engagement in terms of dedicated leaders and their time commitment. Thus less organized groups might go underrepresented or face a steep learning curve that will hamper their ability to participate. Additional barriers may include difficulty in attending engagement activities because of a lack of reliable internet access or dedicated staff and the capacity to carry out engagement, as well as the health, education, and food challenges that are common in disadvantaged communities.

4.3. Corporate Challenges

Issues also often exist on the corporate side of engagement. Many companies are not used to two-way engagement and being bound by group decisions. Research on the social license to operate sought by oil and gas companies has revealed that most companies conduct one-way engagement—that is, they provide information to communities but are not good at listening (Schneider 2013), and they rarely make costly alterations to their plans except when facing even larger “penalties,” such as from local government refusal to grant permits. In the case of the CBP process, the penalty is a low rating on the 20 percent CBP factor, which may be large enough to counteract this business-as-usual behavior.

Putting together a robust CBP requires a significant amount of up-front work in identifying groups to engage and partner with, drawing up and signing CBAs, getting support from myriad business partners, and putting together an achievable strategy to deliver benefits to stakeholders. These costly and time-consuming efforts could deter small or less well-resourced potential applicants. As many innovations arise from start-ups, there may be a trade-off between funding dynamic demonstration projects and funding projects that score well on the CBP. Perhaps this trade-off is worthwhile, but it should be recognized and ameliorated so that the application process does not deter innovative firms from entering a bid.

4.4. Community Benefits Agreements and Other Enforceable Agreements

Although community benefits agreements are supposed to ensure that benefits are distributed to underserved communities, researchers have found that typical CBAs struggle to reach that goal because of their narrow scope and the weak enforcement mechanisms included in the agreements (Belongie and Silverman 2018). With the ad hoc approach of most CBAs, which are designed from scratch for each specific project, the learning curve is steep for the parties involved, and different agreements are inconsistent in terms of community involvement. Belongie and Silverman argue that institutionalization of these agreements could prevent inconsistencies and underwhelming results by providing a template to community leaders, local governments, and developers in their negotiations. However, adopting model CBAs and other forms of institutionalization can also set in stone processes wherein one party indefinitely holds the decisionmaking power.

In the past five years, several CBAs were used in the context of climate infrastructure for offshore wind developments (Fraser 2023). However, two of these CBAs were negotiated for offshore developments on Martha’s Vineyard and Block Island, relatively homogeneous communities with high percentages of white, high-income inhabitants.

4.5. Economic Efficiency

The four main goals of CBPs do not speak directly to economic efficiency, the important goal of maximizing the net benefits of a project, taking into account its positive and negative impacts. Language that could relate to efficiency is contained in the Justice40 section of the FOA, which says that an assessment of the needs of communities should be made, burdens (negative impacts) characterized, benefits characterized (what they are and where they flow), and a plan made for their tracking and quantification. The Industrial Decarbonization and Emissions Reduction Demonstration-to-Deployment program, introduced in the Inflation Reduction Act, is the only FOA we reviewed that prioritizes projects seeking “the greatest benefit for the greatest number of people within the area in which the eligible facility is located” (U.S. Congress 2022). This definition is based on a utilitarianism that does not necessarily map fully onto the efficiency criterion of maximizing net benefits.

Furthermore, enumerating, let alone quantifying, net benefits to communities and disadvantaged groups is a major challenge. Benefits are often described as job creation and economic development, which are both difficult to measure. Economists prefer to measure social welfare, which is an even more complex metric. Additional government revenues should also be counted as a benefit, depending on how they are spent, although this is not mentioned in the FOAs we reviewed. New industrial development in a small community can lead to major demands for public services, such as sewers, drinking water, and new roads, in what is sometimes referred to as a “boomtown effect.” Thus, the additional revenues may be needed to maintain baseline levels of service for a growing community and may not necessarily be used to increase social welfare for the residents. Moreover, the development may result in more road congestion and other negative side effects, such as air pollution, additional waste streams, and changes in water use. Sometimes developers attempt to provide public and private services to a community as a way to ameliorate such impacts and reduce community opposition.

4.6. Lessons from the Front Line

In the CBP session of the workshop, it was noted that these plans have the power to move markets, as they force developers to think about disadvantaged communities at every stage of the project. In New York State, the Climate Leadership and Community Protection Act includes community benefits planning activities for large public projects similar to DOE CBPs. In the state’s third offshore wind solicitation, 30 percent of the scoring for project selection focused on non-price-related elements, such as a CBP and local supply chain requirements. Applicants that would have imported most materials to the construction site were encouraged to consider ways to deploy a local supply chain, thereby creating new manufacturing jobs in underdeveloped communities.

The CBP goals in DOE’s guidelines may be tough to achieve, given that applicants may devote only a small amount of funding to them and that these plans are trying to address deeply rooted issues. Underserved communities might benefit more from an impactful targeted program that can create actual change than from applicants scattering money among multiple marginal actions. For an offshore wind project in Salem, Massachusetts, that required bids to include a local and disadvantaged workforce, an applicant consortium established a pre-apprenticeship program to train a local workforce that will benefit from wages significantly higher than average. Because of the size of the project and the associated funding, the program will likely directly benefit less than 50 people, but it will be transformative for these local workers.

Because of a history of tense relations between communities, local governments, and industries in some locations, DOE might consider incentivizing new engagement structures. For example, in the River Parishes in Louisiana, local communities are in active litigations (DeRobertis 2022) with a blue hydrogen project because of their distrust of the oil and gas industries and the state government, which, according to the communities, historically have not served their best interests. To ameliorate relations, DOE could take a more active role in oversight or have third parties be responsible for administration of the benefits and agreements.

4.7. Recommendations

Given the novelty of CBPs, we have developed an extensive set of recommendations for DOE to incorporate in future FOAs and decisionmaking.

Develop an efficiency criterion.

Applicants should be asked or required to assess net community benefits of the project. We are suggesting not a formal benefit-cost analysis but rather a listing of the types of benefits and costs and, as best as possible, their quantification in physical or monetary terms, or both. The list of costs and benefits would include local revenues and supporting infrastructure costs (e.g., from new roads and greater road maintenance). Applicants should also list any in-kind or monetary community contributions they plan to make and subtract these from the costs.

Better define the job creation metric.

Applicants need to be held to a higher standard regarding what job creation means. They should be asked to provide an assessment of whether the jobs will be full- or part-time and whether these will be new jobs or just transfers from other jobs that result in zero job creation. Applicants could also be asked to comment on their expected pay scales relative to prevailing wages in the community.

Encourage better targeted efforts and appropriate expectations.

DOE’s CBP guidance should encourage applicants to focus on one impactful and targeted program to deliver transformative benefits to disadvantaged communities rather than planning multiple engagement activities for each goal that will deliver only marginal benefits and potentially dilute the effectiveness. Cost-effectiveness is especially important because the FOA has no budget requirement, so the funding for CBP will likely be small.

Consider new engagement structures, such as continuous two-way engagement.

To facilitate dialogue in places with deeply rooted tensions between disadvantaged communities and developers or even municipalities, the CBP guidance should encourage applicants to rely on third parties to participate in the engagement process and administer the benefits. Designating a liaison agent between developers and communities has been a successful engagement strategy in the past. This liaison can bring a better understanding of the stakes if the agent comes from the impacted communities. Alternatively, DOE could assume this position with more active oversight of the process.

Look for actions that go beyond compliance with existing federal, state, and local regulations.

Applicants should be encouraged or required to identify applicable statutes governing their construction and operations, and they should note where they plan to exceed regulatory requirements.

Look for meaningful commitments to community benefits.

The inclusion of a CBA or other legal agreement in the application should not result in a better score if it seems inadequate to meet CBP goals. Indeed, agreements can fail to maximize the benefits delivered to local and underserved communities, since they often have a narrow focus on labor and weak enforcement mechanisms. CBPs that include monetary and temporal commitments may better serve communities.

Look for who is representing the community.

Applications should include all communities affected by a project in the engagement process and not only the ones that have the leaders, the resources, and an existing structure to represent them. Efforts to reach out and include communities that face serious barriers to participation should be rewarded. The selection panel should be especially cautious of CBP processes in which one party holds all the decision power.

Implement third-party audits or direct government involvement.

Audits by third parties and government agencies can ensure that applicants are following the processes outlined in the CBP.

Implement community surveys.

Even if a project operates as planned, some groups in the community will not be happy and may still oppose the project in its entirety. Both DOE and project leaders should recognize that the concerns of community groups should be heard and given their due in the evolution of the project. DOE should consider mounting community surveys periodically and at the end of the grant period to gauge community involvement, agency, and sentiment. Project leaders who know these surveys will be implemented may be more likely to take community concerns seriously.

Develop metrics of success for CBP goals.

DOE should develop metrics of success for CBPs, especially for Justice40 goals that are measured at the all-of-government level. These metrics of success should be embedded in the overall program evaluation strategy.