What does the Department of Defense (DOD) community know about the effectiveness of the Commander's Emergency Response Program (CERP), how to use it to greatest effect, and cost versus its value to operations? DOD has been employing this "weapons system" for over a decade, and so much has been written on the topic that one would think quite a lot should be known. In fact, the surface of understanding has barely been scratched.

Claims regarding CERP effectiveness are wide ranging and include arguments that it is ineffective and increases instability or that it is effective only for short-term stabilization.¹ Proponents of these perspectives base their conclusions on anecdotes, surveys, case studies, and/or statistical analyses of large datasets. The argument presented in this article is that CERP has yet to be studied in a manner that permits confidence in any conclusions presented to date.

Analyzing a capability's effectiveness in a complex, dynamic environment is challenging. Compounding that challenge with ill-defined and a priori measures, or incomplete and in some cases inconsistent data or data access constraints, dramatically increases the burden on analysts. Today's environment demands a research approach and strong research designs structured to minimize the constraints and maximize the opportunities presented. The measures and data challenges require a significant effort on several fronts: understanding desired effects, what data are appropriate, where those data reside, obtaining access to those data, and "cleaning" the data to support analysis. Is such a significant effort warranted to better inform the debate? Absolutely.

Importance of the Debate

CERP has been in the commander's toolkit for years. It has been employed for many types of projects large and small in both rural and urban environments. Since its inception in 2003, DOD has requested, and the Congress obligated, $6 billion for the program in Iraq and Afghanistan. The United States and its coalition partners will continue reducing both forces and funds for kinetic and nonkinetic operations in Afghanistan, thereby making a better understanding of the effectiveness of CERP all the more critical. Moreover, over the next several years, in a future security environment that includes operations in which population engagement is an important component, commanders will likely expect CERP to be available to support their campaign plans and operations.² The history of CERP is reason enough to warrant an effort. When combined with considerations of near-term and future operations, there is little room to argue against a sustained commitment to rigorous analyses.

Lessons Learned

With its first appropriation in 2003, CERP could have been characterized as a Quick Reaction Capability (QRC), which by its nature is often fielded in ongoing operations without predeployment training or training materials and without doctrinal guidance on its most effective employment.³ Such materials do tend to emerge from lessons learned in operations; however, those lessons—and the materials that follow—are often based on anecdotal evidence.⁴ In some cases, this is understandable, as applying the rigorous test and evaluation regime used for cost-equivalent major defense acquisition programs is not feasible given the nonpermissive operational environment in which QRCs tend to be employed. Rigorously evaluating such capabilities in a counterinsurgency environment requires another approach.

Over the past year, we have been laying the foundation for, and begun executing, studies on CERP and other nonkinetic programs and activities. Through these efforts, a number of serious issues have been identified that should be taken into account before the defense community can state with confidence that it understands the effectiveness of CERP. Such understanding is necessary to develop training materials, implement CERP to the greatest effect in current operations, and value it against other capabilities in current and forecasted operations in order to inform doctrine, concept development, and programmatic decisions.

Tales of caution and optimism are offered in the sections that follow. Caution is expressed most often in reference to having too much confidence in the findings of studies that have been done to date. Optimism is offered because the serious issues can be addressed—several studies are presented as examples—thereby increasing the confidence that DOD and others can have in decisions regarding the employment and funding of CERP.

Understanding the Independent Variable

What is the size of the population of projects? For CERP projects in Afghanistan, the focus of the analytical efforts discussed in this article, data are entered into three separate databases—initially as an Afghanistan Development Report record, again with additional data and supporting media files in Combined Information Data Network Exchange (CIDNE),⁵ and finally as a CERP Checkbook entry.⁶ Working with data from 2008 to 2010, it became clear that all three databases required thorough review in order to build an accurate and comprehensive account of each project's vital statistics, including but not limited to subprojects, start/end dates, locations, cost, type, and desired effects. Consider, for example, the fact that the number of CERP records in the CIDNE database does not equal the number of CERP projects. Many records (for example, those in the category of Bulk Funds) often represent several projects that are visible only upon review of attached media files, the details of which are not accounted for when exporting unclassified CERP data from CIDNE. This brings to the fore the first cautionary note: studies that have been done to date based on unclassified data have in all probability drawn conclusions from an incomplete dataset.

The critical need to use classified data in CERP analyses will be a theme throughout this article. In spite of the best efforts of some talented analysts, no comprehensive study of CERP in which DOD could have confidence is possible using only unclassified data.

What is the substance of the population of projects? As several analysts have described, CERP projects are varied by category (20), location, date, duration (days to years), and cost. Within "category," there is potential for further differentiation by "type." For example, the category of "Repair of Civic and Cultural Facilities" includes such disparate types as mosque construction and telecommunication, radio, and support services. Projects are also varied in specified desired effects, a point to be discussed in depth later. This degree of variance in combination with the significant number of CERP projects leads to the second cautionary note: findings regarding CERP effectiveness drawn from anecdotes or case studies alone should be viewed as a weak basis from which to draw general conclusions. In addition, studies that have looked at thousands of cases but treated CERP projects indiscriminately—by aggregating all projects by cost, number, province, or country—may not serve the community well since they also fail to account for the variance described above.

Accounting for the Context

CERP is being employed in a complex and dynamic environment. Measuring effectiveness in such an environment, given the multitude of extraneous factors that may influence the achievement of a desired effect, is challenging but not impossible. This is where a strong research approach and rigorous research designs come into play.

CERP is not used in a laboratory setting in a way that one could, as in a true experiment design, control for the potential influence of specific extraneous factors by physically excluding them from the environment. It is instead used in an environment where other factors such as high-tempo operations may or may not be taking place, coalition forces may or may not be present, ethnic or tribal boundaries may or may not be crossed, projects may or may not have been consistent with community leaders' priorities, good governance may be present in one locale but not another, and so on. Many of these data are classified, serving as yet another example of the important role classification plays in CERP analyses. Comprehensive analyses of CERP effectiveness must take the potential influences of these other factors into account. This can be done systematically in several ways, two of which are described below.

In one type of comparative method, the research design controls for extraneous factors by comparing cases that share the same or similar values for the factors (Most Similar Systems Design) but differ on the value of the dependent variable (was the effect achieved or not?) on the assumption that doing so increases confidence that changes in the dependent variable are associated with changes in the independent variable. For example, were one to identify two cases—one in which CERP was employed and one where it was not—where the extraneous factors in both were similar in many respects and the desired effect was achieved in the case where CERP was employed and not achieved in the other case, one could have some confidence that the effect was a consequence of the CERP project and not other extraneous factors.

While this research design would increase confidence that CERP effectiveness was actually being measured, it has limitations. The two-case focus decreases confidence that the measured effectiveness can be generalized to the larger project population. A different research design, one that could account for context and increase confidence in the generalization of results, is the statistical method.

In the statistical method, extraneous factors are not controlled for; rather, they are measured and their influence is accounted for through statistical analysis. If a well-designed statistical model looking at thousands of cases concluded that CERP had been associated with changes in the desired effect in a statistically significant manner, confidence that an effect can be generalized would be warranted.

Given that different research designs have different strengths and weaknesses, a research approach to studying the effectiveness of CERP should be multimethod in character. Robust findings across multiple methods would increase confidence in conclusions regarding effectiveness. In addition, as often occurs in research, findings from one method will tend to suggest important questions that require additional analyses using a variant of the same method or another method altogether.

Understanding the Desired Effect

That Which Has Been Claimed. There are as many perspectives on the desired effect of CERP employment as there are conclusions regarding CERP effectiveness. Analysts have claimed it is intended to improve security through the purchase of loyalty or information, improve governance, protect forces, build capacity, or improve the relationship between the Afghan population and government of the Islamic Republic of Afghanistan.⁷ U.S. Forces–Afghanistan (USFOR–A) guidance on CERP as a weapons system instructs commanders to employ it in order to build capacity, promote peace and hope for future generations, and build trust and lasting support for the Afghan government.⁸ Other Regional Command–East guidance uses similar phraseology and adds that CERP projects enhance economic opportunities as a viable alternative to the insurgency.⁹

These desired effects run the gamut from the tactical to the strategic level of war. CERP certainly could be characterized as a tactical weapons system capable of generating tactical, operational, and strategic effects. From an analytical perspective, this is an important issue, as the focus of and research design for measuring effectiveness at the three levels would be different. The lack of clarity on the level of analysis in the CERP debate has made a large contribution to its inconclusive nature.

Given the wide range of CERP project categories, it is conceivable that each of the desired effects listed is valid in conjunction with one or more of those categories or combinations thereof. Indeed, an example of this from our own research is offered later. However, and to repeat, there is no guidance specifying such relationships and no research that, as yet, provides empirical support for them. Without an understanding of the desired effect (were CERP to be treated in the aggregate) or of certain effects associated only with certain categories of expenditures, little progress can be made toward evaluating CERP effectiveness.¹⁰

This plethora of perspectives is important to consider from another angle: training. Without a clearly specified mechanism of action and desired effect, commanders cannot be trained to understand well and plan against the effectiveness of this weapons system at any level of analysis. The absence of a comprehensive training program puts warfighters at a disadvantage and at risk.

That Which Has Been Analyzed. Ideally, assessing effectiveness should be done after a desired effect and a concomitant direct measure of effectiveness have been identified. Absent a direct measure, indirect and surrogate measures can be pursued. Agreement on a desired effect is far from an apt description of the current state of affairs. Nonetheless, in analyses of CERP (treated in the aggregate), the direct measure of effectiveness most often used has been changes in levels of violence. The hypothesis is that if CERP were effective, it would manifest as reductions in violence. This hypothesis likely follows from several analysts' claims that the desired effect is improved security. The availability of unclassified data may also have played a role in this choice—improvised explosive device (IED) events, for example, can be downloaded from CIDNE in an unclassified format.

Does the measure changes in levels of violence serve well as a valid direct measure of effectiveness? It is insightful to tally how often it is listed as a desired effect by CERP practitioners. In a review of over 2,000 CIDNE CERP records from Afghanistan, less than 10 percent specified a primary or secondary intended benefit that could reasonably be equated with changes in levels of violence, with the intended change being a reduction.¹¹ This should give one pause, and it highlights the third cautionary note with regard to recent studies: the conclusions being drawn about effectiveness may not be valid as they may not speak directly to the effects specified by the initiators of the projects under analysis.

It may be that this clear disconnect lies in the levels of analysis issue raised previously. Perhaps the approximately 10 percent specified by the practitioners are desired tactical effects and the recent studies were more strategically focused. A strategic-level study of the effectiveness of CERP using changes in levels of violence as the direct measure of improved security should take into account other factors that may be associated with such changes, the most significant of which is the presence of U.S. and coalition forces. No such rigorous accounting has been seen in recent studies. So again, one should give pause when considering the conclusions.

Does the changes in levels of violence measure serve well as a valid indirect measure of effectiveness? Indirect measures are useful when gauging a direct effect poses a significant challenge, and they are valid if they correlate with direct measures. Since there is no agreement on the valid direct measure, analysts can only speculate as to whether it may correlate with changes in levels of violence. But why bother doing that when several of the direct measures specified by practitioners are measurable? For example, the effectiveness of projects intended to increase activity at a market or bazaar could be measured by analyzing Ground Moving Target Indicator (GMTI) data on access roads to those locales in periods before and after project completion. As one might expect, those data are classified.

Are changes in levels of violence a reasonable surrogate measure? Surrogate measures are useful when issues such as time-sensitivity or cost prohibit waiting until a desired effect is measurable. They are often referred to as path-dependent measures whose values correlate with those of the desired effects. Similar to what was stated above, as there is no agreed-upon perspective regarding the mechanism of action of CERP, there is no way at this time of determining if changes in levels of violence is a valid surrogate measure of CERP effectiveness.

Finally, just as care should be taken when considering analyzing CERP in the aggregate, so too should care accompany approaches to treating violence in the aggregate. The difference between Afghan-on-Afghan violence vis-à-vis other types can be an important discriminator in understanding the effectiveness of CERP projects.

That Which Requires Greater Consideration. Our experiences suggest that several additional nuances should be brought to bear in analyses of CERP's effectiveness. The first is found in a careful reading of USFOR–A guidance on CERP employment; CERP is intended for projects that can be sustained by the local population or government, and commanders need to evaluate how projects can add value to the local community.¹² This emphasis on local suggests that the measurement of a desired effect should be focused in an area proximate to that in which the CERP project was completed. For projects where the desired effect may be to create or restore goodwill or suppress ill-will (for example, from a battle damage payment), this notion also suggests an interesting sensitivity analysis that could explore degrees of decay in effectiveness as a function of proximity to the project.

A second nuance is the importance of the temporal domain. Many of the positive effects that may be generated by CERP projects likely fade over time—but over what time period, and to what degree? At what point in time after project completion is an effect most pronounced, and at what point does it begin to diminish? Sensitivity analyses focusing on decay over time could answer these questions and could inform a commander's decisions regarding the sequencing of activities. Of course, one could also perform sensitivity analyses on combinations of time, proximity, and project type.

A third nuance, tightly coupled to the first, is the recognition that not all violence is local. If reduction in violence is the desired effect of a project that was in an area proximate to a main supply route, including any IEDs or other violent events along the route in the analysis may be inappropriate.

The more one studies CERP, the more one comes to understand the nuances that should be taken into account when drawing conclusions regarding its effectiveness.

Reasons for Optimism

Over the past year, we have steadily and patiently built the foundation for executing studies of CERP that satisfy the demanding criteria put forth in this article. The base of that foundation is data.¹³ It has been a significant undertaking getting access to, gathering, reviewing, and cleaning all of the data required to execute analyses that satisfy the minimal criteria specified above. We are grateful to the Joint Advanced Warfighting Program for sponsoring that work and to the warfighters who have taken the time to share data. Creating the data foundation is not as simple as downloading information from CIDNE. It requires a review of brigade battlefield/commander update assessments, patrol reports, Human Terrain Team reports, U.S. (and other coalition) force tracking data, GMTI data, and Afghanistan National Security Force development reports. It also calls for interviews with brigade, battalion, and company commanders. Finally, it requires cleaning and translating all these data into a format suitable for analysis. We have reciprocated that cooperation by communicating findings back to commanders who have, in turn, requested additional analyses in order to better inform their resourcing decisions. One such study and additional findings are presented below, albeit in a way that takes account of classification issues.

Study: Culvert-denial Systems

A deployed unit requested an analysis of the effectiveness of culvert-denial systems. Many roads in Afghanistan are undercut by culverts that present opportunities to insurgents seeking to intimidate or disrupt freedom of movement of the Afghan population. Culverts are ideal for the emplacement of high net explosive weight (HNEW) IEDs as they are ubiquitous, ample (allowing for significantly sized IEDs), and accessible (allowing for repeated visits over which to build an IED). When a vehicle encounters a culvert-emplaced HNEW IED, the immediate outcome often includes a destroyed vehicle, casualties, and road debris. The effects of such an event could include reduced freedom of movement of the Afghan population, which could, in turn, impact commerce and a host of other issues.

Research Design, Context, Desired Effect, and Findings. Culvert-denial system projects are a subtype that appears in several different CERP categories. After a thorough review of the three CERP databases described previously, several hundred such projects were identified. This large number and the desire to test for findings that can be generalized suggested the statistical method. The comparative method was also brought to bear on the question.

Culvert-denial systems have been employed in many different locations throughout Afghanistan, crossing district, provincial, ethnic, and tribal boundaries; in areas that are densely and sparsely populated; and in areas where the coalition's operations tempo has been both high and low. A statistical model was designed to identify the influence that such factors may have on whether installing a culvert-denial system produces the desired effect.

What is the desired effect? Since these systems are employed to deny insurgents the ability to hastily emplace IEDs in culverts that can injure or kill the Afghan population, the direct measure of effectiveness is changes in levels of IED events on or immediately proximate to culverts. This was measured using IED data exported from CIDNE for a defined period preceding and following the installation of culvert-denial systems.

In reviewing the data on culvert-denial systems, an opportunity for additional analysis was identified for which the comparative method served well. There was a period during which a commander had units of similar capability dispersed over similar terrain (both in terms of terrain features and population density) that included an 80-kilometer (km) stretch of road. The first 40 km of this road had culvert denial systems installed and the second half did not, as the commander was waiting to receive funding for it.

Using the statistical method, we identified a statistically significant positive association between the introduction of culvert-denial systems and a subsequent reduction in IED events on or near the culverts. In the comparative case study, no IED explosions were reported for the 40 km of road on which culvert-denial systems were installed versus five IED explosions for the 40 km of road where they were not installed. Upon completion of these studies, the findings were swiftly communicated back to the field.

This analysis could be characterized as a tactical analysis of effectiveness. Emplacing these systems can also have operational and strategic effects. Follow-on operational and strategic-level analyses will focus on measuring changes in freedom of movement on these roads (using GMTI data) in addition to other measures.

Cost versus Value. USFOR–A guidance regarding CERP instructs that CERP will not be used for "direct or indirect benefit to U.S., Coalition, or other supporting military personnel."¹⁴ However, that does not prevent the analytic community from assisting DOD in considering direct or indirect benefits to U.S. forces for purposes of calculating value. The culvert-denial system analyses can be used to illustrate what is possible.

U.S. forces, mounted and dismounted, cross culverts with frequency equal to, if not greater than, the Afghans. Insurgents are aware of this, and are also aware that a culvert provides one of few opportunities to emplace a HNEW IED that is capable of disabling or destroying a Mine Resistant Ambush Protected (MRAP) vehicle. A number of different systems are used to deny insurgents access to culverts, but for purposes of illustration, the cost of a cage constructed of reinforcing bar (rebar) is approximately $5,000 per unit. There are many variants of MRAPs whose unit cost can range from $500,000 to over a million dollars. A simple material-based, cost-versus-value calculation suggests that, in reference to MRAP survival, one rebar system has a value ranging between 100 and 200 times its cost.

Additional Findings

The culvert-denial system analyses demonstrate a value of analyzing CERP in a far more discriminate manner than has generally been done. The findings were of immediate use to warfighters in Afghanistan and offer a concrete example to policymakers of how CERP can significantly impact operations. We have continued to receive requests for analyses from the field and in the course of research have identified a number of intriguing findings that speak to the salience of nuance raised in this article.

Terrain Features. Aspects of the operational environment are proving to be important factors to consider in CERP analyses. It was suggested previously that effects should be measured proximate to a project. Based on analyses of completed CERP projects in both urban and rural environments, we encourage analysts to consider terrain features when setting the distance about a project in which to measure for the desired effect. We have discovered that urban terrain features appear to limit the geographic reach of a desired effect. If we cast the net too far from an urban project, evidence of an effect can be overwhelmed by unrelated activity at the periphery of the area of analyses. This finding is quite consistent with the observations offered in joint doctrine on military operations in urban terrain.¹⁵

Types of Projects and Types of Violence. In one province under analysis, a relationship was identified between economic development projects and increased proximate levels of violence. This was not the desired effect of these projects, of course, so the finding was troubling. Once identified, it would have been of no use to warfighters to share this finding and then simply walk away, leaving them with the impression that economic development projects invited violence. Identifying the types of projects and type of violence proved insightful in revealing an underlying dynamic—a particular set of projects was fueling a turf war among local powerbrokers resulting in Afghan-on-Afghan violence. This level of detail is of value to commanders as it suggests the motivations of the powerbrokers; in turn, that may offer a means through which to manage their participation in the counterinsurgency more effectively.

Operational and Strategic-level Analyses Interaction Effects. The culvert-denial system comparative case study controlled for the presence of U.S. forces by selecting a period when units in both cases were of comparable capability and patrolled a space of comparable geographic scope both before and after the installation of culvert-denial systems. That is, U.S. presence was essentially constant. Our most important findings from a strategic and operational perspective are emerging from analyses of projects in which U.S. presence proximate to a project actually changed after project completion.

This research line of effort focused on testing for a strategic- or operational-level effect and assumed the desired effect was improved security. The effect was operationalized as changes in level of violence. CERP projects were not treated in toto since our other research identified significant differences in effectiveness across categories. Instead, projects from several "like" categories were bundled and those bundles were used as the units of analysis.

As mentioned, any analysis of CERP projects in which changes in level of violence is the desired effect must take into account other factors that could have an impact on violence. The most significant of those factors is the presence of U.S. forces proximate to a project. Changes in presence were measured by analyzing Blue Force Tracking data.

Analyses have led us to conclude that there is an important interaction effect associated with changes in levels of violence in the proximate area where a project is completed and changes in the density of Blue Forces. Increases or decreases in presence nearby a project are systematically and significantly associated with the changes in levels of violence and the degree of the changes. This interaction effect is significant in analyses of several bundles of CERP projects, during different phases of operations, and across different locales in Afghanistan. It is not an understatement to describe the interaction effect as substantial for certain bundles of CERP expenditures when compared to cases in which changes in U.S. presence occurred in areas where projects may or may not have been completed. (A random sample of locations was analyzed to establish a baseline against which to compare the interaction effect.)

Should this relationship be robust in the face of updated data sets (both longitudinally and in richness), it could advance the conceptual development of combined arms (or, more appropriately, combined capabilities) warfare for counterinsurgency operations. Its more immediate value, however, from both an operational and strategic perspective could be as an input into the calculus behind decisions regarding how best to surge in an area—by sequencing or synchronizing kinetic and nonkinetic capabilities—or whether and how best to "thin out" for purposes of transitioning to a host nation government.

Conclusion

The debate regarding the effectiveness of CERP has been fueled by studies that may reasonably be characterized as incomplete since they do not address satisfactorily the several issues identified in this article. This discourse, as does any that is not well-supported by evidence, carries risk—in this case to the warfighter employing the capability and the policymaker determining whether to fund it. Such risks can be reduced by a sustained effort to better understand and then gather the data required to support rigorous analyses of CERP and other nonkinetic activities. That said, further increasing the burden on unit commanders to record even more data is not a course of action advocated here.

For capabilities already fielded, there is plenty of data out there: it takes only one trip from a company headquarters to its battalion headquarters to its brigade headquarters to its regional command headquarters to International Security Assistance Force (ISAF) joint command headquarters and to ISAF headquarters to be made patently aware of that fact. That mass of data, of course, is not "clean," is in many formats, and tends not to be consistent across units. As noted at the outset of this article, it is a challenging environment for analysts, and the analytical community has quite a bit more work ahead in that regard. With greater access to these data and strong research approaches and designs, improved analyses in which DOD and others can have increased confidence are possible. Greater access is not a call for prematurely declassifying much of the data required for thorough analyses. It is a call for the development and execution of a comprehensive data-gathering strategy so a better understanding of these nonkinetic capabilities can be developed.

The findings presented here demonstrate that committing resources to this analytical task can have substantial returns in answering questions regarding CERP effectiveness. A final note of caution is in order, however. Such questions should be thoughtfully crafted and informed by the issues and nuances identified in this article. A response to a simple query regarding CERP effectiveness will provide little if any insight to the one asking the question. Well-structured queries that take into account the nuances discussed previously will result in far more informative responses.

The analytical challenges presented here come from having to analyze CERP effectiveness in a nonpermissive environment long after its employment. In the future, to better understand such capabilities before they are employed and to establish an evaluation regime, the first task is to step back and consider what effects are desired, what mechanisms of action exist for their success or failure, and then and only then what data should be captured to determine operational effectiveness. This typical preemployment operational test and evaluation approach for kinetic capabilities should be made typical for nonkinetic capabilities as well, so they can be employed in the most operationally- and cost-effective manners. PRISM