Eccles and Templeton (2003) amplified these criticisms in a recent discussion of how to identify successful youth development programs. As they reviewed the research done on programs for youth, they were struck by the heterogeneity in virtually all design features:

It is worth emphasizing that the evaluation of program effectiveness can be compromised by how the program itself is run. If it encounters difficulties with the recuitment or retention of participants, if it is delivered inconsistently, if it is changed before its effects have a chance to play themselves out, or if it is unduly affected by societal fads (some of which may be legislatively mandated), one can say little about its success.

A variety of methods are used to study programs for youth. Most studies rely on either cross-sectional or longitudinal surveys that link activity participation to individual level outcomes—such as school achievement and engagement, mental health, social development, and/or involvement in various problem behaviors—with the primary research goal of describing the relation between participation and outcomes. Unfortunately, few of these studies measure characteristics of the programs themselves. Consequently, the studies tell us little about the actual features of the programs that might explain any observed change in participants.

Selection concerns are a constant threat to understanding the effects of extracurricular activities and after-school activities. When participation is a choice, those who opt for a given program may well differ in the first place from those who do not; the program itself may be irrelevant in producing long-term differences. Some recent longitudinal studies have included the most obvious third variables. And other longitudinal studies have gathered data consistent with a theory-based evaluation perspective. In these studies, the researchers measure the hy pothesized mediators of participation on individual change and then use causal modeling techniques to test these hypotheses. Such designs tell us something about the plausible “causes” of the many longitudinal changes that might be associated with participation in the activity. Too few of the studies of extracurricular activities have used experimental designs with random assignment to pin down more definitively the consequences of activity participation.

Researchers studying after-school programs, whether in schools or in community organizations, have typically used two research strategies: nonexperimental descriptive studies and quasiexperimental or experimental program evalua-tion strategies. Although experimental methods using random assignment are rightly considered the gold standard of program evaluation, they can be quite expensive and difficult to implement. Accordingly, they may not always be the best method to study community-based after-school programs (cf. Agodin & Dynarski, 2001; Hollister & Hill, 1999).

Rather, the best method depends on the question(s) being asked. The method also depends on the nature of the thing being studied. Studies of these types of experiences on positive youth development have focused on at least four different levels: the individual across time, programs, organizations, and communities. Programs themselves are also composed of different types of specific activities. Similarly, organizations usually contain a wide variety of programs and activities. Finally, organizations themselves are very heterogeneous, ranging from after-school centers tied to such national youth organizations as the YMCA, YWCA, 4-H, Girls Incorporated, Beacons, and the 21st Century Learning Centers, to local parks and recreation centers, amateur sports leagues, and faith-based centers. The best method of study depends on the level one wants to study.

The most comprehensive theories about programming effects typically focus on either the program level or the activities within the program. Not surprisingly, most of the quasiexperimental or experimental program evaluations focus on this level for three major reasons: (1) programs and activities are simple enough to allow for explicit theories regarding the nature of

the proposed impact of the program on youth development; (2) programs and activities are small enough to make random assignment to the treatment and control groups possible; and (3) programs often have sufficiently well-developed manuals and resources materials to allow dissemination.

Doing randomized trial experimental evaluations is much more difficult at the organization and community level. For example, there are a variety of challenges to using experimental designs to evaluate large nationally visible organizations such as the YMCAs or YWCAs. First, national organizations differ in their local programming. Consequently, even if one could successfully implement a truly randomized trial evaluation design for specific sites, it is not clear that the information gained would generalize to other sites. This is why multisite trials are advocated in this setting. However, this is a common feature of experimentation that highlights internal validity at the cost of external validity. Replication and extension are part of the experi-mental mantra. In addition, because these organizations are complex and offer a varied assortment of programs, the level of evaluation needs to be quite general. Such information is likely to tell us little about which specific aspects of the organizational context produce positive developmental results for the participating children and adolescents.

Even evaluation of programs within organizations can be quite difficult. Most after-school and in-school nonacademic programs are voluntary. Although parents may try to insist that their children attend, their ability to enforce their desires on their children declines as their children move into and through adolescence. In addition, as noted above, many community organizations for youth include a diverse array of programs from which youth select. Often their selections vary from week to week or day to day, making each individual youth's experiences at the organization quite unique. Again, there are methods to meet the messiness of the real world.

Each of these program and organizational characteristics has implications for experimental program evaluation. For example, the voluntary nature of many community-based programs cre ates a problem with selection bias. When such programs are offered at school during the regular school hours, random assignment may be easier and more successful because the participants are more likely to attend regularly and complete the program. In contrast, the voluntary nature of joining and attending after-school community-based youth programs, particularly if they are in nonschool settings during nonschool hours, leads to more sporadic attendance and higher rates of dropping out. Consequently, researchers are faced with uncontrolled factors that influence attendance. In this case, rigid adherence to random assignment classification in analyzing one's results is likely to underestimate the program's effectiveness for those youth who are actually exposed to it over an extended period of time (Zaff, O'Neill, & Eccles, 2002). Length of participation is not an infallible moderator of program effectiveness, however, if youth who are more likely to stay with a given program do so because of preexisting differences.

The challenge for program evaluators is to specify the features that make complex programs effective. Because individual participants may select which parts of a program to attend and how often, evaluators may know little about each individual's exposure to various aspects of the center's programming. Such variation makes it difficult to determine the aspects of the program that are responsible for certain developmental outcomes.

Finally, the evolving nature of many youth programs poses problems for evaluation. Experimental methods usually assume a static treatment. Nonexperimental research on youth programs suggests that the most highly respected and well-attended programs are dynamic—changing, for example, in response to seasonal activity structures, changing clientele, changing staff, and information derived from ongoing reflective practice and self-evaluation, as well as from the youth themselves (McLaughlin, 2000; McLaughlin, Irby, & Langman, 1994). These are often the problems associated with ineffective treatment, shifting client presenting issues, shifting responses, never knowing if anything works. Rather than make these chaotic and unorganized aspects of the environment the driving aspects of the field, the professionalization of and dis

ciplined approach by the field needs to understand what works.

Given these concerns, it is not surprising that some of the most careful studies of extracurricular and other positive youth developmental programs use either nonexperimental methods or mixed methods in which small experiments are embedded as part of an action research agenda. Also not surprisingly, some of the strongest experimental evaluations of nonacademic programs for youth have been conducted on school-based programs offered during regular school hours.

Studying organized community efforts at increasing the provision of quality experiences for youth is even more challenging. Nonetheless, there is a growing interest in efforts at this level. Both researchers and policy advocates are coming to the conclusion that substantial and sustainable increments in the quantity and accessibility of high-quality after-school experiences for America's youth need community-wide initiatives.

In principle, the best design remains one that uses random assignment. The policy question invariably posed is whether Program X adds value to business as usual; only a true experiment can allow this question to be answered with certainty. If there are no demonstrable differences, policy makers will not see the point in supporting new programs. Accordingly, there is a practical as well as a scientific reason for true experiments in the “real” world.

These are controversial issues, and we think the wisest conclusion is cautiously even-handed: Use a variety of methods, each with its strengths and weaknesses for given purposes, and look for convergence in conclusions. For instance, in the arena of psychotherapy research for adults, randomized clinical trials established conclusively that therapy can work (Smith & Glass, 1977), and nonexperimental studies extended this conclusion by suggesting that therapy as typically practiced does work (Seligman, 1995).

Especially given the struggle for credibility over the years, the field of positive youth development should not give up on experimental designs simply because they are difficult to implement or are too expensive or too messy. To do so would invite justified skepticism. To sustain and further the gains made by the positive youth development field, we must demand rigorous evaluation, despite demands on cost and imagination.

As noted, enough outcome studies have been done to warrant reviews of these programs to abstract what works and why. We offer the provocative conclusion that at least as much is known about effective youth development programs as is known about effective clinical interventions—psychotherapeutic and pharmacological—for adolescents that are described earlier in this volume. Indeed, recent years have seen the publication of many reviews of the effectiveness of youth development programs, some qualitative (narrative reviews) and others quantitative (meta-analytic reviews). Before we turn to an overview of these reviews and their conclusions, some framing is in order.

Each review starts by demarcating its terrain—e.g., character education programs, school-based programs, after-school programs, prevention programs (those that decrease problems or risky behaviors), promotion programs (those that encourage positive outcomes like social skills). In some cases, two reviews overlap in their terrain (and hence the research studies included), and in other cases, the terrains are distinct enough to result in completely nonoverlapping studies. Some large number of outcomes studies from either published or unpublished sources are then identified and then winnowed according to one or more design criteria (e.g., comparison groups, quantitative data, adequate statistical power, behavioral measures, replication). Among included studies, some reviews distinguish between high-quality evaluations and others, a criterion difficult to judge because relatively few programs assess fidelity or quality of implementation. And in some reviews, only programs that work in the sense of yielding significant differences between intervention groups and comparison groups are examined in detail.

This latter strategy makes sense if one is trying to discover best-practice interventions—model programs—but it is suspect if one is trying to ar