9 JANUARY/FEBRUARY 2003 In this article, the authors first indicate the range of purposes and the variety of settings in which design experiments have been con- ducted and then delineate five crosscutting features that collectively differentiate design experiments from other methodologies. Design experiments have both a pragmatic bent������engineering��� particular forms of learning���and a theoretical orientation���developing domain- specific theories by systematically studying those forms of learning and the means of supporting them. The authors clarify what is in- volved in preparing for and carrying out a design experiment, and in conducting a retrospective analysis of the extensive, longitudinal data sets generated during an experiment. Logistical issues, issues of mea- sure, the importance of working through the data systematically, and the need to be explicit about the criteria for making inferences are discussed. Iriety n this short article, we draw on our collective experience of conducting design experiments for a range of purposes in va- of settings in order to delineate prototypical characteris- tics of the methodology and to describe what is involved in conducting a design experiment. Although the term design exper- iment is most closely associated with Brown (1992) and Collins (1992), pedagogical design has informed the development of theories of instruction for well over a century. Prototypically, de- sign experiments entail both ���engineering��� particular forms of learning and systematically studying those forms of learning within the context defined by the means of supporting them. This de- signed context is subject to test and revision, and the successive iterations that result play a role similar to that of systematic vari- ation in experiment. Design experiments are conducted to develop theories, not merely to empirically tune ���what works.��� These theories are rel- atively humble in that they target domain-specific learning processes. For example, a number of research groups working in a domain such as geometry or statistics might collectively de- velop a design theory that is concerned with the students��� learn- ing of key disciplinary ideas in that domain. A theory of this type would specify successive patterns in students��� reasoning together with the substantiated means by which the emergence of those successive patterns can be supported. This emphasis on theories reflects the view that the explanations and understandings in- herent in them are essential if educational improvement is to be a long-term, generative process. Design experiments ideally re- sult in greater understanding of a learning ecology���a complex, Design Experiments in Educational Research by Paul Cobb, Jere Confrey, Andrea diSessa, Richard Lehrer, and Leona Schauble Educational Researcher, Vol. 32, No. 1, pp. 9���13 interacting system involving multiple elements of different types and levels���by designing its elements and by anticipating how these elements function together to support learning. Design ex- periments therefore constitute a means of addressing the com- plexity that is a hallmark of educational settings. Elements of a learning ecology typically include the tasks or problems that stu- dents are asked to solve, the kinds of discourse that are encour- aged, the norms of participation that are established, the tools and related material means provided, and the practical means by which classroom teachers can orchestrate relations among these elements. We use the metaphor of an ecology to emphasize that designed contexts are conceptualized as interacting systems rather than as either a collection of activities or a list of separate factors that influence learning. Beyond just creating designs that are effective and that can sometimes be affected by ���tinkering to perfection,��� a design theory explains why designs work and sug- gests how they may be adapted to new circumstances. Therefore, like other methodologies, design experiments are crucibles for the generation and testing of theory. Design experiments are pragmatic as well as theoretical in ori- entation in that the study of function���both of the design and of the resulting ecology of learning���is at the heart of the method- ology. This emphasis on function in a realized context holds for all design experiments even though they are conducted in a di- verse range of settings that vary in both type and scope: ��� One-on-one (teacher-experimenter and student) design ex- periments in which a research team conducts a series of teaching sessions with a small number of students. The aim is to create a small-scale version of a learning ecology so that it can be studied in depth and detail (Cobb & Steffe, 1983 Steffe & Thompson, 2000). ��� Classroom experiments in which a research team collaborates with a teacher (who might be a research team member) to as- sume responsibility for instruction (Cobb, 2000 Confrey & Lachance, 2000 Gravemeijer, 1994). ��� Preservice teacher development experiments in which a re- search team helps organize and study the education of pro- spective teachers (Simon, 2000). ��� In-service teacher development studies in which researchers collaborate with teachers to support the development of a professional community (Lehrer & Schauble, 2000 Stein, Silver, & Smith, 1998). ��� School and school district restructuring experiments in which a research team collaborates with teachers, school ad- ministrators, and other stakeholders to support organiza- tional change (Confrey, Bell, & Carrejo, 2001). Crosscutting Features of Design Experiments We identify five crosscutting features that apply to these di- verse types of design experiments. First, the purpose of design

experimentation is to develop a class of theories about both the process of learning and the means that are designed to support that learning, be it the learning of individual students, of a classroom community, of a professional teaching community, or of a school or school district viewed as an organization. We interpret processes of learning broadly to encompass what is typically thought of as knowledge, but also the evolution of learning-relevant social prac- tices and even constructs such as identity and interest. When we look across these diverse types of design experiments, the means for supporting learning encompass the affordances and con- straints of material artifacts, teaching and learning practices, and policy levers (e.g., performance-based pay), as well as other forms of mediation that might, for example, include the negotiation of domain-specific norms���such as what counts as a ���good��� scien- tific question in a classroom (Wertsch, 1998). It is apparent from this broad view of means of support that it is often necessary to document learning ecologies at multiple levels (Kelly & Lesh, 2000). In the case of an in-service teacher development experi- ment, for example, the research team might focus simultaneously on the norms and practices of a professional teaching commu- nity, the participating teachers��� pedagogical reasoning and in- structional practices, and their students��� reasoning in a particular content domain. A challenge that arises in such cases is therefore that of coordinating multiple levels of analysis. Although, as a practical matter, a design experiment is con- ducted in a limited number of settings, it is apparent from the concern for theory that the intent is not merely to investigate the process of supporting new forms of learning in those specific set- tings. Instead, the research team frames selected aspects of the en- visioned learning and of the means of supporting it as paradigm cases of a broader class of phenomena. In the case of a one-on- one design experiment, for example, the broader theoretical goal might be to develop a psychological model of the process by which students develop a deep understanding of particular math- ematical ideas, together with the types of tasks and teacher prac- tices that can support that learning. In the case of a school district restructuring experiment, the theoretical goal might be to de- velop an interpretive framework that explicates the relations be- tween teachers��� instructional practices and the institutional settings in which teachers develop and refine their practices. In these and other types of design experiments, the initial design formulated when preparing for an experiment and the new form of learning it is designed to support are viewed as instances of broader classes of phenomena, thereby opening them to theoret- ical analysis. The second crosscutting feature is the highly interventionist nature of the methodology. Design studies are typically test-beds for innovation. The intent is to investigate the possibilities for educational improvement by bringing about new forms of learn- ing in order to study them. Consequently, there is frequently a significant discontinuity between typical forms of education (these could be studied naturalistically) and those that are the focus of a design experiment. The design developed while preparing for an experiment draws on prior research and attempts to cash in the empirical and theoretical results of that research. The process of engineering the forms of learning being studied provides the research team with a measure of control when compared with purely naturalistic investigation. Furthermore, in attempting to EDUCATIONAL RESEARCHER 10 support a specified form of learning, the researcher is more likely to encounter relevant factors that contribute to the emergence of that form and to become aware of their interrelations. By its very nature, the study of phenomena as complex as learn- ing ecologies precludes complete specification of everything that happens. It is therefore all the more important to distinguish in the specification of the design between elements that are the tar- get of investigation and those that may be ancillary, accidental, or assumed as background conditions. For example, in a study of children���s mathematical development, classroom norms of justi- fication might be assumed as background and the emphasis placed instead on conceptual development. Alternatively, the de- velopment of norms might serve as a primary target of investiga- tion (e.g., Yackel & Cobb, 1996). The use of prior research to both specify a design and justify the differentiation of central and ancillary conditions is central to the methodology. The third crosscutting feature builds on the first two: Design experiments create the conditions for developing theories yet must place these theories in harm���s way. Thus, design experiments always have two faces: prospective and reflective. These two faces are familiar to all empirical scientists, but the forms they take in design experiments are somewhat specialized. On the prospec- tive side, designs are implemented with a hypothesized learning process and the means of supporting it in mind in order to ex- pose the details of that process to scrutiny. An equally important objective is to foster the emergence of other potential pathways for learning and development by capitalizing on contingencies that arise as the design unfolds. On the reflective side, design experiments are conjecture-driven tests, often at several levels of analysis. The initial design is a con- jecture about the means of supporting a particular form of learn- ing that is to be tested. During the conduct of the design study, however, more specialized conjectures are typically framed and tested. For example, during a classroom design experiment, an initial conjecture about a prospective interaction between charac- teristics of tasks as they are realized in the classroom and student responses may be tested. If this conjecture is refuted, alternative conjectures can be generated and tested. Together, the prospective and reflective aspects of design ex- periments result in a fourth characteristic, iterative design. As conjectures are generated and perhaps refuted, new conjectures are developed and subjected to test. The result is an iterative de- sign process featuring cycles of invention and revision. Of course, to design iteratively demands systematic attention to evidence about learning and, as we later describe, this often involves the parallel development of measures sensitive to the changing ecology of learning. The intended outcome is an explanatory framework that specifies expectations that become the focus of investigation during the next cycle of inquiry. The fifth feature of design experimentation again reflects its pragmatic roots: Theories developed during the process of ex- periment are humble not merely in the sense that they are con- cerned with domain-specific learning processes, but also because they are accountable to the activity of design. The theory must do real work. General philosophical orientations to educational matters���such as constructivism���are important to educational practice, but they often fail to provide detailed guidance in or- ganizing instruction. The critical question that must be asked is