Excerpt taken with permission from: Hattie, J. (2009). The Black Box of Tertiary Assessment: An Impending Revolution. In L. H. Meyer, S. Davidson, H. Anderson, R. Fletcher, P.M. Johnston, & M. Rees (Eds.), Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research (pp.259-275). Wellington, New Zealand: Ako Aotearoa The Black Box of Tertiary Assessment: An Impending Revolution John Hattie Visible Learning Labs, University of Auckland Abstract There has been a formative assessment revolution that has swept our compulsory schooling, and is about to descend on higher education. It has the potential to make major changes as to how assessment is undertaken, with more emphasis on ���feedback from assessment��� or ���assessment for learning��� (alongside the traditional ���assessment of learning���), using assessment to improve and change what and how we teach. Students familiar with the power of this assessment model in high school (especially since the introduction of NCEA) will expect and demand different forms of feedback from assessment from their lecturers. At the same time, more international comparisons of universities will place pressure on universities to enhance their teaching and quality of assessment. This chapter reviews the multiple outcomes of higher education, the importance of alignment of curricula and assessment, outlines these newer models of assessment, reviews ���What works best��� relating to teaching and learning in tertiary settings, and outlines the effects of newer assessment models for selecting higher education students. Keywords: Assessment for learning, higher education outcomes, power of feedback Introduction A revolution of assessment has swept through many of our primary and secondary schools in New Zealand (and in many other countries). This revolution relates to Assessment for Learning and it can be witnessed in innovations such as the National Certificate of Educational Achievement (NCEA) and its standards-based approach, the emphasis on reporting more than on scoring, constructive alignment of learning and outcomes, peer collaborative assessment, learning intentions and success criteria, and the realisation of the power of feedback. These issues have become commonplace discussions in our schools. The higher education community is yet to catch up, but the doors seem to be opening and one hopes not too slowly. My theme is how we need to re-open the black box of tertiary assessment and what a wonderful set of new initiatives we can place into our black box. The revolution of using assessment as an integral part of teaching and learning, and not primarily as a summation of teaching and learning, are about to enter the halls of academia. There is nothing wrong with black boxes: a ball-point pen is one and it works well ��� I do not want to know what is inside nor do I need to know, but it works fine. We have, in large part, a black box of assessment in tertiary assessment ��� it has worked for us for many years. We implicitly trust our academics to know what they value in their subjects, to set examinations and assignments, to mark reliably and validly, and then to record these marks (using more and more sophisticated marking engines), and the students move on, upwards, and/or out. We look at pass marks, we ���red flag��� courses in which students satisfaction is not high, and we run the occasional course in teaching or assessment (occasional on the part of the lecturer, not staff

Excerpt taken with permission from: Hattie, J. (2009). The Black Box of Tertiary Assessment: An Impending Revolution. In L. H. Meyer, S. Davidson, H. Anderson, R. Fletcher, P.M. Johnston, & M. Rees (Eds.), Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research (pp.259-275). Wellington, New Zealand: Ako Aotearoa development personnel) that touches a few staff (usually the more committed). The newer models for teaching academics how to be teachers and set more appropriate assessment are exciting, as is the trend (mainly out of Europe) about the role of peers in assessment in new and powerful ways. The most exciting, however, is the move to include formative assessments notions in the all-so-often summative black box of tertiary assessment. The aim of this chapter is to address these new moods. First, the inclusion of multiple outcomes from tertiary assessment begs the introduction of newer forms and foci for assessment. Second, the necessity for tertiary educators to spend more attention to the constructive alignment of what is desired to teach, what is taught, and what is then assessed. Third, there is a review of ���What works best?��� in the teaching and learning equation which highlights the power of ���assessment for feedback���. This leads to the fourth and major claim about the revolution in assessment for learning and how it needs to penetrate the corridors of higher education. Finally, the recent changes in New Zealand towards more formative assessment and standards-based assessment in upper secondary school is reviewed to show, among many other benefits, that students may not be satisfied if these methods are not introduced into tertiary assessment. They expect better, they know the power of these methods, and they are more self-regulated than those who came through the black box of normative assessment, where assessment was primarily used to sum up the student���s status in terms of their ���mastery of what the instructor thought he/she was asking in the final exam���. Multiple Outcomes Both society and students are demanding more from higher education. Chickering���s (1969 Chickering & Reisser, 1993) model is still worth noting in terms of these outcomes: i. Achieving competence. ii. Managing emotions ��� from those that interfere with learning (anger, anxiety, hopelessness ��� see Au, Watkins, Hattie, & Alexander, 2009) to those that assist (optimism, hopefulness). iii. Mature interpersonal relations ��� respecting differences, working with peers (probably the most underestimated power in higher learning). iv. Moving from autonomy to independence ��� moving from needing assurance and approval of others to self-sufficiency, problem solving, and making decisions. v. Establishing identity ��� self-esteem and self-efficacy. vi. Developing purpose ��� from Who am I? and Where am I? to Where am I going? vii. Developing integrity. Chickering���s argument is that universities are in the business of developing all seven outcomes and/or they will be assessed by many on the institutional missions, processes and successes on all seven outcomes. While we spend the most time assessing the first, our funding and public success as well as our continual justification as an attractive place for people to come for higher education is as dependent on the other six. Higher education is as much about identity, reputation enhancement, and growing as it is about becoming critics and problem solvers. The (valuable) by-products are knowing more about a topic, being passionate about content, and being learned about a subject.

Excerpt taken with permission from: Hattie, J. (2009). The Black Box of Tertiary Assessment: An Impending Revolution. In L. H. Meyer, S. Davidson, H. Anderson, R. Fletcher, P.M. Johnston, & M. Rees (Eds.), Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research (pp.259-275). Wellington, New Zealand: Ako Aotearoa My prediction is that there will become more emphasis on the latter six outcomes while of course not forgetting the importance of the first (achieving competence). There have been attempts to assess success on the latter six (e.g., the Australian Course Experience Questionnaire), and occasionally universities consider whether they enhance critical thinking and similar non-subject-dependent attributes. I recall, when I was in an Australian university, the debate about measuring critical thinking and one of the best measures was the ���number of complaints��� received by students ��� but it soon lost favour. Probably the most important development hovering on our horizon is the Organisation for Economic Co-operation and Development (OECD)���s ���Assessing Higher Education Learning Outcomes (AHELO)", to explore the possibilities for developing comparative quantitative measures of graduate learning outcomes. This is a project of quite considerable importance as it aims to compare universities across nations, systems and cultures. The AHELO feasibility study has four strands: The assessment of generic skills: critical thinking, analytic reasoning, problem-solving, written communication skills, generation of knowledge, and the interaction between substantive and methodological expertise The assessment of discipline-specific skills: complements the generic skills and will initially focus on selected discipline areas that are most common among universities in the participating countries and less likely to be influenced by unique cultural features. ���The aim will be to assess competencies that are fundamental and ���above content��� that is, with the focus on the capacity of students to extrapolate from what they have learned and apply their competencies in novel contexts unfamiliar to them, an approach that is similar to PISA.��� The first subjects to be included are engineering and economics The measurement of the ���value-added��� or contribution of tertiary education institutions to students��� outcomes: This will be based on two measures. The first one relates to the absolute performance or raw scores of students, since ���prospective students or employers would want to know the ���bottom line��� of the performance of HEIs, departments or faculties��� and the second is a measure of incremental learning (or ���value-added���) ���with a view to assess the quality of the teaching services provided by HEIs. It focuses on the scores an institution would attain after accounting for the quality of prior schooling or the degree of selectivity of the programmes and HEIs��� and The Contextual strand: This will capture contextual measures at institutional level as well as appropriate indirect proxies of learning outcomes, such as: Academic studies and teaching (contact between students, counselling, courses offered, opportunities for e-learning, study organisation and teaching evaluation) Equipment (IT-infrastructure, library, computer workstations, spending for books and journals, rooms) International orientation (support for stays abroad) Job market and career orientation (employment market-related programmes, practice support) Research (number of doctorates, publications and internationally visible publications, extent of third party funding) Study location and TEI (amount of sport, level of accommodation rent, size of TEI) and