Reading students’ minds: design assessment in distance education

This paper considers the design of assessment for students of design according to behaviourist versus experiential pedagogical approaches, relating these to output-oriented as opposed to process-oriented assessment methods. It is part case study and part recognition of the importance of process in design education and how this might be applied in other disciplines generally, through the use of visual thinking and assessment. Making use of experience gained from The Open University’s entry level design course, U101: Design Thinking, the main assessment software (CompendiumDS) is described and presented as an alternative to ‘convergent endpoint’ artefacts of assessment. It is argued that the software and assessment design allow the evaluation of ‘unseen thinking’, providing an immediate focus on process rather than deterministic or behaviourist outcomes alone. Moreover, this evaluation can be applied at scale, without extensive changes to existing systems and may offer a compromise between measuring outcomes and the value of student-centred learning experiences.

At the operational level, Bloom's taxonomy (Bloom, 1984) still stands as one of the cornerstones of educational curriculum design, despite criticisms of its essentially behaviourist agenda and the linearity of hierarchy (see, for example, Furst, 198, Ormell, 1974). Anderson and Krathwol's (2001) update of Bloom's Taxonomy certainly helps address these concerns by introducing subtly (but very significantly) different cognitive elements. One notable difference between these two taxonomies is particularly relevant to this paperthe first was concerned with "things" (such as Knowledge and Comprehension) while the second introduced "actions" (such as Remembering and Understanding).
This difference is at the heart of this paperthe difference between a deterministic and extrinsically-driven view of education, contrasted with an emergent and intrinsic (or learnercentred) vision. Returning to Chomsky(2012) to sum up the dilemma for educators: "do you train for passing tests, or do you train for creative inquiry?"

Tick boxes or ideas?
This is also a challenge in design education and a key starting point for this paper. On the one hand we hope to develop students capable of genuinely creative thinking and ability -activity that goes beyond the simple behaviours that might be associated with a profession or discipline. But on the other hand, we require educational institutions to somehow measure and demonstrate the delivery of such graduateswhilst doing so with fewer resources.
In design education, it is easy to think that we are in some way immune to the dangers of the culture of the "tick box", perhaps believing that the subjective of nature of design ensures that it is not susceptible to objective measures or criteria of assessment. But it could also be argued that design education is one of the most deterministic forms of pedagogy because it requires indoctrination into a way of behaving and thinking in itselfeven more so if the design subject is discipline focused. If the measure of successful indoctrination is by the subjective judgement of those already in the discipline, then this is just as much a "tick box," albeit one with criteria that are at least partially hidden.
Increasing calls for creativity and innovation in graduates (Craft, 2006) require the development of a different set of behaviours -indeed, many of these are not even behaviours. It is suggested that by considering alternative methods and modes of design education we can begin to experiment and implement genuine alternatives to existing paradigms. This position arises partly from the epistemology of design in itself -that we can accrue new knowledge by engaging in the act of design. But it is also proposed because it has been demonstrated as being effective before. For example, the previously cited Colomina et al. (2012) presents a compelling history of this in design education and it is something observed in the case study presented in this paper.

Tensions or difficulties?
This paper is particularly concerned with the tension between the behaviourist and experiential approaches to design education and how mediation of these might be achieved. In particular, the paper responds to two further triggers. A first particular aim is to address Gleeson and Donnabháin's (2009) call for alternative indicators that will "achieve balance between process and product, between responsive and contractual accountability and between individual and system outcomes" (p. 27). It is intended to show that the method and system of assessment presented also mediates between behaviourist and experiential approaches introduced earlier.
Secondly, this paper aims to consider Lawson's (1980) observation that taking an experiential approach in design education is particularly difficult because "there is not a lot of action to be seen [in observing the designer in action] and what is there cannot be readily understood" (p. 216). Further, it is intended to show that the learning design, assessment methods and technologies presented in this paper go some way to actually seeing and understanding the thinking of design students studying at a distance.

Case study or position paper?
This paper is part case study in that we are examining the design and implementation of a system of learning, teaching and assessment in design education. It is a reflective piece exploring a single major aspect of the open online distance learning course U101: Design Thinking. It therefore does not quite fulfil the complete requirements advocated by Boling (2010) or Howard (2011), principally because it extends the question proposed by both authors as the essence of a case study, "How did the design come to be as it is?" This paper considers the design as embodied in the artefact in practice (design as a noun and verb) and, more importantly, that this process continues into operation and practice.
As with the practice of design itself, design education inhabits the middle ground between objective and subjective epistemologies, dealing with "the 'halfway' between people and things" (Koskinen et al., 2011, p. 204) or the multiple readings of reality (Charmaz, 2000). This necessarily requires design education to become as adaptive as the discipline(s) it supports and, as with the practice of design, this paper presents the idea that we take grounded approaches whilst at the same time maintaining ideals and values that we deem important. Unlike any other discipline, design requires both the theoretical idea and the tangible output, the essence of Cross's notion of designerly ways of knowing (Cross, 1982). It seems natural, therefore, that the teaching and learning artefacts should be no less emergent and adaptive.

The context
The Open University is the largest provider of distance-learning higher education in the UK, allowing students to study at under-and post-graduate level on a part time basis, without having to physically attend an institution. Students study individual courses (or modules) that contribute to their overall study plan and these are provided as physical or online material using an online Virtual Learning Environment (VLE). This material is specifically designed and written to allow learning at a distance, containing everything students require in terms of information, activities and assessment materials.
Students are also supported by a regional tutor responsible for around 20 students in a single tutor group. Tutors are the direct point of contact for students studying at a distance and provide support in the subject area, general study skills and advice and pastoral care. Tutors on the course presented in this paper have been largely selected from design practice backgrounds adding a further positive dimension to students' experience (Lloyd, 201;Saddler 1989). The tutor group and tutor-student relationship is an essential component of the OU model of learning and perhaps the most significant differentiator to other distance learning institutions.
A further key aspect of The Open University is its open access policy -students require no previous qualifications or evidence of study to undertake a course. This ensures that the university maintains a very diverse student population in all demographic senses. In design education, it has a particular significance in terms of "self-selection," where other institutions will find themselves with students who have some specific notion of a particular discipline-oriented career or study path . Put together, this model of distance education is the Supported Open Learning (SOL) model (Ison, 2000).
Design education at the Open University has been provided since the 1970s and the unique nature of the institution has required that it always took a general rather than specialist view of design and design education (Holden, 2009). This arguably allowed the unique formulation of the early notions of design thinking through the work of the Open University's Design Group. Design at the OU is necessarily focused on the individual student's methods rather than the domain specific output exhibited in discipline focused courses.
In the newest course (at time of writing), U101: Design Thinking, the focus is entirely on design thinking and doingin other words the process of design. The course requires students to learn design thinking by doing. Students are challenged with activity from the very start of the course and these are continually encountered in the course material as students' progress. A variety of learning material is provided to allow students to make use of different modes of learning, permitting genuine blended learning to emerge. In fact, it is thought that the design and conception of the module itself taps into deeper emotional and metaphorical connections with students .
A typical Open University course will take years to design and develop, with considerable resources going into the production of material and the design of the learning and assessment elements. U101: Design Thinking began its design life in 2008 and had its first live student presentation in 2010. To date (January 2014), 1,805 students have successfully completed the course.

The artefact
The course is divided into four blocks of six weeks study, each of which have a substantial "scale theme" associated with all the learning material and activities (ranging from individual design thinking to global contexts). At the end of each block, students submit a piece of work online that is formally assessed by their tutor who also provides written tuition feedback.
This continuous, discrete assessment is a key feature of the OU assessment system and provides students with summative and formative checkpoints to reflect on their progress and to develop their work. The formal mechanism for assessment satisfies the academic standards required by the institution and the detailed tuition feedback method provides critical advice and feedback on students' progress and work. This is known as the Tutor Marked Assessment (TMA).
For U101: Design Thinking, this method of tuition and assessment is absolutely critical in allowing student designers to learn a process of design thinking for themselves. A key feature is the focus on the assessment of process, not final product, in a medium that fits within existing systems and procedures. In order to deliver this in a distance learning environment, software was specifically developed for the course. This software, CompendiumDS, is a "digital whiteboard" in which nodes can be arranged spatially similar to a mind-map ( Figure 1). Nodes can consist of a variety of media and these can be connected to allow patterns to be represented, or "maps." These maps are created by students and communicate their design process. Thinking (screen shot) The CompendiumDS software was developed to be as simple as possible to use making use of "drag and drop" or visual interaction wherever possible. Students can quickly populate a map with artefacts and then experiment with setting out these maps -or create a process they might wish to go through by actively using CompendiumDS to "design their process" as well as communicate it.
The entire assessment process was designed to fit into existing institutional systems, a critical design constraint from the start. A typical offering of U101: Design Thinking will have a student population of around 500 students, meaning that assessment has to be handled efficiently and effectively whilst still maintaining the standards required for academic and discipline rigour. The output from CompendiumDS is a file that stores the data (as native file formats) and map (as xml) in a single package, readable using the software on any major operating system. This file is then submitted online and processed at the institutional level. Tutors then assess and provide feedback to students on their work. In parallel to this, a sample of students' work and tutors' assessment is itself checked for quality and consistency of assessment and tuition feedback Four substantive Tutor-Marked Assessments (TMA01 -TMA04) are required in the module covering most of the learning objectives. Students also submit a final End of Module Assessment (EMA) to complete the learning outcomes and produce a portfolio of work. Each TMA is based on the "scale" of the block it relates to and builds on skills and processes developed in previous work.
The premise is that, as students engage in increasing "scales" of design contexts, they also have greater freedom to represent the process they undertake as they work towards a concept proposal or embodied design. Each TMA gradually develops students' skills in familiarising themselves with this mode of communication until they are confident enough to complete blank maps according to how they wish to communicate their process of design thinking. As they do so, they are required to engage in active reflection on the processes they are going through. It might already be clear that the pre-populated maps in early TMAs are representations of design processes themselves. For example, in TMA01 (Figure 2), students were asked to undertake three activities in an exploration phase which then led to three idea developments in the next stage. It is very rare that students are unable to understand and make use of this simple process of "seeing and doing" and quite often the results (Figure 3) exceed anything that might usually be expected from such a simple process . For this paper, a single example is provided to give a sense of the outcomes that can be achieved. It will focus on the second assessment, TMA02, which is broken up into 4 weeks of activity, each week represented using nodes in the TMA map ( Figure 4). Each week contains a "sub-map" node which allows further nodes to be added inside it to communicate sufficient detail of the substantive activity at each stage of the process ( Figure 5). As students progress through the weeks, they work through an iterative design process by recording their activity and thoughts. At each stage (each week) students are also required to reflect on their process and describe particular formative events that have occurred. The design layout of this particular TMA has already seen several iterations since the start of the course, mainly to deal with the complexity of the activities required and to respond to student feedback. This iteration and improvement is an essential aspect of the overall processthe design is never "complete" and has to continue to respond to its context and users.

The experience
The experience of this form of assessment must considered from two points of view: the student and tutor. The tutor is attempting to understand and empathise with the student's thinking in order to help develop it through feedback. The motivation behind this is to "close the gap" between where the student is and where they need to bea classic definition given by Ramaprasad (1983). For feedback to work, the student first has to understand it.
An original curriculum design intent for the course was that, rather than describing exactly what it is we wish students to be able to appear to do (what behaviours we might wish to see copied), we considered what the student will experience (what activities we might hope they engage in or discover for themselves). By allowing students to demonstrate their own methods and approaches, tutors have no choice but to take a student-centred approachin effect, to try to understand what the student was thinking and doing in going about their work. It is proposed that the format and structure of assessment in the course allows precisely that. Figure 6 shows a student's TMA 02 as the tutor would see it. At a glance, the tutor can readily see some of the key stages in the overall process. The top row of nodes represent the substantial activity and content that lead to the outputs presented in the second row. This second row provides the tutor with an immediate overview of what was achieved. The software allows tutors to see further detail by rolling over images ( Figure 7) and text (Figure 8) in the nodes, providing an opportunity to read the map quickly and effectively.  The simple expediency of being able to preview images and text quickly cannot be overlooked. Tutors wish to quickly understand what the student is thinking and doing at each part of the process being presented. In doing so, they are recreating (in some sense) the process the student has undertaken. For example, Figure 9 shows a "sub-map" of Figure 6, providing evidence of the actual work that has taken place. In this example, a tutor can quickly see (both literally and conceptually) the depth of activity and thinking that has gone into this part of the work. Figure 9. Problem identification in Week 1 (cf. Figure 6) This view then allows tutors to provide specific feedback on the process the student has gone through, directing students to look at particular nodes and make observations or pose questions. From student feedback on tutor feedback, we are confident that comments around particular nodes and artefacts do allow students to recall and reflect on particular aspects of their process more directly. This allows tutors to reinforce students' confidence in those activities that have been beneficial to the process and to reflect further on those that have not. In fact, the type of observations that are made by tutors can be quite subtle and nuanced, going way beyond what might be expected from the assessment artefact itself.
A further critical aspect of all assessments are the reflection nodes (see bottom row of nodes in Figure 4) where students are required to discuss their thoughts on the process they have undertaken in each week. Students are encouraged to consider their emotional and personal responses to the process they go through, starting in the first TMA by considering what they like or dislike; what they thought went well or not so well; and (most importantly) what they would do differently if they were to repeat the work. By focusing on their personal (and conscious) response to the work we are trying to engage students in the process of reflection-in-action (Schön, 1987) by consciously externalising their reflection. This mechanism is essentially the same as Sadler's (1989) premise that, for effective summative learning, students should monitor the quality of their own work during production.
This synchronous reflection is also reinforced by asynchronous summative reflection of the feedback received from the tutor. Walker (2009) discussed this form of feedback as a mechanism, stressing the importance of the students' understanding of the feedback to allow this to take placewith reference to Biggs and Tang (2011) in terms of taking a student-centred approach to allow this to be more effective. It is argued that the ability to view and feedback on process allows a more effective form of feedback to take place: that by communicating to a student using their own representations of process a tutor is immediately in a student-centred position.
The form of assessment presented here allows the student and tutor to "close the gap" between the production stage and the artefact submitted for assessment. In many ways, assessing the final outcome is perhaps too late for many students and may also miss critical elements in the process. By assessing process and providing feedback directly on this (and doing so in a visual way) it is argued that students have a greater opportunity to develop their abilities during production.

The reflection
CompendiumDS has been in use since the start of U101: Design Thinking and in this time we have built up a considerable community of knowledge around assessment of the design thinking process using the software as well as evidence of student success and feedback from students themselves.
The narrative presented here is typical in many ways but it is also atypical in that it represents a student who has actually "got it" (possibly even in the material just presented)that is, they have demonstrated a level of internalisation of a number of cognitive processes, attitudes and behaviours. It is suggested that they have demonstrated several of Anderson and Krathwol's (2001) taxonomic levels as embodied design activitythey have not simply created, they have applied, analysed and evaluated as well.
For some students (as with educators), however, the very notion of non-behaviourist learning is incredibly difficult to understand or believe in. The word "believe" is used here deliberately since the evidence from student feedback is showing that cognitive dissonance occurs for a certain percentage of students or, at least, a significant "resistance to learning" (Atherton, 1999). That is, the ideas being presented in the course are incompatible with their own worldview of education and/or design. For some students, only a deterministic and behaviourist approach is ever imaginable.
In between these two extremes, sit the majority of students, who will take on board some aspects of the module but not necessarily complete the entire conversion to process-driven methods of design. Research is ongoing to determine how to improve the "reach" of these methods to these students, for example, multimedia feedback is currently being trialled to engage student-tutor dialogue beyond the current linear model.

The Analysis
In the introduction to this paper, a number of dualities and contrasts were presented. For each of these dualities, it is argued that the method of assessment presented demonstrates that, as with the discipline of design itself, it is possible to inhabit a middle ground of ideologies and approaches.
Firstly, it is entirely possible to create learning material and assessment artefacts that embody both deterministic and subjective elements. It is hoped that this is clear from the case study presentedit is both a deterministic and student-centred method of assessment. This is possible because it is the process that is assessed, not the output, meaning that the entire assessment system is necessarily student-centred from the start. The change to existing paradigms is simply this: experiential processes are assessed using objective frameworks (not behaviourist outcomes), within which the individual student can be considered using experiential criteria. The mediating factor, the tutor, is still absolutely vital in taking a "human" view of student work but this is still perfectly possible within a deterministic assessment framework. This observation is certainly not new or unique. What is important here is the context in which this takes place: distance education at scale. It is argued that this presents an alternative in line with Gleeson and Donnabháin's (2009) call for alternative indicators.
Secondly, it is also possible to develop behaviours at the same time as taking a student-centred approach to teaching and learning. One of the interesting features of the summative aspect of the assessment presented is that it starts with a behaviour (creating something) and then actually works backwards to embody other cognitive elements such as analysis or reflection. Returning to Anderson and Krathwol's (2001) updated taxonomy, it is argued that the hierarchy presented is not necessarily a progressive model. That is, it is not simply the case that students will begin with Remembering, moving on to Understanding, and so on. In fact , students may start at any level or may even work across several. It is argued that the form of assessment described in this paper allows this to take placefor example, students may create without knowing how or why (and that then becomes their next learning goal through summative feedback). The key to this aspect is to realise that any taxonomy is an idealthat, in the real world of students and tutors, individual approaches matter. Having systems and processes that allow this to be recognised and developed is vital.
Finally, it is argued that by students presenting their work in this way it is actually possible to "see" what they are thinking, answering Lawson's (1980) dilemma of not being able to read designers' minds. It is not proposed that this is in any way a literal glimpse into the thinking of studentsat this stage in the research it can only be described as, at least, a view of what the student "thinks they are thinking." But even this is still a significant step beyond the further remove of starting with the final artefact with no reference to the process at all.

Summary
In summary, it is contended that U101: Design Thinking demonstrates that accepted paradigms of existing modes of design education are not the only approach that can be taken in design education. Moreover, the positive effect on both student outcomes and experience is such that these alternative approaches are well worth exploring further, whether or not the context is one of distance learning. The success of this simple change to communication and assessment of process using visual mapping software allows students and tutors to engage in a significantly valuable dialogue. By supporting this method with institutional frameworks and standard practices, the tensions between "tick boxes and ideas" in learning can be resolved; the behaviours can be considered as part of an experiential process; and all this can be accomplished in a systematic way that allows large numbers of students to be considered as individual learners.
This comes from and is supported by taking a design thinking approach to learning itself. By treating the learning design as a design process itself, we are able to avoid polarised positionsthey can become embodied in the process itself. And this process continues. U101: Design Thinking is constantly evaluated, changed and exposed to critical review from academics, tutors and students. It is perhaps this responsiveness that matters most rather than which polarised position is taken.