EVALUATION OF LEARNING MATERIALS : A HOLISTIC FRAMEWORK

This paper presents a holistic framework for evaluating learning materials and designs for learning. A holistic evaluation comprises investigations of the potential learning potential, the actualised learning potential, and the actual learning. Each aspect is explained and exemplified through theoretical models and definitions.


Introduction
The aim of this paper is to present a holistic framework for evaluating learning materials and designs for learning.We and our colleagues have been participating in the development of a variety of learning materials based on empirical research and theoretically generated design principles.On the basis of both empirical data and theoretical principles, we have argued for the success of some learning materials and the shortcomings of others (Bundsgaard, 2005(Bundsgaard, , 2009(Bundsgaard, , 2010a(Bundsgaard, , 2010b;;Bundsgaard & Hansen, 2010;Hansen, 2010aHansen, , 2010bHansen, , 2011)).But we have felt a still stronger need for a firmer basis for these claims and a still more urgent call for a thorough framework on the basis of which comparisons between learning materials can be made.The ultimate, yet unattainable, goal is to develop a yardstick to measure which one of several learning materials is the best.
Before we present the framework, we will give a short overview of essential work done in the area.A number of studies and frameworks focus on an analysis strategy where the learning material or course is evaluated as a text, i.e., the object of study is the explicit and implicit descriptions of student behaviour and expected learning outcome.Baker (2003) has developed a framework for design and evaluation of Internet-based distance learning courses on the basis of Ralph Tyler's (1949) theory of curriculum development, Bloom's (1956) learning taxonomy and Blanchard and Johnson's (1984) SMART framework.Leacook and Nesbit (2007) have developed a very popular and well-described framework called Learning Object Review Instrument (LORI).By using LORI, evaluators can create reviews consisting of ratings and comments on nine dimensions: content quality, learning goal alignment, feedback and adaptation, motivation, presentation design, interaction usability, accessibility, reusability, and standards compliance.A primary goal of LORI is to balance assessment validity with efficiency of the evaluation process (Leacock & Nesbit 2007), and thus evaluation is not based on close reading or detailed analysis, but on the evaluator's overall impression of the learning object's dimensions of quality.
Another congenial framework is Laeremiddeltjek (English: learning material check out), a systematic tool for assessing learning material through evaluation within six parameters: accessibility, progression, differentiation, facilitation, coherence and legitimacy.The framework is intended to be used by researchers as well as teachers to give a swift evaluation of learning material, qualified by a theory-based model which offers a functionalistic approach to learning materials as tools helping the user with solving specific problems related to learning and teaching (Hansen & Skovmand, 2011).
Another group of frameworks focus on teachers' evaluation of learning materials.In the Mod4L project (Falconer et al., 2007), the goal was to investigate what made teachers choose a specific learning design, and to develop a framework to evaluate learning designs from that perspective.The framework consists of four stages of sharing and reuse of learning designs and of eight principles related to the descriptions of learning designs and their underpinning pedagogies that influence practitioners choice of learning designs (Falconer et al., 2007).
A third group of studies focus on evaluating learning material in concrete situations through ethnographic research, and, in some cases, such studies include test-based evaluation of the participants' learning outcome.Harley, Seals and Rossen (1998) studied a learning tool to support object-oriented design of computer programs through a problem-based approach.They used video to capture the students' collaboration and screen-capture to record the interactions with the computer.The analysis focused on the quality of the learning material through analysis of students' interaction with the system and the resulting success in problem solving.Further, the research group centered around Neil Mercer and Rupert Wegerif have developed a number of innovative computer programs based on the hypothesis that students improve their thinking skills by developing a few ground rules of dialog.The group has done a number of evaluation studies to support their hypothesis including discourse analysis of transcripts based on video recordings of students' collaboration, pre-and post-intervention reasoning tests etc. (Wegerif 2004;Wegerif & Mercer 2000).In Krauss and Ally (2005), the outcome of integrating the evaluation of learning objects in the design process is studied through a variety of methods from student and faculty ratings and survey questionnaires to think-aloud sessions.
The literature reviewed here and all other literature we have knowledge of in this area focus on some aspect of learning materials, such as its intentions, its use, and the learning outcome, but none provide a holistic view of the task of evaluating learning materials.Therefore it is our intention to provide such a holistic perspective.

Holistic evaluation of learning materials
As will appear from the following, we use two connected but separate concepts: learning materials and design for learning.We understand learning materials as artifacts, e.g.textbooks, blackboards, computers, while we define a design for learning thus: Definition: A design for learning is a constellation of artifacts (which can be called learning materials) arranged (in space) and articulated (in time) by someone with the intention to initiate and support someone's learning.This means that a design for learning can comprise several learning materials and learning materials can "prescribe" or implicitly presuppose a design for learning.A prototypical learning material can, for example, be produced by a group of authors and published by a publisher as a text book or an ICT learning object, while a design for learning can be the product of a teacher's explicit or implicit plans for the layout of the room with the texts and objects it contains.Learning material includes more or less explicit descriptions or expectations of the learning situation.An ordinary textbook implicitly presupposes that the teacher will teach from a desk at the front of the room while the students will sit in rows and listen, and carry out the assigned tasks.For example, an educational computer game presupposes that the students will sit in front of their computers and play while a storyline framework presupposes that teachers and students will collaborate in many places and through a variety of interactions.
On the basis of these definitions, we present the framework as a three-step method for performing a holistic evaluation.

Definition: A holistic evaluation of learning material comprises investigations of:
a. the potential learning potential, that is, the affordances and challenges of the learning material, and the competences supposedly supported when working with the material; b. the actualised learning potential, that is, the potential for learning when the design for learning is enacted by integrating the learning material in a situation in a given context; and, c. the actual learning, that is, how the participants actually develop their competences through working with the learning material or enacting a design for learning.
The first part of an evaluation consists of analysis and interpretation of the learning material as a text.This analysis leads to a description of the potential learning potential.When introducing learning materials into a specific situation, some aspects of the potential of the learning material are actualised through integration with the historical constellation of artifacts in the situation.The students and teachers more or less deliberately choose to supplement the learning material with tools and materials, and these phenomena comprise the design for learning, which constitutes the actual learning potential.When the teachers and students have worked with the learning material and have thereby enacted the design for learning for a period of time, they are expected to have learned something.The producers of the learning material might have articulated the expected outcome, and the analysis of the potential and actual learning potentials may point to a further range of potential outcomes.The actual learning outcome can be measured against these two forecasts.
In all probability, no single evaluation of learning materials can comprise all aspects of this complex framework.However, the framework can be used as a heuristic to evaluate and discuss the shortcomings and benefits of evaluations of learning materials and as a tool for planning an evaluation of a learning material.
The triple division can be understood as a temporal structure: before, in, and after use.However, it is also important to understand it as a methodological structure.Each of the three points can be regarded as a perspective on the evaluation of learning materials, each characterised by its own logic and accessibility because the three temporal phases correspond to three different types of participation and relation to the teaching situation.

Some basic definitions
The object for evaluation seems quite distinct and obvious, at least in the case of learning materials.But what are learning materials?At first glance they appear solid and observable; objects that can be investigated, with distinct features to describe.But in fact the notion of learning materials is a fluid concept used to denote different types of materials and resources in relation to teaching situations: textbooks, on-line learning materials, educational games, mobile phones, blackboards, digital boards, charts and literature among others.For the purposes of our study, we define learning materials by dividing them into three types: Definition: Types of learning materials • Functional learning materials (tools) characterised by their facilitation of learning and teaching: including black and white boards, computer applications, projectors, and mobile phones.
• Semantic learning materials (texts) characterised by their meaning as constituted by signs and semantic references: including film, literature, charts, pictures, paintings and other texts and objects with references to specific domains of experience.
• "Didacticized" learning materials characterised by combining tools and texts and facilitating learning and teaching: including textbooks, online teaching materials, and educational games.
The word didacticized is a neologism created from the word "didactic," which in its Germanic form (Didaktik) signifies planning and reflecting on teaching.Thus, something which is didacticized is the result of the process of anticipating and prescribing the activities that the teacher and students (should) perform; that is, a didacticized learning material has a more or less implicit design for learning inscribed in it (Hansen, 2006).
The first two types have something in common, because they do not have designs for learning coded into the material; they become learning materials through being part of a contextual design for learning which codetermines the use of the learning material in the situation.These different types of learning materials and their integration into designs for learning must be taken into account in order to have an inclusive definition of learning materials.Learning materials are materials and tools integrated into a design for learning: whether a contextual design for learning co-determining the use of the materials, or an immanent design for learning coded into the material, or both.Thus, as we have already underlined, there is a close relationship between the notions of learning materials and designs for learning, at least viewed from the material point of view.
In the definition of the concept "design for learning," the word "artifact" was included.We use this concept in a similar way to theoreticians (for example, Cole, 1996;Säljö, 2005;Wartofsky, 1973), who define artifacts as both material tools and objects and as ideal "in that their material form has been shaped by their participation in the interactions of which they were previously a part and which they mediate in the present" (Cole, 1996, p. 117).Some artifacts have a material aspect while others are more or less completely ideal even though ideas or mental models need some kind of external representations (for example, spoken words) if they are to be shared with others.In an educational context, the external representation is crucial, underpinning the important role of learning materials.We have elaborated a typology of artifacts adapted from Cole (1996), who, in turn, adapted it from Wartofsky (1973), while emphasising that there are important differences between our definition and those of Cole and Wartofsky.

Definition: Artifacts
• Primary artifacts: perceptual objects we can point to, such as (a) physical objects, tools and technologies, (b) layout of the physical environment, and (c) texts, that is, the physical signs or marks that can be pointed to; • Secondary artifacts: conceptual objects we can refer to as existing, such as (a) rules: laws of nature, laws of the state, moral laws, and (computer) algorithms, (b) processes: social algorithms: what is to be done first, next, and then in a social context, and (c) mental and social models: world view, personal relations, authority, ideology; and, • Tertiary artifacts: objects of the imagination that we can refer to as if they were existing, which Wartofsky (1973) defined as imagined worlds, like those of art and fiction.
"Artifact" is a term used for all kinds of objects and processes that are the results of human activity.In our conception, artifacts are social phenomena that derive their value and meaning from the situational and social context in which they are used and created.This becomes clear when we analyse concrete use of learning materials in situations where (a) the status of artifacts is constantly negotiated, and (b) interactions between teachers and students are mediated by artifacts.

Potential learning potential
There are some consequences and noticeable challenges emerging from our definitions and hence some challenges for the demarcation of the analytical field that is the object of evaluation of learning materials.First of all, we will point out that although desk-based analysis and evaluation of learning materials might result in relevant and important findings, they will suffer from being de-contextualised and distant from their situated use.It is possible to a certain degree to analyse and evaluate learning materials without leaving one's desk, but one must be aware that the analysis is (to be) grounded on a presupposed knowledge of prototypical designs for learning in prototypical situations.It is not enough to analyse and evaluate the learning material on the basis of a "check list" or a formalistic grammar of the well-formed text.We therefore present a functionalistic and phenomenological model for analysing designs for learning.This model (Figure 1) describes a number of focal points or perspectives on learning materials and identifies a series of steps constituting a meticulous textual analysis of the specific learning material.Each step in this model requires further explanation and elaboration.In this context, we focus on the analytical triangle shown in Figure 1, which we explain with reference to the speech act, that is, when someone is saying something, about something, or in order to have someone do something.All learning materials and designs for learning can be seen and analysed as speech acts (cf.Austin, 1962) containing a perceptual representation (the expression: in Austin's words, the locutionary act) signifying a thematic field of attention (the intention: in Austin's words, the illocutionary act) and appealing to an addressee, framing some kind of enactment (the activity: in Austin's words, the perlocutionary act).
Even in traditional teaching, where the teacher lectures and the students sit in rows, there is an act and a framing of a certain activity, because the students are supposed to sit still, listen and understand the theme of the lesson in a certain way.Thus the teacher's speech contains an implicit speech act: "I want you to sit still, listen very carefully, see this before your eyes and understand it in a certain way, specified by virtue of my framing."This triangle becomes explicit when the participants experience problems with the communication and the teacher or students have to focus on one of the three aspects; for example: 1. expression, when the forms of representation (medium, modes, genres and text types) must be modified in order to achieve an adequate presentation; 2. intentionality, when there is a need for reorganizing the intentional structure in order to link the aim and content to the students' life world, and finally, 3. activity, when the teacher finds it necessary to make an appeal to the students or request a certain action.
Similarly, the triangular structure often becomes explicit and observable when designs for learning are coded into learning materials.We use the analytical triangle as a frame of reference when analysing the learning materials.Our focal points can be divided into four categories: C. Situational perspective: organisation (of interactions, space and time, room requirements, organization of furniture, consecutive time needed etc.), and relations (between participants, and to the outside world).
D. Society's perspective: integrity (concerning transcendent coherence in relation to the norms and values of the actual school culture), and legitimacy (measured using governmental policy, national curriculum and up-to-date knowledge as yardsticks).
It is possible to argue that these focal points are core elements in teaching situations, with their basis in general theories of learning and education.In this connection, we restrict ourselves to arguing for their relevance with reference to the logic of the teaching situation: a) the students are supposed to learn something and develop relevant competences within the horizon of a democratic society, b) the teacher is supposed to carry through the planning, implementation and evaluation of teaching, c) the participants are rooted in a situation in time and space (and thereby confined by the material basis) and co-determined by the relations between participants and their historically developed local culture; and d) the realisation of learning and teaching aims is supposed to be legitimated with reference to the surrounding world.These shifts in focus from the students' to society's perspective imply an epistemological and methodological shift from cognitive and socio-semiotic to sociological approaches to the teaching situation.
We use the analytical triangulation of expression, intentionality and activity as a pragmatic tool to ensure an adequate frame for analysing the focal points and mapping the learning potential, stressing that there is no direct access to learning and the content of teaching, whereas all processes of learning have expression, that is, appearance of forms and representations, as their points of departure.

Competences
The potential learning potential can be described in terms of the competences that students can be expected to develop.These can be described from two perspectives.Firstly, competences are closely related to the situation in which they are developed and used and can thus be described as competences of the community of practice (Wenger, 1998) in which the students participate.Thus students first learn to act as students in the ways in which the institution, design for learning and other people structure their participation.If students are expected and forced to sit in silence and listen, that is what they learn -though they might of course also learn something through what they hear and see.If, on the other hand, students participate in a more elaborate practice, for example as journalists on a newspaper, they can be expected to learn to act, think and feel like journalists (Bundsgaard, 2009;Shaffer, 2006).From a community of practice perspective, competences can be seen as knowledge of and capacity for the practices, attitudes and approaches of a given practice.The potential learning potential can thus be outlined on the basis of a description of the practices in which students will participate while enacting the design for learning co-determined by the learning material under scrutiny.
Secondly, competences can be described on a more generalised level (cf.Bundsgaard, 2006;Rychen & Salganik, 2003) as the cognitive and practical skills, the knowledge, attitude, and motivation that students develop while using the learning material, cf. the demand-oriented competence definition by Rychen & Salganik (2003).This perspective defines competence as the capacity to rise to the challenges of a situation; in that way it is comparable to the first perspective, but will have more focus on transferability between practices, and will seek to describe competences in a cross-situational wording.For example, when students collaborate on the production of a newspaper, this perspective will focus on the students' development of productive communicative competences and describe these under such headings as attention to the addressee, knowledge of the typical characteristics of the genre and ability to conform or consciously deviate from it.
To estimate the potential learning potential, it is necessary to have a thorough understanding of the prototypical design for learning implied by the learning material, and thus to analyse the design in all its complexity.This part of the evaluation is a textual analysis and an analysis of the implicit or explicit description of the teaching and learning situation.
The next step in a holistic evaluation of learning materials is research into the actual use in practice.Such investigation can lead to an identification of the actualised learning potential.

Actualised learning potential
The actualised learning potential of learning materials is a function of several factors in the situation where the learning material is used.Thus, a design for learning unfolds some potentials of the learning material in the situation and prevents others from unfolding.To further develop the understanding of how learning material functions together with other factors in the teaching situation, we elaborate on three dimensions: the spatial situation, the social relations, and the temporal processes.

Spatial situation and social relations
One of the most common conceptual models of the teaching situation is the so-called learning triangle, which depicts three important aspects of a teaching and learning situation, namely the teacher (T), the student (S) and the content (C) being taught.But the learning triangle has a number of shortcomings, especially if it is the mental model of the participants or the producers of the learning materials.It conceals some of the most important traits of the teaching and learning situation: for instance, that there is more than one student, that artifacts are dealt with by participants who are related in complex power relations, and that the interaction is situated in a context.These shortcomings are addressed in the model of the teaching and learning situation in Figure 2 (cf.Bang & Døør, 2007;Bundsgaard, 2005). .The media may be paper, screen, or stone while the mark might, respectively, be the inkstains, light-dots (pixels), or carving made by using a technology, perhaps, a printer, a light emitter, or a hammer.
There are always at least three subjects participating in a communication situation: S1 and S2, who produce and consume the marks in the media, and S3, those who participate in setting the stage, existing in person or in the minds of S1 and S2 or experiencing the consequences of the interaction between S1 and S2.S3 can be understood as the ones influenced by or influencing the situation.Because we are addressing questions regarding teaching, we substitute the general concepts of S1 and S2 with the specific roles of teacher (T) and student (S).
This model shows that there is more than one student in a teaching situation and that the students relate to each other as well as to the teacher in distinctive ways.Even when the students are not expected to work together, they will make a difference to each other.Through the concept of "possessed" texts (|T) the model moreover shows that each student and teacher has his or her own individual understanding of the situation and the texts and artifacts involved.In a teaching situation, the participants each carry their own understanding of the "meaning" of the manifested texts and other artifacts.Successful learning material therefore supports many different acquisitions of the content, and not least, it allows for the presence of several interpretations and value systems in the situation, leading to different equally well-grounded understandings and texts.Nothing happens outside of a context.We define the context as the entirety of artifacts, subjects, society and culture related to the situation.In principle, that consists of the whole world, but in a specific situation some part of the context will be more closely related to the situation than other parts.

Temporal processes
The models presented above are situational models representing still images of the situation.But as teaching and learning take place in time, an investigation of processes is needed in order to understand how learning materials prescribe and support these processes.We present a few models showing how different perspectives lead to different insights.

• Interaction sequence models
Following studies into classroom talk patterns, Sinclair and Coultard (1975) described a typical teaching process as consisting of three functions: the teacher initiates the process by asking a question, a student responds, and the teacher evaluates the response.Sinclair and Coultard (1975) briefly describe this as the IRE structure of teaching.Extending from this line of thought, other models of teaching and learning can be described as a number of speech acts.For example, Wegerif and Mercer (2000) point out that introducing a single component, namely student discussion of the question before responding, may lead to a completely different process and learning.This type of interactional process model is open to ethnographic observation supported by video recordings, and the function might for example be found through analysing transcribed passages of student talk and action.

• Workflow models
On another level, we find workflow models showing how teaching and learning are comprised of sequences of interaction sequences, that is, the participants enact a certain workflow.Staffan Selander and his group of researchers are developing and working with such a model, which they call the learning design sequence.This model (Figure 3) shows how teaching and learning are performed as transformations of content.In the primary transformation unit, students transform the modes and media (which we understand to be what we call learning materials) by working with the content in a social context, thereby forming their understanding.In the secondary transformation unit, the students transform their forming to a representation, which in itself or through presentation to an audience gives them the opportunity to discuss and meta-reflect on their learning process.
Figure 3. Learning design sequence (Selander, 2008, p. 17) Even though this is described as a model of "teaching in formal contexts," we will maintain that it is a model of certain kinds of teaching organisation where the focus is on processing knowledge, that is, for example, where students alone, in groups or together with the teacher in class investigate a subject, solve a problem or write an assignment.The model does not, at one end of the scale, describe a teacher giving lectures, nor, at the other end of the scale, does it describe situations where students collaborate on common tasks, for example simulating an out-of-school practice or producing art or handicraft products.
Phase models describing the different phases or functions performed by students in activities such as projectbased learning and simulations of out-of-school practices are complex workflow models.An example from the other end of the spectrum is a traditional teacher-led workflow.The teacher gives a lecture, checks the students' understanding (often using IRE-structured interaction), points out which pages in the text book should be studied and which tasks solved, and then the students work on the tasks alone or in peer groups.Finally, the teacher leads a class conference in which individual students present their solutions to the tasks.We call this type of workflow a Lecture-Recitation-Seatwork-Plenary session (LeReSeP) (Bundsgaard, 2010b;Bundsgaard & Hansen, 2010).
It appears that different workflow models accentuate different aspects of the complex processes in classroom practices.Different workflows offer differing opportunities for the participants in the teaching and learning situation.When participating in a traditional teacher-led workflow with integrated IRE-structured interaction sequences, students become good at answering questions, but they do not practise such activities as discussion, idea generation, and problem identification.Therefore certain competences are trained and developed in some workflows and not in others.Learning materials relate to workflows in two ways.On the one hand, learning materials (more or less implicitly) prescribe workflows, and on the other, the learning materials support the teacher and students in the implementation of workflows.Therefore the workflow models prescribed in the learning material should be part of the analysis of the potential learning potential.In the investigation of the actualised learning potential, attention must be paid to how far the workflow follows the intended models, and how it deviates from these models, and models should be proposed that might better describe the actual practice.
The learning materials prescribe and support workflow, but, of course, they are not actualised until they are put into practice.The processes are orchestrated in practice by students and teachers who use the resources in the learning material as well as other tools and experience from other processes (including especially, of course, those known from previous educational contexts).When investigating how learning materials support workflow, focus can be oriented towards the ways in which these re-organisations are possible.
For the participants, some workflows are well known and comfortable while others might be experienced as boundless and confusing.The adequacy of the prescription and support of workflows in specific learning materials is therefore very context dependent.

Researching into situated designs for learning
The models above are intended to serve as heuristic models for research on practices both inside and outside of the classroom.When evaluating the actualised learning potential of learning material, one is oriented towards how the constitution of the situation, that is, the contextualisation of the learning materials, takes place.The analysis should focus on (a) which aspects of the potential learning potentials are actualised; (b) whether they work in this context, in which ways is the situation a success, how far do the participants feel engaged in and motivated by the work; and (c) what are the challenges in this context of using these learning materials and thereby enacting a design for learning.
The final goal of these analyses is to generalise analytically from the specific context to more general claims about the affordances and challenges the learning material will meet in a context of the kind examined.
Research into situations is oriented towards effects, that is, what are the effects of changing these factors in this situation.The research often makes use of anthropological methods, but it can also be more quantitative, with questionnaires answered by the participants, collection of products, artifacts used and created in the work, and participants' self evaluations.

Actual learning
The last perspective of a holistic evaluation of a design for learning is the analysis of the outcome of working with the design for learning under scrutiny.Outcome can be measured in many ways and be related to many factors.A measurement and estimate of the outcome must take its point of departure in the explicit learning goals and in the potential learning potential that was analytically found in the first phase of the evaluation.
When evaluating the outcome of a given effort, one must have a clear definition of what kind of outcome counts as important.With respect to education, the outcome is learning, which we define very briefly as change in capacity (Illeris, 2002).More specifically, learning can be defined thus:

Definition:
Learning is the movement from chaos/conflict/non-capability that takes place in response to combination and change (or because the intuitive understanding or capability matures) so that the person is capable of doing something he or she could not do before (in situations in which it could not be done before).Here, capable of means: can do (bodily and mentally), can express (communicate), can understand and combine (think), can evaluate (ethically, politically), can perceive (sensibility), can feel (emotion), and wants to (attitude).
The outcome of education should be learning, but not any kind of learning.Thus one must differentiate between: 1. Intended learning 2. Unintended, but valuable learning 3. Unintended and undesirable learning A design for learning that supports the intended learning goals but at the same time makes the students learn something very undesirable (for example, that there is no point in participating in developing the community for the sake of the common good) might be even worse than a design for learning that does not support the intended learning goals very well.Thus, even in an "objective" measurement of learning outcome there is a normative aspect.
One of the significant challenges of assessing the actual student learning is related to connecting the potential learning potential of the learning material to the students' change in capacity.In this framework, the SOLO taxonomy (Structure of the Observed Learning Outcome) developed by John Biggs and Kevin Collis (1982) can serve as an offset for this purpose.
Biggs presents the SOLO-taxonomy as a model on his homepage 1 .It consists of five steps from: incompetence (prestructural) over one relevant aspect (unistructural), several relevant independent aspects (multistructural), integrated into a structure (relational), and ending with learning being generalised to a new domain (extended abstract).Each step is described by sample verbs indicating levels of understanding and a coordinate system with competence on the y-axis indicates that higher steps corresponds to higher competence.Biggs (2011) describes the characteristic of the progression thus: As learning progresses, it becomes more complex.SOLO, which stands for the Structure of the Observed Learning Outcome, is a means of classifying learning outcomes in terms of their complexity, enabling us to assess students' work in terms of its quality not of how many bits of this and of that they got right.At first we pick up only one or few aspects of the task (unistructural), then several aspects but they are unrelated (multistructural), then we learn how to integrate them into a whole (relational), and finally, we are able to generalised that whole to as yet untaught applications (extended abstract).(para.1) Biggs' description can give the impression that learning is taking place as a linear progression from the simple, unstructured, to the complex and abstract.From a situated sociocultural view this conception is too simple.Rather, we as humans face and perceive a coherent whole, before we identify simple elements.In this regard, we face the world as a complex relational whole (as in Biggs' Level 4).From this point on, we develop our competences in coping with the challenges we face and, thereby, we begin to be able to identify relevant independent aspects and investigate the connection between aspects (Levels 2 and 3).On this basis, our understanding of the relational structure (Level 4) and our competence in coping with still more complex challenges might develop.And, finally, our competence in generalising and abstracting will be able to develop.This is why we recommend that the Structure of the Observed Learning Outcome model is regarded as a verbal model perceived as a spiral, depicting that progression happens on both complex and simple levels at the same time.
The SOLO model can be used to analyse different degrees of complexity in the students' understanding of the content and context of a learning material.Because this understanding will be observable in the students' actions in relation to the learning material's expression and activity, students' competence levels will be evaluated through an analysis of their actions and products related to these aspects.This corresponds to the secondary transformation unit in Selander's (2008) model, which designate this phase as summative assessment.
If we combine a verbal version of the SOLO taxonomy with the Model of textual evaluation of learning materials (Figure 1), it appears that we can analyse and compare the degree of complexity in the learning material's representation of content with the students' acquisition of the content and competence in coping with new challenges and thereby to what extent the learning material's potential learning potential has turned into actual learning.
In an analysis, for instance, the learning material's initiation of activities can be classified in different levels of competence, depending on if the activities demand understanding on unistructural, multistructural or relational levels.It rarely happens that a single activity demands an extended abstraction, but from a more comprehensive analysis of the activities initiated and content conveyed, it can be evaluated if the material does prepare the ground for students developing a more extended abstract understanding.
The SOLO taxonomy has mostly been used in relation to Mathematics, but it can also be used in Social Studies and The Arts to bring attention to the point that progression is not only related to the cognitive steps evident in Bloom's taxonomy (Bloom, 1956) but also encompass different degrees of structural complexity in relation to the content.
The methods used for measuring learning outcomes are often standardised tests, but as it appears from the above, such tests might be poor at assessing the more complex aspects of the students' understanding, and thus such assessments might miss the point that the learning material tries to make.There is a range of interesting work done in relation to developing more appropriate assessment frameworks from OECD's PISA (OECD, 2009) to IEA's ICCS (Schulz et al., 2011) and ICILS (Fraillon & Ainley, n.d.) studies and projects such as ATC21S (ATC21S, n.d.).But assessment can also be done as part of the actual practice by identifying signs of 1 www.johnbiggs.com.au/solo_graph.html,visited September 17, 2011.
learning, such as changed patterns of action and student products, for example, written assignments, slide shows, posters, documentary, portfolios, and learning logs.

Conclusion
In this paper, we have presented a holistic framework for the evaluation of learning materials.Very few actual evaluations will include all three steps, let alone all the aspects of each step.Thus the framework is neither a cookbook nor a demand for completeness or exhaustiveness, it is a heuristic to substantiate the design of evaluation and to discuss or evaluate accomplished evaluations.
Compared with LORI and Laeremiddeltjek introduced earlier, it is obvious that the holistic evaluation framework presented in this paper does not offer swift and simple to use evaluation typologies.It is not meant to be either swift or simple.The aim is to present the complex causality of using learning materials implying a whole range of challenges to be dealt with in future research on learning materials.
Teachers are not supposed to take all the aspects of the framework into account designing learning and teaching.It would not be advisable.Rather, it is useful to view the framework as a kind of map, depicting the field of challenges and thereby mirroring the need for educating teachers with a range of professional competences.Simultaneously, we have to extend the research in learning materials in all three aspects and temporal phases, so we can help qualifying the teachers' evaluation of potential learning potentials with a basis in evidence on actual learning and the complex causal relations between potential and actual learning.It is not enough to say what works.We have to investigate and demonstrate under which specific circumstances learning materials actually work in real settings.

Figure 1 .
Figure 1.Textual evaluation of learning materials

A.
Students' perspective: accessibility, differentiation, progression, and organisation of teaching B. Teacher's perspective: facilitation (while planning, implementing and evaluating teaching), and integrity (concerning immanent coherence within the material itself and transcendent coherence in relation to the norms and values of the teacher).

Figure 2 .
Figure 2. Teaching situation Other important factors co-determining the actual learning potentials are what could be termed the Zone of Proximal Development (see, Vygotksy, 1997/1931) of the participants (students and teachers) and the organisation of the situation (participants, space and time, history, context, and cultures).It is therefore clear that what is observed in a concrete situation is not solely due to the learning material.