USING CONCEPTUAL MAPPING AS A TOOL IN THE PROCESS OF ENGINEERING EDUCATION PROGRAM DESIGN

An evaluation of application of the conceptual mapping technique (Inglis, 2003) to the team-based design of fourteen courses in an Engineering undergraduate program is described. The evaluation employed a method of scoring each conceptual map against criteria tied to the objectives of the design process. The evaluation method was found to be capable of revealing differences in application of the conceptual mapping technique. The evaluation indicated that the course teams concerned appeared to have focused closely on the adequacy of intended learning outcomes but less closely on matching the student assessment adequately with the intended learning outcomes. The evaluation also indicated that to obtain the full benefit of use of the technique, more training of instructional design staff was required than had been provided.


Introduction
Course renewal in Engineering at RMIT University has received major attention over the past five years with significant funding and other resources devoted to the task.(In this paper, the meaning of the term 'course' refers to a modular element of academic study and is synonymous with the words 'subject' or 'unit', sometimes used in the Australian context).Five years ago the Faculty of Engineering established a team of instructional designers to provide individual support to academic teaching staff.Within each budget year, a strategic analysis of priorities targeted a set of courses for renewal.Academic and teaching staff with responsibilities for coordination of a selected course were granted time release to undertake the renewal project and an instructional designer was assigned to consult with the academic/ teaching staff in a team relationship.
Course renewal was implemented as a two-phase process.The first phase involved the learning design process.The term 'design', in this context, does not refer to layout, typography and use of illustrations, but rather to the establishment of learning outcomes and associated performance criteria, setting assessment measures for each outcome, selecting and defining necessary resources and specifying the nature of the learning activities associated with each outcome.The learning design phase also involved estimation of the time required for completion of each activity and for completion of the set of activities as a whole, so as to ensure that student workloads would not be excessive.This first phase of course renewal led to the development of what is termed a 'conceptual map' (Inglis, 2003;Inglis and Armstrong, 1993).A conceptual map is a document that specifies the design of a course.It identifies the key design elements and the relationships amongst them.A concise explanation of the process involved in development of a conceptual map is given below while a detailed explanation appears in Inglis (2003).The second phase of course renewal, involved development of a comprehensive learning package, including a learning guide developed in accordance with predetermined standards.

Objectives of the study
The purpose of this investigation was to establish the extent to which implementation of the conceptual mapping process had resulted in an improvement in the quality of information generated by the course development team during the design phase.In other words, the investigation attempted to gauge the extent to which the conceptual mapping technique achieved the first of its intended purposes discussed above.More specifically, the study was undertaken in order to ascertain whether or not introduction of the conceptual mapping process had resulted in a discernible shift in the specificity with which Engineering Faculty teaching staff specified key design attributes.The study made no attempt to evaluate the adequacy of the coverage of the courses.This is a matter that falls within the province of the academic/teaching staff responsible for the course, and in turn the academic/teaching team responsible for the program as a whole.

The process of courseware design
The literature of instructional design (ID) contains many prescriptions for documenting courseware designs.Some design methods are concerned with the organisation of subject matter.For example, Reigeluth, Merrill and Spiller (1980) described the Elaboration Theory for selecting, sequencing and synthesizing subject matter.At a more basic level, Novak and Gowan (1984) described a method for representing knowledge structures and Lambiotte, et al. (1989) described and compared a wide range of knowledge mapping approaches and summarised the research regarding the actual and potential uses of knowledge maps in education.Jonassen (1993) also examined the use of mapping techniques for representing and conveying knowledge in a teaching context.Other methods have been described for representing a wider range of learning types than just the acquisition of knowledge.It is rare however to see any of these methods being used in practice at the tertiary level.One reason for this situation is that most tertiary teachers are not familiar with the ID literature.Another important reason is that few tertiary teachers have the opportunity to work with experienced instructional designers.Academic teaching staff who have had no training in education can sometimes think of the teaching process in terms of transmitting knowledge.This tendency has led to teaching that is characterised by transmission of subject matter and reflects a 'transmission theory of learning'.Although, given that what is being referred to is a form of teaching practice, it would more appropriately be described as reflecting a 'transmission model of teaching'.Moving beyond a transmissive approach requires a teacher to think more deeply about a broader rage of learning outcomes and to conceptualise learning as involving more than simply the acquisition of knowledge.
Distance education providers commonly utilise a course team for the development of courseware.In a course team context, the member, or members of the team responsible for the curriculum are often referred to as subject matter experts (SMEs) (Keppell, 2000).SME's obviously include the appropriate academic/teaching staff members, but may also include external experts brought in to contribute to the development of a course.A course team would normally comprise at minimum a SME and an instructional designer, however it may include two or more SMEs, an editor and media production staff.The course team model enables a range of expertise to be brought to bear on the design and production of courseware.Input from an instructional designer can often result in a more holistic learning design and development process and avoid a sole focus on content which sometimes characterises the approach of SME's working in isolation on course preparation.

Use of conceptual maps in courseware design
The development of courseware, and particularly the development of interactive multimedia courseware, involves a substantial investment of human resources.This investment needs to be based on a sound learning design methodology if it is to return learning outcomes satisfying a predetermined performance specification.Part or even the whole of this investment may be wasted if the basic assumptions on which the courseware is structured are flawed.The main purpose of developing a conceptual map is therefore to provide a separation between the design and development phases of courseware preparation and to document specific outcomes of the design phase.In this context 'design' refers to decisions taken in relation to the 'what' and 'how' associated with student learning and the resources that are to be provided to facilitate this learning.'Development' refers to preparation of the materials themselves: the writing of textual materials by the subject matter expert(s), and the assembly of other resources.Given that analytic thinking and systematic problem solving are so fundamental to the practice of engineering, and as a foundation for design and project execution, it has not been an overly difficult task to work with engineering academics to translate these principles to the tasks of conceptual mapping and educational design.
The conceptual map as a courseware development process, delivering a specification of outcomes and an educational design methodology, is analogous to the formal requirements analysis, specification of outcomes and design proposal phases of an engineering design project.The conceptual mapping method has been developed over some years across a range of disciplines including careers education (Inglis and Armstrong, 1993), art and design (Inglis and Bradbeer, 1996), and nursing.However, use of the method in the discipline of engineering represents its most extensive application to date.A conceptual map is meant to serve three distinct purposes: (1) to specify and document the design of a course to a level of detail that is sufficient to support the subsequent development phase; (2) to provide a shared context for discussion between the subject matter expert and the instructional designer in relation to course design; and (3) to provide a starting point for consideration in a subsequent revision cycle.A conceptual map provides a vehicle for steering courseware development and as such, it describes the functional aspects of the educational product rather than the characteristics and performance capability of a physical product, as may well be the practice in many engineering design approaches.
One of the attributes of a well-designed conceptual map is its parsimony.The conceptual map is meant to serve as a means of communication between the instructional designer and the SME(s).It is a working document and is not meant to reach the student.It is not meant to include learning material or resources.It is therefore important that it not be filled with so much unnecessary detail that the overall structure of the course is lost.The information provided in the conceptual map should be succinct.On the other hand, it is important that a conceptual map provides sufficient information to support the subsequent development stage.If insufficient information is provided at the design stage then time will be lost during the development phase while the missing details are sought.Part of the skill of producing a well-designed conceptual map therefore lies in striking an appropriate balance between specificity and succinctness.When developing a conceptual map, teaching staff were advised to be as parsimonious as possible.
The structure of the conceptual map is intended to encourage teaching staff to move in particular directions while engaging in the design or redesign of a course.It is intended to have staff focus on the most important elements of a course's design: the intended learning outcomes, the learning activities through engagement in which the student will be expected to attain the intended learning outcomes, the learning resources needed to support the activities, and the types of assessment that will enable students' attainment of the intended learning outcomes to be confirmed.However, it is also intended to focus the teaching staff member's attention on the interrelationships between corresponding elements, especially between corresponding learning outcomes and learning activities, between corresponding learning activities and learning resources, and between corresponding learning outcomes and assessment methods.In this respect the method is consistent with Biggs' (1999) constructive alignment model, although it predates Biggs' model and goes beyond it insofar as it includes other pedagogical elements and is focussed on courseware design rather than curriculum design.Finally, consideration of the time dimension, which normally takes place towards the end of the design phase, is intended to focus attention on the magnitude of the workload implied by the learning activities that have been chosen so that this can be kept within the limit set down for the particular course.For an example, see Inglis (2003).

Method of investigation
The method used in the study was to score each of a number of conceptual maps developed for a range of specific courses against a set of criteria tied to the objectives of the design process.The conceptual mapping process focuses on certain aspects of the design of courses (i.e. the identification of particular design elements and the specification of the interrelationships amongst them).The study therefore used measures that corresponded to the type of design output the conceptual mapping process seeks to generate.

Design of the scoring instrument
The method used to score conceptual maps did not employ the conventional type of Likert rating scale.Nor did it require a judgement to be made of the quality of the structure and content of conceptual maps.Rather, it sought the identification in each case of the number of individual elements that met or exceeded a stipulated criterion.It was considered likely that this type of judgement would be made more reliably than allocating a score on a rating scale.The measures used in each case were ratios (estimated as percentages).Responses were recorded using an interval scale with intervals: 0-25, 25-50, 51-75 and 76-100, recoded 1, 2, 3, and 4, respectively.These intervals were chosen as being sufficiently narrow as to permit differentiation between conceptual maps that are well documented and those that are not so well documented, yet not so narrow as to make selection of the most appropriate response category a difficult choice.
The criteria for each of the measures were chosen so as to reflect the goals of the instructional design staff in working with SMEs.The conceptual maps evaluated in the study describe the learning outcomes, the learning activities, the learning resources, the time that it is expected students will require to complete the activities and the types of assessments that will be used to measure students' attainment of the learning outcomes.However, the adequacy of a conceptual map may be further measured in terms of the quality of the information provided and its relationship to other items in the conceptual map.

Definitions of the measures
The reliability of the scoring method depended on gaining a high degree of agreement on interpretation of the response scales.Each of the measures was therefore defined explicitly.The definitions are shown in Table 1.A more detailed explanation of each of the measures follows.The label for each measure identifies the relevant component of the conceptual map and the quality being assessed.Learning outcomes for which the proposed assessment appears to provide an adequate measure

Learning outcomes -Appropriateness
Often students are expected to demonstrate capabilities that have little or no relevance to their future professional roles.For example, students may be required to provide a definition of a particular concept (which they have already been provided and therefore can memorise) whereas the capability they would require in a professional context would be to recognise examples of the concept.When students are expected to demonstrate capabilities that would never be needed in a professional role, this is usually because the designated capabilities are either more easily assessed or more easily taught in a formal educational setting.This measure gives the proportion of those learning outcomes that are listed that describe capabilities that students may need to employ in their future professional roles.

Learning outcomes -Level
Learning outcomes can be categorised into different types that vary in terms of the level of learning implied (Gagné, 1985).Rote memorisation is considered to involve low level learning whereas problem solving is considered to involve high level learning with concept learning falling somewhere between.Rote memorisation is generally considered an inappropriate type of learning outcome in tertiary education except in relation to certain areas where the ability to recall specific information is important.The term 'rote memorisation' refers to the situation where the focus is in the words themselves rather than the meaning behind the words (Mayer, 2002).Yet it is recognised that learning outcomes are commonly written in such a way as to expect rote memorisation.More desirable are outcomes that expect students to engage in formation of concepts, learning of principles or problem solving.Whether a particular learning outcome expects rote memorisation depends on the student's previous learning.Merrill (1971) pointed out that the nature of learning is such that learners are constantly striving to push their learning down to a lower level.Consequently, what is concept learning for one learner may be memorisation for another.The cognitive level at which learning outcomes are pitched therefore depends on taking into account the learner's existing capabilities.In a class situation this implies that the design of learning materials allow for the student whose capabilities in the domain in question are least well developed.

Learning outcomes -Generic capabilities
As indicated, Engineers Australia promotes a set of generic attributes that graduates in engineering should be capable of demonstrating.This measure compares the number of generic capabilities that it is judged could be reasonably reflected in the learning outcomes for the course with the number of generic capabilities that are actually reflected in the learning outcomes for the course.Using this measure requires an assessment to be made first of the generic attributes that ought to be reflected in the learning outcomes for the course.Once a list of the course-specific generic attributes had been prepared, the number of generic attributes actually reflected in the intended learning outcomes was compared to the number of generic attributes that could be appropriately reflected in the intended learning outcomes for the course.

Activities -Match to learning outcomes
The purpose of learning activities is to provide learners with the opportunity to acquire the capabilities described in the specified learning outcomes.For learning activities to serve their intended purpose they need to be closely matched to the corresponding learning outcomes.If they are not closely matched to the learning outcomes students may devote a substantial amount of time to a course but still not acquire the capabilities that are expected.This measure compares the number of activities judged to be satisfactorily matched to their corresponding learning outcomes with the total number of activities that have been set down for completion.

Time -Estimate plausibility
Academic teaching staff are sometimes more able to make a much more accurate estimate of how much time a student is likely to take on an individual activity than how much time a student will take to complete the full set of learning activities for a course.The method of preparing the conceptual map therefore requires SMEs to record their estimate of the time that they expect students will typically require to complete each individual activity.In practice, once all activities have been specified, the time requirements are summed and if the sum varies by more than 10% from the total learning time nominated for the course, the staff member is encouraged to consider modifying the requirements of the course so as to adjust the workload.This measure compares the number of activities for which the time estimate seems plausible with the total number of activities.A low ratio may either indicate that no estimate has been given for the time required for many activities or that the estimates that have been given are not plausible.

Resources -Sufficiency
When students are learning from a learning package or online, their ability to complete the activities specified for a course depends on whether all the resources required for the activities have been provided.In some cases the resources that need to be provided are directions for locating materials from elsewhere rather than the materials themselves.To ensure that all the required resources are provided, the SME needs to work systematically through each activity, identifying what is needed, and gathering those materials and transforming them into suitable forms.Each of the resources required should be identified in the conceptual map.This measure compares the learning resources that are identified in the course design process with the learning resources that, in the judgment of the researcher, would be needed for completion of the activities.

Resources -Specificity
The purpose of identifying the resources required for completion of the activities is to enable the resources to be assembled in the subsequent development stage.Each item need only be specified in sufficient detail that it is clear to all of the members of the course team as to what is meant.For example, it would be quite sufficient to identify an extract from a book by the author, year of publication and page number(s) if the members of the team knew exactly which book was being identified from the author and year.Later, when a reference to the book is included in the learning package the full bibliographic details would be cited.This measure indicates the proportion of the resources that are listed that are specified in sufficient detail to allow the learning package to be compiled without further research.

Assessment -Coverage
Within the Engineering Faculty an attempt has been made to increase the comprehensiveness of assessment in courses.The general principle that is being advocated is that if attainment of a learning outcome is important, then attainment of the learning outcome should be assessed.The conceptual mapping technique contributes to the achievement of this goal by having course teams identify the method by which it is proposed the attainment of each learning outcome will be assessed (e.g. by a problem in the examination, by a practical report, by one of more object test items).This measure indicates the proportion of the intended learning outcomes for which the proposed assessment is judged to be adequate.

Scoring method
The conceptual maps were scored by both researchers independently.The researchers then came together to compare their scores.Where a discrepancy between the scores on an individual item was found, the basis on which the scores had been assigned were discussed and consensus was reached on an appropriate score for that item.By this means an agreed score sheet was produced for each conceptual map.
Such documentation as previously existed for each course was then scored.The documentation that existed prior to establishment of the conceptual mapping process was much substantial than that provided in the conceptual map.In most cases the only documentation that was previously available was the Course Guide provided to students when they enrolled in a course.
What was being looked for in this study was evidence of improvement in the quality of the information generated at the design stage.An inherent assumption of the conceptual mapping process is that the first attempt at generating a conceptual map will not produce the most ideal result.Each iteration of the design-development-production-delivery cycle provides an opportunity for improving the design of a course further.

Selection of conceptual maps for scoring
member of the team of instructional designers was asked to select two conceptual maps from amongst those in which they had worked.It was indicated that the conceptual maps should be chosen as being representative of those on which each instructional designer had worked and which they considered were developed to the point that they were ready to be used in the subsequent development stage.The first author of this paper, as a member of the instructional design team and originator of the conceptual mapping technique also selected two conceptual maps.All conceptual maps were then scored.Some weeks later, after allowing for further development of remaining projects, the members of the instructional design team were asked to select two further conceptual maps.These were also scored.

Results
In a study of this type one would normally wish to undertake a before-and-after comparison of the impact of the change in practice on the quality of what is being produced.However, in this case not only did the introduction of the conceptual mapping technique mark the first attempt to focus attention specifically on the design of courses as an activity separate from the development process but it also coincided with initial development of learning packages for distance education.There was no previous equivalent practice with which the new approach could validly be compared.
The aim of this study was therefore not primarily to establish whether adoption of development of conceptual maps had led to improvement in documentation of the design decisions, but to establish the effectiveness with which the conceptual mapping technique was being applied.A secondary aim was to determine the suitability of the scoring method as a means of gauging the adherence to the design principles embedded in the conceptual mapping technique.
The scores given to each of the conceptual maps on each of the ten measures is shown in Table 2.Because of the relatively small number of cases and the small number of scoring categories no attempt was made to analyse the data for variability.It was considered that this would convey a focussed more on subject matter than on learning outcomes.While the evidence provided by this study does not allow one to say conclusively that the use of the conceptual mapping process was responsible for a shift in focus, it would seem that it is likely that a shift in focus did in fact occur.By contrast, there were two areas where the teams' efforts appeared to be deficient.The first was in encompassing generic capabilities and the second was in providing coverage of the intended learning outcomes in the assessment.
The fact that generic capabilities were not adequately reflected in the overall learning outcomes was a disappointing finding, given the importance being placed on the development of generic capabilities both the Faculty and by the engineering profession.However, this is explained by the relative lack of experience amongst teaching staff at integrating generic capabilities into the engineering curriculum and their uncertainty as to how best to go about doing this.The advice that instructional designers were asked to give SMEs was to ensure that generic capabilities were reflected in the set of intended learning outcomes wherever this was appropriate.In some other universities, generic capabilities are separately assessed (Atlay and Harris, 2000).From discussion of importance of generic capabilities in the literature it appears likely that the extent to which courses facilitate students' attainment of generic capabilities is an area of the curriculum that will come under increasing scrutiny in the future (Bowden, et al., nd;Nunan, 1999).
The issue of the coverage of the assessment is a matter on which there is some debate.It is not uncommon for teaching staff at university level to assess only a subset of the intended learning outcomes that have been identified for a course.Whether or not students' attainment of the intended learning outcomes for a course should be comprehensively assessed is therefore a matter on which one is likely to find a variety of opinions.However, there has been some attempt within the Faculty of Engineering at RMIT to assess more comprehensively and uniformly than has been the case in the past.The very low scores on this measure indicate that this thrust has largely been ineffective in the case of the courses included in this study.
Improvements in the specificity of the design information may arise in either of two ways.The form in which the conceptual map documents information calls for a greater amount of information to be recorded and for this information to be recorded in a more useful form.However, improvements may also result from the greater amount of thought being given to their design.Many staff have commented that, as a result of the experience of preparing a conceptual map, they have thought much more deeply about the design of their courses.
Use of a scoring instrument closely matched to the aims of the design process together with the independent scoring of each conceptual map followed by the process of reaching consensus on the final score provided an objective measure of the quality of the information contained in the conceptual maps.What the scoring process revealed was not only a variation in the quality of the information generated by the process but also a considerable variation in the application of the process itself.
The ability of course development teams to use the technique is a function of both their understanding of the technique and their willingness to persist with the process of documentation of the design until that task has been sufficiently advanced.Anecdotal evidence from the instructional designers who were involved with the course development projects that provided the conceptual maps for this study suggests that while course teams were keen to move on from design to development, the variation in the quality of the conceptual maps can best be explained by the course teams' lack of familiarity with the method.This finding highlighted the need for the instructional designers to receive more intensive training in use of method.While it had been the standard practice for the members of instructional team to meet weekly to discuss the progress of projects, these weekly discussions did not get down to the level of detail that was capable of revealing the extent of variation in application of the technique.This variation is ascribed to the relative familiarity of different instructional designers with the attributes of the concepts on which the technique is based, such as learning outcomes, resources, and assessment.

Conclusion
The purpose of the conceptual mapping technique is to enable course development teams to document the designs of courses to be taught with the use of learning packages prior to the development of the learning materials.However, the value of the technique depends on how effectively course teams are able to apply it in actual course development situations.This investigation of the quality of conceptual maps produced by a variety of course teams across a range of engineering subjects suggests that while teams had mastered some aspects of the design process there were other aspects with which they were having difficulty.
It is expected that as academic teaching staff become more familiar with the conceptual mapping process the quality of the conceptual maps they produce will improve.The maps will become more tightly structured and the information provided in the maps will be more specific.However, one would expect that these improvements would be less obvious than the improvements in moving from earlier forms of documentation to the conceptual map.
It is anticipated that the conceptual mapping process itself will be further refined over time.Changes may be made to the nature and quantity of information recorded in conceptual maps as well as the procedures used to produce them.The results of this and future studies will help to ensure that any changes that are made contribute to the overall improvement of the process rather than simply reflect the personal preferences of those using the process.

Table 1 .
Measures used in scoring conceptual maps