MEASURING COURSE LEARNING OUTCOMES

Accreditation criteria of programs require effective learning outcomes, assessment with documented procedures, tools, results, and actions to close the assessment loop with broad faculty involvement. This article describes a methodology for providing quantitative measurement of a course’s learning outcomes. The methodology uses a linkage matrix that associates each course learning outcome to one or more course assessment tool. The approach adopted provides a numeric score between 0 and 1 for each learning outcome with respect to each assessment tool and a combined score be calculated for each learning outcome from the tools associated with that outcome. The proposed methodology also provides insights into the consistency of the various assessment tools used to measure the achievement of a particular course learning outcome. The methodology described here has been successfully adopted in obtaining accreditation and reaccreditation to all programs offered by College of Computer Engineering at University of Tehran.


Introduction
Academic program assessment and evaluation is becoming an important process in providing improved education to students through modified curriculum and instruction.Assessment has also become a tool of accountability in education by providing evidence on the effectiveness of teaching (Biney et al., 2008).An assessment plan determines how well students are benefiting from a learning experience offered by a program of study.We accept here that assessment: … is an ongoing process aimed at understanding and improving student learning.It involves making our expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analysing, and interpreting evidence to determine how well performance matches those expectations and standards; and using the resulting information to document, explain, and improve performance.
When it is embedded effectively within larger institutional systems, assessment can help us focus our collective attention, examine our assumptions, and create a shared academic culture dedicated to assuring and improving the quality of higher education.(Angelo, 1995, pp. 7-9) Over the last few years, the learning outcomes of a subject or course of study have become the focus of national and international conferences, and in peer-reviewed education literature, as a means of assessing the knowledge and skills obtained from learning opportunities.Learning outcomes have applications at three distinct levels: (i) the local level of the individual higher education institution for course units/modules, programs of study and qualifications; (ii) the national level for qualifications frameworks and quality assurance systems; and (iii) internationally (for wider recognition and transparency purposes (Adam, 2004).Learning outcomes focus on the measurable cognitive, behavioral and attitudinal development of students as they interact with a learning activity.They are what students are expected to demonstrate in terms of knowledge, skills, and attitudes upon completion of a learning experience (Adam, 2004;Ashiem, Gowan & Reichgelt, 2007).
Learning outcomes and outcomes-based approaches have implications for curriculum design, teaching, learning and assessment, as well as quality assurance.They are likely to form an important part of the twenty-first century approaches to higher education and the reconsideration of such vital questions as to what, who, how, where and when we teach and assess (Adam, 2004).
In terms of curriculum design and development, learning outcomes are at the forefront of educational change.They represent a change in emphasis from teaching to learning that characterise what is known as the adoption of a student-centered approach in contrast to traditional teacher-centered viewpoint.Student-centered learning produces a focus on the teaching-learningassessment relationship and the fundamental links between the design, delivery and measurement of learning (Adam, 2004).Other examples of direct measures include assignments that ask students to perform some kind of conceptual task (e.g., create a concept map) or portfolios compiled over a course of study.It is important to emphasise that these student work products need to be systematically reviewed for evidence of learning in order for them to be of most use.For example, rubrics can be developed and used by groups of faculty or educational researchers to analyse papers, theses, or portfolios in order to assess learning.Grades, of course, can also be a measure of learning although how the grades are determined and reported can sometimes undermine their usefulness.
However, course learning outcomes are crucial to the process.Among other things, they enable the program to demonstrate precisely how specific program learning outcomes are addressed in the curriculum.If course learning outcomes are then assessed continuously and the results are used to improve instruction that address them, then the degree to which the program meets its self-selected goals must inevitably improve.The contribution of this article is to describe a methodology that can be used to provide a quantitative measurement of the attainment of each course learning outcome.
A well-structured course should show clear alignment between the learning outcomes and the assessment criteria used on the course; in turn this leads to the design of appropriate assessment tasks, and to deliver the course in a way which enables students to reach the required outcomes.Biggs and Tang (2007) have developed the fundamental idea of constructive alignment, which is the process of synchronising teaching methods, learning activities, and assessment tasks with course's learning outcomes.Alignment of each of these three elements with learning outcomes is crucial for effective teaching (Biggs & Tang, 2007;Warren, 2005).Teaching activities should be driven by course learning outcomes and should support students in their learning activities and prepare them for assessment.This alignment between learning outcome, learning and teaching methods, assessment tasks and assessment criteria makes the whole process transparent to the students and to other interested stakeholders.
Course learning outcomes should specify the minimum acceptable (threshold level) standard for a student to be able to pass a course.This means that it is important to express learning outcomes in terms of the essential learning for a module or course, so a small number of learning outcomes of central importance should be developed rather than a large number of superficial outcomes.Learning outcomes should be written using action verbs so that students are able to demonstrate that they have learned or achieved the outcome (Reichgelt & Yaverbaum, 2002).
In 1956, Benjamin Bloom headed a group of educational psychologists who identified three domains of educational activities (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956).These are: cognitive, affective, and psychomotor.Knowledge, understanding and intellectual skills fall under the cognitive domain.The affective domain refers to attitudes and the psychomotor domain covers manual and physical skills.The group further divided the cognitive domain into six levels that describe the learning process from the simplest to the most complex.These levels are: knowledge, comprehension, application, analysis, synthesis and evaluation.The first two of these relate specifically to knowledge and understanding, while the remaining four involve intellectual skills.
While it might seem appropriate to concentrate on the lower two categories for lower level courses, it is recommended that students should be engaged in higher level activities on a smaller more focused scale from the outset (Scott, 2003).Bloom's levels of cognitive skills are shown in Table 1 along with a description of each skill.understanding something that has been communicated without necessarily relating it to anything else Application using a general concept to solve problems in a particular situation; using learned material in new and concrete situations Analysis breaking something down into its parts; may focus on identification of parts or analysis of relationships between parts, or recognition of organizational principles Synthesis creating something new by putting parts of different ideas together to make a whole.Evaluation judging the value of material or methods as they might be applied in a particular situation; judging with the use of definite criteria The following is a list of verbs for use when creating student learning outcome statements (Scott, 2003): • To measure knowledge (common terms, facts, principles, procedures), ask these kinds of questions: Define, Describe, Identify, Label, List, Match, Name, Outline, Reproduce, Select, State.Example: "List the steps involved in building an information system." • To measure comprehension (understanding of facts and principles, interpretation of material), ask these kinds of questions: Convert, Defend, Distinguish, Estimate, Explain, Extend, Generalise, Give examples, Infer, Predict, Summarize.Example: "Summarize the basic principles of software design." • To measure application (solving problems, applying concepts and principles to new situations), ask these kinds of questions: Demonstrate, Modify, Operate, Prepare, Produce, Relate, Show, Calculate, Solve, Use.Example: "Calculate the shortest path from node A to node B in the following graph." • To measure analysis (recognition of unstated assumptions or logical fallacies, ability to distinguish between facts and inferences), ask these kinds of questions: Diagram, Differentiate, Distinguish, Illustrate, Infer, Point out, Relate, Select, Separate, Subdivide.
Example: "Analyse the requirements of a school information system." • To measure synthesis (integrate learning from differ oriented software design with structured software design." The following are guidelines assembled from various sources as well as the author's experience in writing course learning outcomes: i. Action verbs from Bloom's Taxonomy with an emphasis on higher-order thinking skills should be used.ii.
To facilitate the assessing of outcomes, one verb per learning outcome should be used.iii.
There should be between 4-8 learning outcomes for each course, in fact the fewer the better.iv.
Course learning outcomes should describe what a student should be able to DO at the end of a course rather than what the instructor teaches.v.
Course learning outcomes should be written in language that students (and those outside the field) are able to understand.vi.
Course learning outcomes are typically not content-specific.

vii.
Ideally, each course or program should include learning outcomes from more than one domain (cognitive, psychomotor, and affective).viii.
Each course learning outcome should be measurable and can be assessed, preferably using more than one assessment tool.ix.
Weak verbs such as ""be aware," "appreciate," "identify," "read," and "recognize," are to be avoided in general.For example, recognising a phenomenon is weak compared to understanding that phenomena.
The following is an example of a poor learning outcome: "Develop skills to analyse a large volume of data."Obviously students should acquire these skills during the course and not develop them after finishing the course because that defeats the whole objectives of creating course learning outcomes.The following is an example of course learning outcomes written for a course of a program offered by the College of Computer Engineering at University of Tehran.After completing this course, students are able to: • explain the problem-solving process used to construct a computer program.
• construct an algorithm using pseudo code.
• select a suitable name, data type, and initial value for a variable or constant.
• create code using selection control statements.
• create code using repetition control statements.
• manipulate data using arrays, strings and records.
• use files for input and output processing.
• construct and test a user defined function.

Course Assessment Methods
Assessment is usually classified into summative and formative for the purpose of considering different objectives of course assessment methods (Scriven, 1967).Summative assessment refers to the assessment of the learning and summarizes the achievements of learners at a particular point in time.After a period of work, the learner sits for a test and then the teacher marks the test and assigns a score.The test aims to summarize learning up to that point.Midterm and end of course exams fall into this category.In an educational setting, summative assessments are typically used for evaluation purposes to assign students a course grade.Formative assessment is generally carried out throughout a course or project.Formative assessment, also referred to as "educative assessment," is used to aid learning.In an educational setting, formative assessment might be a teacher, peer, or the learner, providing feedback on a student's work, and would not necessarily be used for grading purposes.Formative assessments are diagnostic.Robert Stake, an educational researcher, provided the following interesting analogy: When the cook tastes the soup, that's formative; When the guests taste the soup, that's summative (Stake, 1998).
Summative and formative assessments are often referred to in a learning context as assessment of learning and assessment for learning respectively (Earl, 2003).Assessment of learning is generally summative in nature and intended to measure learning outcomes and report those outcomes to students, parents, and administrators.Assessment for learning is generally formative in nature and is used by teachers to consider approaches to teaching and next steps for individual learners and the class (Educational Assessment, 2006).
In general, high-quality assessments are considered those with a high level of reliability and validity.Reliability relates to the consistency of an assessment method.A reliable assessment is one which consistently achieves the same results with the same or similar group of students.
Reliability is affected by factors such as ambiguous questions, too many options within a question article, vague marking instructions and poorly trained markers.A valid assessment is one which measures what it is intended to measure.An exam is valid when it properly assesses the syllabus upon which the examination is based.A common form of formative assessment is diagnostic assessment.Diagnostic assessment measures a student's current knowledge and skills for the purpose of identifying a suitable program of learning.Self-assessment is a form of diagnostic assessment which involves students assessing themselves (Educational Assessment, 2006).
Assessment methods should be designed such that they are able to measure the full range of outcomes associated with a particular course.For example, for information technology courses, they should include (Biney et al., 2008): • assessment methods that measure the ability of students in demonstrating subject knowledge,

Measuring Course Learning Outcomes
The next step is to provide a data-backed quantitative measurement of how well students are achieving each course's learning outcomes.The process used to get these measurements should be easy to implement and not time consuming to instructors.It is not acceptable to determine a student's achievement of course learning outcomes on the basis of the final grade obtained in the course alone.These grades represent the aggregation of too many factors, causing the student's ability in any particular topic area within the course to be lost in the aggregation (Lord, 1980).A more detailed level of analysis is needed.One approach that we suggest in this article is to create a linkage matrix that associates each course learning outcome with one or more assessment tool.This matrix is shown in Figure 1.This approach of measuring the achievement of a particular course learning outcome can provide two very important observations.First, it indicates which course learning outcome the students have failed to achieve.Secondly, this measurement technique also indicate if a learning outcome has been achieved consistently by all assessment tools assigned to it which further provide an insight into the consistency of the various assessment tool in measuring a particular course learning outcome.
The suggested methodology for providing a quantitative measurement of course learning outcomes may impose a heavy burden on the instructors in terms of designing the questions for each assessment tool and data collection and analysis if implemented manually.However, we feel that an automated system might alleviate much of the burden on instructors' time and efforts.The assessment process described above applies successfully in obtaining accreditation and reaccreditation to all programs offered by College of Computer Engineering at University of Tehran.

Closing The Assessment Loop
Assessment of course learning outcomes is a continuous process and cyclical in nature.The assessment cycle as Implemented by College of Computer Engineering at University of Tehran is depicted in Figure 2. The final steps in the assessment cycle are referred to as "closing the assessment loop" (Bailie et al., 2010;Ramesh, & Mattiuzzi, 2001).Closing the loop refers to the process of using the results obtained from various assessment activities to improve the program and to document such improvements with the intent to positively impact future student learning.In Figure 2, the shaded rectangles represent the steps required to close the assessment loop.After course instructors conduct assessment activities, collect assessment data and analyse it, the course coordinator provides the quantitative measurement regarding the achievement of each course learning outcome as described in the table shown in Figure 1.This information will be reviewed by the College Assessment Committee.In addition, the College Assessment Committee also takes into consideration the results of student's surveys which are conducted for each course towards the end of the semester.These surveys provide valuable information with regard to the suitability of textbook and references; academic background and prerequisite courses; course delivery modes; and lab activities among others.

Conclusion
In this article, a description of the full assessment cycle for course learning outcomes and what actions are needed to close the assessment loop have been presented.The main contribution of the article is in proposing a methodology for providing a quantitative measurement of the level to which each course learning outcome has been achieved.In addition, this methodology provides valuable information regarding how each learning outcome is being assessed by the different assessment tools giving insights into the consistency of the various tools in measuring a particular course learning outcome.The approaches successfully apply in obtaining accreditation to all degree programs at the Colleges.
designing and conducting experiments, • gathering data, analysing and interpreting data, • demonstrating and applying knowledge, • defining a technical problem, • planning a project, • conducting a review of the literature, • generating ideas and creativity, • perform preliminary and detailed design, • functioning effectively and as a member of a team, • solving technical problems, • defining computing requirements to solve a particular problem, • formulating and analysing engineering/technical/ computing problems, • solving engineering/technical/computing problems, • understanding and demonstration of ethical issues and professional responsibilities, • understanding and demonstration of social responsibilities, • written and oral communications, • making effective use of library and on-line resources, and • awareness of contemporary issues in industry.Current assessment tools used to assess courses learning outcomes in other countries include: Mid Term Exam, Final Written Exam, Short Article, Team Project, Oral Discussions, Lab work, Presentations, Seminars, Reports, Tests and Quizzes, Student Portfolio, and Individualised Products areas or solve problems by creative thinking, ask these kinds of questions: Categorise, Combine, Compile, Devise, Design, Explain, Generate, Organize, Plan, Rearrange, Reconstruct, Revise, Tell.Example: "Design a data flow diagram for the following software requirements specification:" To measure evaluation (judging and assessing), ask these kinds of questions: Appraise, Compare, Conclude, Contrast, Criticise, Describe, Discriminate, Explain, Justify, Interpret, Support.Example: "Contrast object."

Figure 1 .
Figure 1.Linking course outcomes to assessment toolsThe measurement criteria is as follows: for each course outcome I and assessment tool J that address an outcome I, the maximum grade allocated for outcome I and the average scored grade obtained by students for outcome I is entered.An outcome I is achieved if the ratio total_scored/total_max.Where, total_scored is the sum of average grades obtained by students from all tools J for outcome I, and total_max is the sum of the maximum grade of all tools used to measure outcome I.

Figure 2 .
Figure 2. Closing the assessment loop College, 2007;als and program learning outcomes have been articulated and the curriculum has been designed, measurable course learning outcomes must be developed for each course in the curriculum.Each course learning outcome must map to at least one program learning outcome to ensure that all courses in the program of study are addressing the overall program learning outcomes.This process also verifies whether each program learning outcome is addressed in at least one course (LaneyCollege, 2007; University of Connecticut, 2007).Designing courses using learning outcomes leads to a more student-centred approach: it emphasises a shift from what staff members teach towards what the student is able to do on successful completion of the course.Specifically, learning outcomes can help staff focus on exactly what they want students to achieve in terms of both knowledge and skills; inform students of what is expected of them and help concentrate their efforts; and provide a useful guide to stakeholders about the general knowledge and understanding that a graduate possesses