Traffic Light Report provides a new technique for Assurance of Learning

The Traffic Light Report (TLR) project is an educational intervention designed for pharmacy undergraduates. This paper reports on analysis of TLR data specifically focusing on its potential as an innovative tool which combines Miller’s pyramid, technology and student voice to examine a curriculum for Assurance of Learning (AoL). In 2014, educators mapped each summative assessment to the relevant National Competency Standards Framework for Pharmacists in Australia (NCS) alongside levels of expected performance on Miller’s pyramid of clinical competence (Knows, Knows how, Shows how, Does). Simultaneously, students were invited to self-reflect using the same performance levels. The Miller’s scale enabled a comparison between students’ and their educators’ understanding of the performance level demanded by assessments. Analysis highlighted a disconnect between students’ and their educators’ interpretations of the same assessed curriculum. The TLR facilitates quality enhancement by providing educators and their students with a logical meeting point for discussing foundation, scaffolding and integration of assessment across a course for AoL. This has portability to other professional disciplines.


Introduction
Higher education providers in Australia are under increasing pressure to integrate professional standards in the development and assessment of Course Learning Outcomes (Australian Government Department of Education and Training, 2015). The National Competency Standards Framework for Pharmacists in Australia (NCS) is the relevant professional standard for pharmacists. The NCS are grouped into eight Domains: (1) Professional and ethical practice; (2) Communication, collaboration and self-management; (3) Leadership and management; (4) Review and supply prescribed medicines; (5) Prepare pharmaceutical products; (6) Deliver primary and preventative health care; (7) Promote and contribute to optimal use of medicines; and, (8) Critical analysis, research and education. The Australian Pharmacy Council who accredits the university courses highlight the importance of the NCS to pharmacy education by explaining that: Since the entry-level competencies are to be met at entry to professional practice, they can serve as a source of guidance to the teaching and learning expected. …The goal of initial pharmacy education is to produce graduates with the requisite knowledge, skills and attributes for entry to an intern training program. (Australian Pharmacy Council, 2012, p. 15) Pharmacist competence assures patient safety. Familiarity with the NCS should begin with first year undergraduates so they can engage with their profession's competence continuum (Coombes, Bates, Duggan, & Galbraith, 2011) and continually monitor and improve their performance. Professional competency standards such as the NCS can be combined with a scale to communicate a logical continuum towards competence for educators and their students. Miller's (1990) Pyramid of Clinical Competence which classifies competence along a continuum of "Knows," "Knows how," "Shows how" and Does" is used by pharmacists and other health professionals worldwide for this purpose (General Pharmaceutical Council, 2011;Kelley & Demb, 2006;Wass, Van der Vleuten, Shatzer, & Jones, 2001). Other frameworks exist, however Miller's pyramid is a simple conceptual model that clearly communicates the early stages of the competence continuum. Each level on Miller's pyramid represents increasing capability and increased integration of knowledge, skills and attributes. Prior to performing at a "Does" level in the practice setting with a client, an individual must have developed essential underpinning knowledge, skills and attributes. For optimal learning and assured practitioner competence, these stages are best scaffolded. In other words, graduates who have had their learning scaffolded and are provided a "solid foundation" having progressed through the "Knows" level (observation and basic knowledge recall), "Knows how" (understands and applies knowledge) and "Shows how" (integrates knowledge, skills, attitudes to perform in simulated settings) (Miller, 1990;Van Der Vleuten & Schuwirth, 2005;Wass, et al., 2001) are given the greatest chance of success.
Familiarity with the NCS among pharmacy students in Australia (Nash, Chalmers, Stupans, & Brown, 2015a) and at the university of interest was discovered to be poor (Nash, Chalmers, Stupans, Brown, 2016a). To address poor student familiarity with the NCS and develop students' self-reflection skills, the Traffic Light Report (TLR) project was conceptualised. The TLR elements (NCS and Miller's pyramid) were combined to ensure the usability of the NCS for educators and their students in their context. Influenced by the Triple Jump Test (Mészáros et al., 2009), the mileMarker (Szilagyi, 2008), topographical curriculum maps (Plaza, Draugalis, Slack, Skrepnek, & Sauer, 2007) and the work of Janke, Traynor and Sorensen (2011), the TLR applies a course wide approach to assure student learning.
Essential to the TLR design is curriculum mapping. Curriculum mapping is commonly used in higher education to provide a course level summary of a curriculum, enable curriculum review and provide evidence for accreditation and quality enhancement requirements. Martone and Sireci (2009) suggested that the mapping process is more useful than the results as it helps educators see how assessments can connect to learning activities. Oftentimes, the mapping process and discussion it prompts amongst educators (and students) becomes the most valuable component (Kelley, McAuley, Wallace, & Frank, 2008). Thus, a teaching team must design and map their course together (Lawson, 2014(Lawson, , 2015a and assess their curriculum first for student learning and second in terms of its effectiveness to measure achievement (Boud & Falchikov, 2005). To support this, a range of mapping tools currently exist (Harden, 2001;Oliver, Ferns, Whelan, & Lilly, 2010) including, of particular relevance to the work reported here, Lawson's Curriculum Design Workbench (Lawson, 2014(Lawson, , 2015a. The Curriculum Design Workbench is built on a philosophy of Assurance of Learning (AoL) (AACSB, 2013;Lawson, 2015b) which emphasises a teaching team approach to course design to ensure student knowledge and skills are scaffolded and, importantly, educators and their students are aware of where this scaffolding takes place (Lawson, 2015a;Lawson et al., 2015). In its most simplistic form, AoL is defined as "the process by which learning outcomes are measured against specific course goals" (Hall & Kro, 2006). This research used the data from the TLR to highlight that educators at the university of interest and their students describe a very different understanding of expected performance levels, as described through the Miller's pyramid. This paper seeks to explore possible reasons for this difference. The findings from the TLR have implications for educators, students and regulators and have portability to other professions.

Method
This project was conducted at one Australian university over two semesters (Semesters 1 and 2, 2014), where the full time pharmacy course is comprised of four units per semester. Each unit had 2-8 summative assessment tasks. Experiential placements took place in Third and Fourth year, students in First year visited their allocated workplace briefly whilst Second year students did not have scheduled placements. The data presented on Miller's pyramid in this paper is one part of the TLR research project, with other applications and outcomes having been described and reported elsewhere, including its ability to improve student knowledge of the NCS (Nash et al., 2016a) and provide opportunities to develop self-assessment skills (Nash, Stupans, Chalmers, & Brown, 2016b).
A pragmatist methodological frame (Creswell, 2013), educational design (Van Den Akker, Gravemeijer, McKenney, & Nieveen, 2006) and action research philosophies (Creswell, 2013) informed the authors' approach. Triangulation of data (Creswell, 2013) from student selfreflection, curriculum mapping data and feedback surveys made it possible to interpret the curriculum from the viewpoint of students and their educators. Data collection for each perspective is now described to aid the understanding of the analyses carried out.

Educators' perspective (assessed curriculum)
For each assessment item in each unit of the course, educators entered details of the task into a customised Access database, the relevant NCS for each assessment, and the level of proficiency on Miller's pyramid that students were expected to achieve ( Figure 1). A map was then produced of all the relevant NCS and level of Miller's proficiency for each assessment in the course. The authors interpreted this curriculum map as the educators' perspective of the assessed curriculum.

Students' perspective (learnt curriculum)
At the conclusion of scheduled semester examinations, enrolled students were invited to selfreflect on their performance against the eight NCS on Miller's pyramid via a 10 minute online survey ( Figure 2). The self-reflection was open for two weeks. Prior to this project, students had limited exposure to the NCS which, as previously noted, consists of eight Domains, each containing multiple competency standards. Specifically students were asked to: consider your current performance level (Knows, Knows how, Shows how, Does) for each competency standard (NCS). The authors interpreted the results from this survey as the students' "learnt" curriculum.

Perspectives combined and contrasted
The educators' perspective was represented by the curriculum mapping entered into the TLR. The students' self-reflection provided the students' perspective ( Figure 1). This paper is focussed on the analyses of data associated with Miller's pyramid. For the purpose of this analysis, data were extracted from the TLR and summarised by year and semester of enrolment, for example, Fourth year, Semester 1. The "instances" used in the analysis were defined as: (i) an individual student's self-reflection on a given standard within the NCS, for example, "Practice to accepted standards" (Standard 1.1), at a "Does" level represents one instance; and, (ii) their educators' mapping against a given standard for an assessment task.
As each NCS may have been mapped to multiple summative assessments, the database was programmed to select the instance where the student performed at the highest level in assessment on Miller's pyramid, for example, "Does" was higher than "Knows." The educators' mapping of the performance level of the summative assessment was directly compared with the Miller's level selected by the students in their self-reflection to determine agreement that is, whether the two perspectives matched. Non-parametric data analyses were conducted using SPSS Version 22. These data were also extracted into the statistical program R to provide heat-maps that described the curriculum at an individual Domain level of the NCS. A "hot spot" was where the ratio, indicating the perceived degree of focus of the curriculum on a particular NCS Domain at a given Miller's level, was greater than 0.6. The following provides an example formula used to generate the ratios reported in the heat-maps:

Results
Students who participated in the self-reflection represented 42% (n=69) of the invited students (N=163) in Semester 1 and 26% (n=52) of invited students in Semester 2 (N=198). Table 1 provides a breakdown of student participation by year level and by semester of participation.
Students are referred to in this paper, by Year of course, Gender and an identifying number, for example, (Year 2, Male19), and (Year 4, Female22).

N=121
Semester 1  These ST (student) participants with multiple responses to individual assessment items, along with respective responses from educators (ED), generated 1878 instances. The key findings overall, related to the levels on Miller's pyramid, are that: • In 397 (21.14%) of instances (n=1878), students self-reported their performance at a level higher than their educators.
• In 860 (45.79%) of instances (n=1878), students self-reported their performance at a level lower than their educators.
Despite a relatively low level of agreement, the scaffolded pattern outlined by students and educators in Figure 3 represents a consistent shift in focus from Knows/Knows how (acquisition of knowledge) to Shows how/Does (application of knowledge) for assessment across the 4 years of the course (p=0.84). Figure 3 also shows general agreement amongst students and educators with the exception of Fourth year students who reported that 39.2% of their assessments examined them at Knows/Knows how level, whereas their educators' reported only 11.0% of assessments required their students to perform at this level; this difference was statistically significant (p<0.001). which had greatest emphasis in the curriculum, that is, which were assessed multiple times, are depicted by "hot-spots" (the areas of darkest shading) on each heat-map. From the educators' 12 and the students' 19 "hot-spots," there were five areas of the curriculum where students' and educators' views aligned. These were: • Domain 2: Communication, collaboration, self-management (Shows how: Year 2). This shared "hot-spot" highlights that Second year students and their educators believed assessment required students to immediately perform at "Shows how" level. The data in Figure 3 suggested that the whole curriculum provides a clear knowledge foundation (particularly in Years 1 and 2), however deeper exploration of the findings (Figures 4a and 4b) reveals that some Domains appear to have stronger foundations than others. The number of occasions where Domains 3 and 8 were mapped to summative assessment tasks are shown in Table 2 and can supplement the heat-map findings. For example, according to the educators' mapping on Domain 8 the assessments covered all three corresponding standards (8. 1, 8.2, 8.3) on multiple occasions across all four Miller's levels. As Table 2 shows, Domain 8 provides greater opportunities for student development. In contrast, according to educators' mapping of summative assessment on Domain 3, students in Fourth year are expected to perform at a "Does" level with much less foundation development.
The TLR data presented in Table 3 also provides an indication of the number of NCS the students were required to combine to successfully complete assessment tasks. Examined with the assessment description it is possible to determine if students were required to integrate the NCS (across and within Domains) at appropriate performance levels. For example, the Fourth year assessment task, a "Health Promotion Services Project Report," required students to integrate Domain 2 (Communication, collaboration and self-management) with Domain 6 (Deliver primary preventative healthcare). Then, within Domain 6 itself students also combined Standards 6. By interpreting the findings from Table 2 and 3 and Figures 3, 4a, and 4b, it is possible to appreciate (i) how the NCS were distributed throughout assessments across the curriculum, (ii) the number of times each Domain had been assessed, (iii) if each competency standard within each Domain was scaffolded throughout assessment from "Knows" through to "Does," and, (iv) where exactly in the curriculum this took place.
Feedback survey participants represented 45% (n=73), and 16% (n=31) of those invited in Semester 1 and 2, respectively. Student feedback comments are included in the Discussion section to supplement the narrative. Interestingly, in Semester 1, Third year students had the poorest feedback survey response rate (30%) whilst Second year students represented the highest response rate (56%). In Semester 2, Second year students represented the lowest of all year groups (10%).

Discussion
Analysis of the TLR highlights two distinct viewpoints of an identical curriculum -those of a cohort of undergraduate pharmacy students and those of their educators. Whilst neither viewpoint should be considered more valid than the other, points of difference and similarity provide valuable insights and a meeting point for discussion of assessment and AoL. The value of these insights was supported by student comments from the feedback survey: • Provided useful feedback on particular aspects of the course. (Year 2, Female19) • It allowed me to see which areas (of curriculum) are relevant. (Year 3, Male21) 48 The authors' claim the TLR is novel for two reasons -firstly, because it explicitly presented Miller's pyramid to students for self-reflection purposes, which has not been done previously. Whilst Miller's pyramid has been used for some time by educators to inform assessment plans (Wass, et al., 2001), in progress testing (Szilagyi, 2008) and curriculum review, and to discuss the difficulty (Epstein & Hundert, 2002) and scaffolding (Kleinert et al., 2015) of assessments, the literature does not appear to describe the explicit use of Miller's pyramid with students for selfreflection.
The second point of difference of the TLR was that it directly contrasted the student voice with educator curriculum mapping for AoL. The TLR supersedes traditional educator-led curriculum mapping focussed on the "intended" or "assessed" curriculum, which is not always the same as the "enacted" or "learnt" curriculum from the students' perspective (Barrie, 2004;Edstrom, 2008;Harden, 2001;Lew, Alwis, & Schmidt, 2010;Porter & Smithson, 2001). Using this novel approach to AoL, the authors anticipated that the students' self-reflection on the learnt curriculum would align with their educators' mapping of the assessed curriculum, as found by other authors (Porter, Floden, & Fuhrman, 1998, Plaza et al., 2007. However, the TLR did not report similar concordance, as evidenced by poor agreement between educators' expected performance levels contrasted with their students' reflections, especially when the curriculum was examined for each NCS Domain by year of enrolment. Kelley and Demb (2006) also reported disconnect in student and educators assessment expectations. The TLR may provide professional disciplines with a new solution to AoL. It provides a logical meeting point for students and their educators to ensure assessed curriculum translates to learnt curriculum. The meeting point is at the intersection of student voice with their educators' curriculum mapping captured, reported and contrasted by the TLR. AoL principles impress that a curriculum which is designed with a solid foundation, scaffolded development and opportunities for students to integrate their knowledge, skills and attributes will enhance student success (AACSB International, 2013, Hall & Kro, 2006, Lawson, 2015b. The TLR findings allow exploration of the design of this pharmacy curriculum against these three elements.

Solid foundation
Student success is enhanced if students are provided opportunities to learn foundational knowledge (Knows/Knows how) prior to being required to apply that knowledge in practice (Shows how/Does). Findings from this study, as seen in Figures 3, 4a, 4b and Table 2, illustrate that the curriculum provided students with opportunities to develop a strong foundation in some Domains but less so in others. This is particularly evident when contrasting Domains 3 and 8. Some of the unexpected patterns described, for example, Fourth year students' disagreement with their educators may be due to conceptual misunderstandings of Miller's pyramid, and the nature of knowledge, itself. This is illustrated by the following student comment:

• I self-assessed at a higher level. I have worked in community pharmacy since first year and I feel/ have been told that I work very competently. (Year 3, Female21)
Contrasting with this student's statement, by definition, 'Does" requires that the student or practitioner be situated in the practice setting integrating competencies to consistently perform a task. To perform that task safely, practitioners require a strong foundation in the underpinning knowledge of the skills (Miller, 1990;Wass, et al., 2001). A student may believe they are capable to "do" a task such as dispensing medicines but may be doing so without the foundational knowledge, namely, pharmacology and therapeutics, required to do so safely.
With a specific focus on providing students with a solid foundation through the TLR assists educators to ascertain the AoL of an existent curriculum design.

Scaffolded development
Outcomes-based educators argue for scaffolded development and multiple opportunities to assess competencies (Conway, Medina, Letassy, & Britton, 2011;Epstein & Hundert, 2002;Van Der Vleuten & Schuwirth, 2005;Wass, et al., 2001). Figure 3 portrays a curriculum which generally builds students' capacity to apply their knowledge in practice by Fourth year. This scaffolded course development, described by both students and educators, is reassuring from a whole of curriculum point of view, however the differences between the students' and their educators' expectations at Domain level evidenced in the heat-maps warrants further exploration. The heatmaps may help us to understand how a scaffolded curriculum (represented by summative assessments) might translate to AoL. Through the eyes of students and educators the heat-maps provide evidence of Domains which appear well scaffolded and those which are not. The TLR data were able to demonstrate scaffolding or lack of scaffolding of summative assessments.
The reported disconnect could be a result of differences in interpretation between educators and students or could highlight a need for curriculum restructure. The findings described may indicate some areas of poorly laid foundations, inconsistent scaffolding, or inadequate opportunities for students to integrate knowledge and skills: • There were many competency standards that I felt I wasn't confident in despite the actual assessment showing that I was (Year 2, Male19).
The students' heat-map on Domain 2 (communication, collaboration and self-management) implies students perceived assessment commencing at a "Shows how" level without assessment of underpinning knowledge development. If educators do not design curriculum to ensure scaffolded development from "Knows" (for example, the theory of motivational interviewing techniques utilised in smoking cessation) through to "Does" (for example, motivational interviewing with a client in practice) this could have negative results for those reliant on communicating with health graduates about their health and medicines.
We now focus again on Domains 3 and 8 shown in Figure 4a, 4b and Table 2. Within Domain 8, it is apparent that a graduate has had their knowledge scaffolded and examined repeatedly to enable competent performance at a "Does" level for all standards, however they may be less prepared for performance in Domain 3. From this it is clear that Domain 3 (Leadership and management) may need to be introduced earlier in the course and scaffolded to ensure our graduates can be leaders of the profession. This example evidences the depth and breadth of information the TLR can provide for course review and AoL.
To uphold AoL, students must be provided with clear instructions of assessment expectations and criteria in order to succeed (Biggs & Tang, 2007). Students' difficulty in appreciating the relevance of the NCS may be a consequence of implicit use of the NCS by educators without explicit communication of their relevance to students. The TLR project highlights that there is a significant role for course level rubrics (Lawson, et al., 2015) or (if adopted) TLR rubrics to ensure consistent scaffolding across course assessment. Student understanding of educator expectations may be improved through students' active participation in the rubric development phase. Rubrics may have addressed some of the difficulties experienced by students, as well as the evidenced disconnect, by ensuring students did not have to assume anything. One student offered that: I assume that we are being taught to meet the competency standards therefore by passing assessments I am demonstrating competency (Year 2, Female20).

Integrate knowledge, skills and attributes
Pharmacists integrate multiple NCS to competently perform in practice at a "Does" level. To ensure workplace readiness, a curriculum must also be designed so that students are assessed for their ability to integrate their knowledge, skills and attributes. Harden and Stamper's (1999) spiral design curriculum integrates relevant knowledge, skills and attributes and given the findings described here may be useful for future course redesign efforts. The benefits of integrated curriculum design and authentic assessment have also been discussed by Van Der Vleuten and Schuwirth (2005). The student feedback comments implied a science versus practice split indicative of potentially poor integration in the curriculum, particularly amongst First and Second year students. The relationship of the NCS to the curriculum was difficult for some students to appreciate and potentially led to students questioning the relevance of the NCS to their curriculum (and vice versa). This was evidenced in one student's comment: The research is great, for me it has highlighted how truly meaningless the competency standards are at the moment. They let me know I'm "tracking" towards competency, still meaningless… I can only assume 3rd year placements will give gravity to the CS and finally allow me to judge myself and the School's curriculum against them. …Without the experience of real life practice (placements or intern) to benchmark the info against, the report doesn't have any meaning to me.
(Year 2, Male31) Highlighted by the data in Table 3, authentic assessment choice can play an important role in providing students with opportunities to integrate their required competencies, be assessed formally and therefore succeed in practice. The Health Promotion Service Project Report which involves students on placement designing a health promotion service, delivering it to the community and evaluating it, required students to address multiple Domains and standards within each Domain. Whilst we cannot assume all the NCS listed were integrated by students, the data give us a positive indication that this was the educator's intention. TLR data interpreted alongside assessment task descriptions confirm whether students were required to integrate their knowledge, skills and attributes to succeed in each summative assessment. This provides students about to enter the practice setting with confidence in their ability to perform.
Overall, these findings have implications for students, educators and regulators; students may find it difficult to succeed if their expectations of curriculum are different to those teaching and assessing them and vice-versa. AoL could be sub-optimal especially as the learnt curriculum appears significantly different to the assessed curriculum at a Domain level. Course accreditation may be approved based on assessed/mapped curriculum which has been evidenced here to be different to the learnt curriculum. Overall, this may produce a mismatch in graduate knowledge and skills for their workplace requirements.

Limitations and influencing factors
To make use of the TLR findings for AoL it is necessary to be cognisant of three influencing factors affecting students' and educators' perceptions of their curriculum -student and educator familiarity with the NCS, familiarity with Miller's pyramid and self-reflection difficulties.
Poor NCS familiarity across the Australian pharmacy profession, including amongst its educators and students (Nash, et al., 2015a;Nash, Chalmers, Stupans, & Brown, 2015b), may have negatively impacted on their ability to apply them in the education setting. As the NCS are actionorientated descriptions, the style of assessment and how well this matched the NCS and corresponding Miller's description may have also influenced interpretations. Given the matching "hot-spots" emerged in instances where students were required to apply their knowledge (Shows how/Does), we can deduce it was easier for students and educators to appreciate the relevance of NCS when the assessment required application of knowledge, rather than acquisition of that knowledge. Finally, some students reported confusion with the self-reflection task. This difficulty can be attributed to self-reflection being a new skill students had not previously encountered in the curriculum, compounded by the fact that students were required to recall multiple assessments that they had completed over the course of a semester and then identify relevant NCS for each. The following comments reflect this difficulty: •

Recommendations and future research
Resourced appropriately, the TLR may scaffold student development of self-reflection skills. The TLR could also become a very powerful form of feedback to engage and empower students in their learning. It provides a central meeting point for educators and students regarding performance expectations for assessment, as well as offering insights on curriculum design for AoL. To ensure sustained engagement, future TLR versions should be embedded within courses and course teaching team cultures. In fact to maximise educational outcomes students and educators could workshop together to discuss the NCS, assessment expectations and co-design rubrics. If the NCS adequately reflect current pharmacy practice requirements, then the learnt curriculum as evidenced by students' reflections must match the assessed curriculum. Student reflections therefore demand greater consideration in the curriculum review process. Internationally, professional courses could consider a similar approach to curriculum review, harnessing student interpretations to compare directly with their educators' expectations for AoL. A project to determine the TLR's impact on AoL amongst educators and students who meet to discuss their TLR is warranted, as are projects to explore the use of the TLR with other professional disciplines.

Conclusion
The TLR provides a new technique for evidencing the AoL of a curriculum. In the pharmacy course described, the TLR was able to highlight potential issues with foundation, scaffolding and integration of professional standards within a curriculum. The TLR reinforces the assessed and learnt curriculum may not always be consistent and provides educators and students with a logical meeting point for AoL. Using self-reflection, students can provide rich data reflective of a learnt curriculum useful for examining a course for AoL. This has potential portability to other disciplines with well-defined professional standards.