TECHNOLOGY SUPPORTED FACILITATION AND ASSESSMENT OF SMALL GROUP COLLABOR ATIVE INQUIRY LEARNING IN LARGE FIRST-YEAR CLASSES

Collaborative learning activities offer the potential to support mutual knowledge construction and shared understanding amongst students. Introducing collaborative tasks into large first-year undergraduate science classes to create learning environments that foster student engagement and enhance communication skills is appealing. However, implementing group work in classes of over 1000 students presents challenges for instructors in terms of task design, group management and assessment. Interdisciplinary scenario-inquiry tasks have been designed for small group work in a large science cohort, informed by literature and current pedagogical practices relating to the integration of collaborative and active learning strategies. Facilitation and assessment of these tasks was perceived as too complex and time consuming for a single instructor to complete manually, so a web-based task management technology was developed. Evaluation of the technology supported collaborative group activities, including peer assessment, was conducted through questionnaires, student interviews and analysis of the artefacts of the learning process. The capabilities and limitations of the technology, and the insights into group learning gained through its use are presented. In general, students felt supported through the task. Evidence of resource interdependence was found between students in functional groups. Journal of Learning Design


Introduction
When is large too large?First-year science students and instructors often find themselves involved in classes with very large enrolments (>1000) that serve multiple programs of study.Traditional learning and assessment environments comprising lectures, tutorials and the laboratory are most commonly encountered in first-year science units.Higher impact pedagogies (Kuh, 2008) such as collaborative small group work can appear daunting to implement effectively in large classes due to the scale.
Collaborative learning is a widely established pedagogical approach.It is based on social interdependence theory (Johnson & Johnson, 2009;Slavin, 1990) whereby students mutually construct knowledge and share their understanding.Reports on the implementation of collaborative-learning pedagogies (Prince & Felder, 2006;Smith, 2010;Smith, Sheppard, Johnson, & Johnson, 2005) endorse their benefits.In fact, there has been a call to make collaborative learning the "core model of pedagogy" in the university of the twenty first century (Tapscott & Williams, 2010, p. 26).The process of collaboration contributes to students' gains in transferable scientific argumentation skills (Sampson & Clark, 2009).
The benefits and learning outcomes of collaborative group learning have potential to provide students with opportunity to achieve several of the threshold (or minimum) level of achievements that can be expected of an Australian bachelor level graduate in science according to the Australian Science Standards Statements (Jones, Yates, & Kelder, 2011).In particular, independent collaborative group work aligns with the following statements: • TLO4 (Communication): Students will be effective communicators of science by communicating scientific results, information, or arguments, to a range of audiences, for a range of purposes and using a variety of modes (4.1).• TLO5 (Personal and professional responsibility): Students will be accountable for their own learning and scientific work by being independent and self-directed learners (5.1), and working effectively, responsibly and safely in an individual or team context (5.2).
While collaborative learning is well established as an effective pedagogical strategy, student perceptions of group work often influence their engagement in this type of assessment task (Gillespie, Rosamond, & Thomas, 2006;Phipps, Phipps, Kask, & Higgins, 2009).One of the most significant design challenges in implementing collaborative group work is to achieve positive interdependence between students within a group.The term collaborative learning is typically used to describe group work where the activities and learning may be divided between group members but the group as a whole is accountable for the outcomes.This description encompasses also cooperative learning (Smith et al., 2005).
Despite a significant body of literature that addresses strategies for individual and collective evaluation of collaborative-task outputs, the assessment of student learning outcomes from collaborative group work is not well formulated.Janssen, Kirschner, Erkens, Kirschner, and Paas (2010) titled their article Making the Black Box of Collaborative Learning Transparent, yet there has been little focus on evidencing or measuring individual outcomes from group processes in the literature.Assessment of collaborative learning outcomes generally relies substantially on peer assessment to gain insight into group activities, allowing the instructor to adjust individual marks.Strategies adopted to measure learning outcomes in terms of communication and teamwork are prone to being influenced by the social function of the group rather than the learning that an individual student experiences.
Technology-mediated assessment of learning has been shown to support the engagement of students in large classes in active learning environments (Kelly, Baxter, & Anderson, 2010).It also offers opportunities to facilitate group formation and management in collaborative learning tasks.
The process of learning design to incorporate small group work in large enrolment classes began in this study based on a review of key cooperative and collaborative learning research literature to identify common themes (Johnson & Johnson, 2009;Johnson, Johnson, Ortiz & Stanne, 1991;Johnson, Johnson & Smith, 1998;Kagan, 1992;Slavin, 1990;Smith, 2010;Smith et al., 2005).Five core attributes of constructive cooperative and collaborative learning environments were identified from the literature as important elements to embed in the task design and are listed in Table 1.

Core attributes of cooperative and collaborative environments identified as central to designing and implementing a group work activity
Attribute Feature

Positive interdependence
Each student values and perceives the individual contribution (resources or intellectual) of team members as essential to the task outcomes.

Individual accountability
Each student perceives that they are required to contribute to the collaborative process.This may be related to a negative outcome if they do not engage in the task.

Social interaction
Students establish group relationships based on respect for individuals and their experience.

Group processing
A collectively shared understanding is developed in regard to how and when members will communicate and contribute with each other.

Communication
Students establish effective communication processes.
As part of this study, a technology tool was developed to support instructors in managing very large (>1300 enrolled) first-year science cohorts through smallgroup collaborative tasks.The design of the tool attempted to introduce positive interdependency by requiring each of the four students in a group to complete an individual research activity to generate information required by the whole group to create their collective product.These individual activities were constructed so that the collective outcome would be of lower quality if one of the four sets of information were missing.Once the collective group product was submitted, the technology directed students through two peer assessment activities.
The effectiveness of these technology-scaffolded collaborative tasks including peer assessment was explored in this study through a mixed-methods evaluation approach including (a) pre/post questionnaires, (b) artefacts of the group processes, and (c) student focus groups.Triangulation was achieved through statistical analysis of quantitative data and analysis of qualitative data to identify emerging themes.

Learning Design
In this design-based study, collaborative active-learning experiences were formulated to foster the attributes listed in Table 1, by introducing small group work in large first-year general chemistry classes.Placing students in a group does not automatically guarantee that they will work productively together.Group membership and communication can be the root of success or failure of group work.Multiple studies exploring the factors influencing collaborative group work recommend that the ideal group composition: • is heterogeneous groups of four students (Johnson et al., 1998;Kagan, 1992); • is of mixed academic ability (Felder & Brent, 2001;Kriflik & Mullin, 2007); • has gender dispersed to minimise the number of same gender groups;, and, • has a distribution of international students to both address hurdles related to English being a second language and to improve student integration into a new environment (Kavanagh & Crosthwaite, 2007).
The success of cooperative learning is underpinned by developing positive interdependence between students, categorised as outcome, means and boundary interdependence (Johnson & Johnson, 2009), defined as follows.Outcome interdependence involves common goals and rewards which, when structured into a task, result in gains in student learning outcomes (Buchs, Butera & Mugny, 2011;Johnson et al., 1991).Means interdependence, which includes resource, role and task interdependence can be generated through activities where each student has responsibility for one set of information in moving towards a shared goal or product.Sharing complementary information has been found to have significant influence on student outcomes, indicating that this situation should be fostered in instructional design (Buchs et al., 2004;Johnson et al., 1991;Kirschner, Paas, & Kirschner, 2009a, 2009b).Boundary interdependence is based in extrinsic factors including physical arrangement of students within learning the environment and inter-group relationships (Johnson & Johnson, 2009).Of these categories, means interdependence represents the most challenging to evidence in terms of outcomes of task design and effective cooperative learning behaviours, because students need to demonstrate both conceptual understanding and the development of skills from working together.
The instructional design also aimed to encourage interdisciplinary thinking so tasks were set in contemporary science contexts and dubbed interdisciplinary scenario-inquiry tasks (IS-ITs) (Gahan et al., 2011;Lawrie et al., 2011).Students worked in groups of four on one of the 27 different IS-ITs.Student investment in both the process and outcomes of the task was enhanced by enabling them to choose their preferred scenario and by providing the option for them to choose their group membership.Interdependence between students was encouraged by requiring that each student begin by working independently on gathering information (designated as individual quests (IQs)) with the understanding that this information would be required by the whole group to successfully write their collective report.The IQs were made available to all group members, leading to positive resource interdependence (Buchs et al., 2004).This individual component was completed prior to embarking on the collaborative process that required students to establish a consensus approach to structuring and writing their collective report.The timeline and progression of students through the IS-ITs during the course of a semester is shown in Figure 1.The students collaborated outside class contact time and iCAS provided a group discussion forum to enable social interaction and communication between group members as well as file upload links for the sharing of draft documents and IQs.
The principal support provided to students during the task was the provision of supporting documentation and resources through the parallel course learning management system (Blackboard).These included the task instructions including anticipated learning outcomes, task expectations, an online module to provide scaffolding for working in teams, a link to the iCAS website and peer-assessment rubrics.Weekly announcements (through the course Blackboard site) reminded students of the expectations in terms of at what stage their group should be and who to contact if issues arose.There was no direct involvement of the instructor in the task other than to facilitate a "drop-in" consultation session to remediate any group issues that arose during the collaborative phase of the task.A summary of Frequently Asked Questions arising from this session and from emails on Blackboard was posted to the Blackboard site.
The final product of this task was a collectively written group report that represented a unique response to an over-arching question (metaquestion), framed by a contemporary scenario drawing upon an interdisciplinary context (Lawrie et al., 2011).Each student within a group was awarded the same mark for the group's report, a form of reward or outcome interdependence because each student was likely to feel accountable for their group's final report mark (Buchs, et al., 2011).
Assessment of an individual student's learning as a result of a group process can be obtained by testing their total knowledge before and then after the collaborative activity (Weinberger, Stegmann & Fischer, 2007).This measurement is, however, a blunt instrument that does not take account of group interactions, and is an example of reward independence (Buchs et al., 2011).Measurement of which student within a group facilitated shifts in understanding or how students encouraged one another to learn is also not easily accessed, with peer assessment offering one possibility.Our own early attempts to adopt peer assessment into our inquiry-based collaborative tasks in large first-year classes established that anxiety due to dependency on peers for summative assessment was significant (Lawrie, Matthews & Gahan, 2010).Despite these challenges, peer review is still attractive because it gives students feedback that instructors cannot provide, and enables them to reflect on their learning from collaborative tasks (Price, O'Donovan & Rust, 2007).iCAS facilitated two forms of peer assessment: (i) confidential evaluation of the contributions of their own team members; and (ii) peer review of other group's products within the same scenario to encourage individual reflection (shared).Each group's report received up to 16 sets of scores that were averaged and awarded as a single mark after moderation by the instructor.

Methodology
The participants in this study were first-year chemistry students (N=1359) enrolled in a general chemistry course in a large public Australian university.The students were enrolled in 40 separate programs of study including Science, Health Science, Pharmacy and Engineering.Of the group, 54% were female, 81% were in the age range between 17-19 years and while 81% identified their nationality as Australian, there were 38 other nationalities represented.This study involved 346 groups of four students working on 27 different scenarios.
Guided by the LEPO (Learning Processes and Learning Outcomes) evaluation framework (Phillips, McNaught & Kennedy, 2012), the learning environment, process and outcomes were evaluated through a mixed-methods approach including both quantitative data and qualitative perception data collected via an online questionnaire (Likert scales) including open-response questions and also by audio recordings of student focus groups.Digital artefacts of the task processes that were available through iCAS included the timing of student processes, information related to group composition, number and nature of interactions in the collaborative domains, submitted files and peer assessment and review marks.The institutional ethics committee for research involving human subjects approved ethical clearance for this study (Application number 2009001480).
The questionnaire was delivered after the completion of the task but before the unit grades had been released.Questionnaire data were filtered for consent and completion resulting in 840 valid responses to the open questions.Two separate focus groups (each with N=8) were conducted at the end of semester comprising participants who volunteered in response to an invitation circulated to the whole cohort.Audio transcripts of the focus group were analysed deductively according to themes that arose from the open responses in the questionnaire.

Scaffolding small groups using technology
In this study, students were provided with the opportunity to self-select into groups of four.For those who did not do this, iCAS enabled the instructor to consider the demographics of group membership (program of study, nationality, gender, age, academic ability) to optimally assign these students into groups according to the recommended principles.Multiple students (99, 7.3%) either requested assignment to a group by the instructor or did not voluntarily engage in the group sign-on process.These students were either assigned to pre-existing groups of three or fewer to form a group of four or were placed together to form new groups in scenarios that had not yet been fully subscribed.Thus, across the cohort, there were 18 groups (N=346, 5.2%) that were formed entirely by assignment of students by the instructor because they had not self-selected into a scenario.
As the semester progressed, there was natural attrition as students withdrew their enrolment for a range of reasons and new students enrolled in the course.iCAS provided an easy interface through which the instructor managed group membership.In addition, students within each group were able to note when a student disappeared from their group and could adjust their activities accordingly when a new member was assigned by the instructor.
It was anticipated that, because this task was completed outside course contact time, students would opt to work asynchronously through the online discussion forum provided for each group within iCAS.It was evident very early on during the task that this forum was underutilised.During evaluation at the end of semester, when asked how the group had communicated during the task, students indicated that the majority preferred to work face-to-face (Figure 2).Apart from the initial introductions to each other, students mainly used the iCAS discussion forum to arrange meeting times for their groups.
Figure 2. Modes of communication that students reported their group had adopted to facilitate collaboration It has been noted previously that, for online collaborative learning environments, students often require a face-to-face meeting before they can commence their online interactions (Goodyear, Jones & Thompson, 2014).The outcomes of this study support the proposition that students prefer face-to-face interactions in collaborative learning activities.
Many assumptions have been made regarding the information and communication literacies possessed by the current generation of students (Jones & Hosein, 2010;Kennedy, Judd, Churchward, Gray & Krause, 2008).Throughout this task, insight was gained into the software skills that this diverse cohort of students possessed including basic skills such as processing files between different formats.It was found that of 1151 individual IQ files submitted to iCAS (no specified format was required and not all students submitted files), 666 were docx (57.86%), 296 were doc (25.72%), 131 were pdf (11.38%) and 58 document files (5.04%) had other extensions (e.g., .rft.odt.htm.ppt).As part of the task requirements, each group of students was required to submit a pdf file of their final written report.Of the 346 final group report files that were submitted, 229 were pdf (66.18%), 66 were docx (19.08%), and 51 were doc (14.74%) revealing a substantial lack of compliance (n=117, 33.82%).These data indicate that students were either not paying attention to the instructions or that the there was a misplaced assumption that first-year students were able to generate pdf files.As a result of the need to provide files for students to complete their peer assessment, the administrative load on the instructor increased because it was necessary to convert submitted documents to pdf format and to troubleshoot other access issues based on versions of software.

Support required for collaborative group work
In the online questionnaire, students were asked to comment on whether they felt adequately supported as they progressed through the task and to identify anything that might have helped them work more effectively in their group.Their responses were coded initially according to whether or not they felt adequately supported in the task (positive/negative) and then categorised according to whether the support was related to their peers, instructional resources or neither.Counts of citations by individual students were made for each of these categories.A small number of students cited both forms of support in a single comment.The majority of students who perceived that they were adequately supported throughout the task cited the support that they had received from their peer group members and their contribution to the collaborative processes during the task (Table 2).This indicates that positive group experiences stemmed from the relationships and communication within the group, and represents social interdependence (Smith et al., 2005).
Of the students who perceived that they had received inadequate support, approximately half cited the instructional resources as being insufficient, indicating a greater reliance on the instructor for guidance in the processes compared with students who felt supported.The resources that they felt would better support them included having the opportunity to be able to consult with a tutor during the task for guidance.It became apparent to researchers that an external factor existed during this study.This was an e-conference assessment task that had been completed in a parallel biomedical science course by a large number of enrolled students in which a tutor had been readily accessible for consultation.Several students requested that an exemplar report should be made available to provide them with a model for a high standard document.Exemplars were not provided in this study because each of the 27 scenarios resulted in a different document structure and composition.It was also believed that provision of a model report might impact on the capacity of students to choose to take either a positive or a negative stance in formulating a response to their metaquestion.Finally, the proportion of students who felt inadequately supported in the task and who cited the task instructions as inadequate was 52% compared to the 24% of students who had felt adequately supported and cited the instructions provided.

Positive interdependency: social aspects, resources and rewards
A critical and desirable component of the instructional design was to promote effective collaboration between students through the development of positive interdependency between group members.The task was designed to foster resource and transactional interdependence (Kirschner et al., 2009b) by requiring students to negotiate with their group members to take responsibility for one of four individual IQs that they completed prior to the collaborative phase of the task.Each student was able to contribute information that could be considered by the whole group as necessary to formulating a collective report in order to introduce accountability in the collaborative process.Group discussion was scaffolded to begin construction of the report using the dot-point stimulus comments or questions provided within the IQs.
Substantial evidence of positive social interdependence arose through student feedback in response to whether they had felt adequately supported (Tables 2 and  3), indicating that their positive experience had been strongly biased by positive interactions with team members.The role of social interdependence as a critical component for effective group work is further emphasised by feedback from students who felt inadequately supported and those who cited lack of participation or low quality contributions from their group members.The absence of interdependence was clearly evident in their comments.
Resource interdependence was introduced through the requirement for students to submit their IQ.This was a key factor cited both in the questionnaire and the student focus groups completed at the end of semester.Several students recommended that the submission of IQs be a formal part of the assessment to improve their group processes (Table 3).In 2011, in a subsequent iteration of the task, the submission of IQ files attracted 5% of task marks and this increased the number of submitted files in iCAS, facilitating group processing.Negative interdependence is defined as competition between students (Smith et al., 2005) and involves oppositional interactions.In very large cohorts, such as that in this study, there will be students who do not engage constructively due to negative perceptions of their colleagues' achievements and abilities.One student who had felt inadequately supported by their peers gave their reason as: "I am trying to get a HD (high distinction) for the course and my fellow students for the course had a GPA (grade point average) of 4 on average.One of them had even failed (the course) last semester." Peer assessment was implemented in the instructional design to introduce outcome interdependency.Using iCAS to facilitate the peer assessment process gave the instructor insights into group function and allowed moderation of the final mark where required.The average marks that students awarded in peer assessment were analysed to determine the mean scores and standard deviation.The outcomes (Table 4) revealed that the mean score for the internal peer assessment (student's own group members) in the course was higher than the mean score for the external peer review (other group's reports).Of interest was the substantially lower standard deviation for the mean score awarded to peer review of group reports.Students were provided with structured assessment criteria for both parts of the peer assessment to guide their award of marks.It would be difficult for a student using the criteria to assign a mark of less than 50% to a report unless it was of very low quality and missing multiple components.It appeared that, while the majority of students made a valid attempt to critically appraise the reports submitted by other students, there were some students (evident in comments in the questionnaire and focus groups) who adopted shallow or strategic approaches to the peer assessment process by awarding full marks to their team members and penalising reports that appeared better than their own (Table 3).The evidence was inconclusive with regard to how successful the peer assessment was in terms of introducing outcome interdependency, particularly in terms of the authenticity of students' engagement in the assessment process.A dimension that has not been explored is whether first-year students are less likely to be highly critical of their peers after their group has functioned reasonably well and they have experienced social interdependence.
One of the capabilities of iCAS to manage peer assessment in large enrolment courses was that the instructor was able to monitor and reduce the impact of a spurious mark awarded by a single student.During the peer review process, each group's report received up to 16 separate marks; hence, one low mark would not have a substantial impact on the final task mark.As part of moderation, the instructor reviewed the marks that students awarded to their group members and adjusted low marks that appeared to have been awarded due to group dysfunction.It had been made explicit in the task instructions and course profile that insufficient justification and discrimination for the award of 100% in peer assessment would attract a penalty.Application of this hurdle in very large enrolment classes would not be possible without the technology displaying the data in an accessible and editable format.

Conclusion
The overall objective of the learning and instructional design was to introduce effective small group collaborative learning in very large enrolment classes where students worked in a self-directed environment.The utilisation of the bespoke technology tool, iCAS, demonstrated the feasibility of providing a mechanism of support for an instructor and provided appropriate scaffolding for processes that typically involve a high administrative load when completed manually (group membership and peer assessment/review).Affordances of the task design promoted through the use of the technology tool included: (a) instructor insights into student engagement, (b) management of student generated products (information files and group reports) and (c) moderation.Students also felt adequately supported, particularly in accessing information retrieved by individual members of their group and this added the dimension of individual accountability to the task.Limitations of the use of technology became evident based on students' own technology skills and subsequent remediation of student access to submitted files.
A second aim of the instructional design was to foster positive interdependency between students within a collaborative group by introducing five core elements (Table 1).Evidence of resource interdependent collaborative processes emerged in student reflections and perceptions, particularly for highly functional groups.It appeared that the attempt to introduce resource interdependence was more effective than outcome interdependency, which was based in peer assessment and review.Interdependency was not evident in dysfunctional groups but this cannot be attributed as the cause of ineffective collaboration.Indeed absence of interdependency is a key feature of ineffective group function (Smith et al., 2005), derives from a lack of social interdependence and is evident in poor communication between students and a lack of shared understanding of their processes.
Through evaluation, evidence was found that this learning design fostered teamwork, communication skills and negotiation between group members and, as such, represented a viable assessment option to provides evidence of student achievement related to Science Threshold Learning Outcome statements 4 and 5 (Jones et al., 2011).Additional outcomes of this learning design study were a template for delivering small group collaborative tasks in very large STEM classes, multiple instructional resources, an open source technology tool and recommendations for practice.

Figure 1 .
Figure 1.Timeline representing the student and instructor activities that are managed through the technology tool (iCAS)

Table 2
Student perceptions of support and nature of support they received or required during the collaborative group task(N = 840, Questionnaire)

Table 3
Student comments from questionnaire and focus groups providing examples of positive interdependency knock everyone down to make them look better … I have a friend and she was in my group and … She was like, I'm just going to give everyone 100 you know hoping that they'll give me 100, like that.[FocusGroup 1, unspecified program]

Table 4
Peer assessment outcomes for the collaborative group work 1. Students who did not participate in the task were awarded 0/100.