Sustainable assessment for large science classes : Non-multiple choice , randomised assignments through a Learning Management System

This paper reports on the development of a tool that generates randomised, non-multiple choice assessment within the BlackBoard Learning Management System interface. An accepted weakness of multiple-choice assessment is that it cannot elicit learning outcomes from upper levels of Biggs’ SOLO taxonomy. However, written assessment items require extensive resources for marking, and are susceptible to copying as well as marking inconsistencies for large classes. This project developed an assessment tool which is valid, reliable and sustainable and that addresses the issues identified above. The tool provides each student with an assignment assessing the same learning outcomes, but containing different questions, with responses in the form of words or numbers. Practice questions are available, enabling students to obtain feedback on their approach before submitting their assignment. Thus, the tool incorporates automatic marking (essential for large classes), randomised tasks to each student (reducing copying), the capacity to give credit for working (feedback on the application of theory), and the capacity to target higher order learning outcomes by requiring students to derive their answers rather than choosing them. Results and feedback from students are presented, along with technical implementation details.


Background and Motivation
In most Australian universities, first year chemistry is assessed largely by multiple choice (MC) examinations because of the enormous class sizes (over 2000 students at some universities; around 850 at the author's institution).These types of examinations have well-known weaknesses in both motivating and assessing quality learning (Nicol, 2007).However, they have the advantage for lecturers of being quickly and easily marked for large groups of students, and for students of being perceived as objective.If best practices in designing MC tests are followed, they can promote and test higher-order learning (Parmenter, 2009); however, it is generally accepted that students do not approach learning in a deep way when they know that assessment is MC (Scouller, 1998).
First year chemistry involves learning a large number of new concepts (Nurrenbern & Pickering, 1987), and the students require extensive practice to be able to apply these concepts to different fact situations (Pendley, Bretz, & Novak, 1994).In the first year chemistry units at this institution, low stakes, progressive assignments are used to give the students the opportunity to practice the concepts and to motivate their study.In three of the four first year chemistry units at this institution, these assignments are made available on the Learning Management System (LMS) to be downloaded by students, submitted several weeks later as hand written documents and then hand marked.There are several problems with this: • The students often copy answers word for word from each other.This is easily noticed when marking, because some strange errors are found on multiple assignments.• The marking sessions are very time-consuming (usually 6 people for a 5 hour marking session) and relatively expensive due to remuneration rates for sessional staff, especially given that the sessions occur multiple times each semester, every semester.• Once the assignments have been marked, the marks must then be input into a spreadsheet, which is not only time-consuming, it is also subject to human error (particularly when several students have similar names).• Although students are awarded part marks for showing their working, it is unfeasible to ensure consistency in the way part marks are awarded over such a long session with a large number of markers.• In 2010 and 2011, lecturers in two of the first year units at this institution found several of their exact assignment questions asked on websites where students request answers (known as Answer Sites).The unique wording of the questions was such that these could only have originated from students from this institution (and in one case the student used their real name, so this could be verified) (Associate Professor P. Fredericks, personal communication, March 20, 2011;Yahoo!7 Answers, 2011).
Thus, the use of a LMS to provide a computer-marked assignment that offered different sets of questions to each student was appealing.This would address the issues of copying (to some extent), time and financial costs of marking sessions, and consistency in awarding partial marks in hand marking.It does not prevent students from using Answer Sites to attempt to obtain the answers to their particular questions (although the quality of answers provided on those sites is known to be low (Adamic, Zhang, Bakshy, & Ackerman, 2008).This last problem can be at least partially addressed by reducing the time available for completing the assignment.

Problems with Multiple Choice Assessment for Chemistry
In university chemistry, students learn a new way of expressing knowledge, through the use of chemical symbols and equations.Therefore it is extremely important that they develop and practice the fundamental skills of writing and balancing chemical equations.MC questions that require only the choice of the correct answer from a list, rather than writing the answer, are unlikely to lead to the development of these skills.
In many cases, the same question can be asked either as MC or as short answer.When MC questions are used, basic skills of logical deduction are required, as well as some chemistry knowledge.When the correct responses are chemical formulae, equations or numerical answers, more and higher order chemistry learning outcomes (LOs) are required within the SOLO taxonomy (Biggs & Collis, 1982;Biggs & Tang, 2007).In particular, a correct answer to a MC question may indicate a unistructural approach, but such an approach is unlikely to lead to the correct response for a short answer question, in the absence of cues or clues in answer choices (Schuwirth, van der Vleuten, & Donkers, 1996).To determine the correct response to a short answer question, students require at least a multistructural or relational approach, depending on the question.The SOLO taxonomy has been used to characterise student learning in organic chemistry by a detailed analysis of written answers (Hodges & Harvey, 2003).Within the tool described here, the approach taken by the student can only be deduced from the answer provided, so such a detailed analysis is not possible.However, requiring students to write their answer rather than choose it will take them beyond a unistructural approach.
An example of the type of question, which requires higher order learning if requiring a written answer rather than MC, is: The products for the reaction of lead nitrate with potassium sulfate in aqueous solution are: none of the above It can be seen that the MC options prompt the student with the clues of correct chemical formulae and therefore allow a lower order level of learning to enable the correct choice; knowledge of the solubility of the products is sufficient, which is unistructural.In contrast, if this question is asked as a written question without the five answer choices, the student must understand the theory to a deeper level.In the latter case, the student must know (or look up) the correct chemical symbols and charges for the ions, which are both essential LOs for first year chemistry, and relate these to the solubility of the compounds, in a multistructural approach.Many variations on this particular question can be asked by altering the two metals and the two anions.
Other examples of questions that require higher level learning when asked as short answer compared to MC are any requiring numerical calculations.When choices are offered, the students are given at least one answer with the correct value and the correct order of magnitude.It is therefore easier for students to work out which choices are incorrect than to obtain the correct numerical value with no prompting.This is apparent from reading the answers given by students in both hand written and online assignments -some bizarre answers are reached, which would never have been given as an option in a MC setting because they are so unrealistic.Thus, the feedback to lecturers on student misconceptions through an analysis of incorrect written responses is also lacking in MC assessment.
A final problem with MC questions is that they can make it difficult for outstanding students to excel.This is because it is very easy to accidentally enter the wrong choice, even when the correct answer has been obtained.

Aim of this Project
In light of the background above, it was desirable to develop assignments for a first year chemistry unit that would be computer-marked but not MC.This institution uses BlackBoard as a LMS, and this package is capable of automatically marking answers in numerical or text formats.It is also capable of assigning students different sets of questions randomly from pools.Thus, the aim of this project was to use the full capability of the LMS to develop non-multiple choice, randomised assignments for a new unit that was run for the first time in 2008.Feedback from students was used to improve the assignments in subsequent semesters and the details are presented below.
It is worth noting that many modern text books have online components, in which the publishers have set up an enormous variety of sophisticated non-MC questions that can be assigned to students for formative and summative purposes.Two popular examples are from Wiley (2011) and Pearson (2011).However, for this project it was thought that it is better to remain within the internal LMS.This allows questions to be tailored to the exact content covered at this institution, and enables the questions to be reused regardless of changes to text books.It also allows the questions to be independent of the students having purchased a particular text book or signing up for an online resource.The policy of the author's institution is that while externally hosted sites are not recommended for assessment, they are permitted.
The assignments form only a small portion (2 × 5%) of the total assessment of the unit involved, which is also assessed by practical laboratory reports (10 × 5%), a progress examination (10%) and a final examination (30%).The goal of the assignments is to encourage students to engage with the theoretical material to support their learning in the laboratory and to obtain feedback on their progress throughout the semester.Within the SOLO taxonomy (Biggs & Collis, 1982), a relational or extended abstract approach is then required of students to relate the theory in the assignments to their laboratory work.

Design and Initial Implementation
The design of the online assignments started with the written assignments used in other first year chemistry units as a model.In those units, two or three assignments are given, each containing 10 questions, and each worth 5% of the final mark.The students have three or four weeks to complete each assignment from when they are made available on the LMS until the date that they must be submitted, as paper documents.
In this project, the LMS was intended to mark the assignments, so the answers could only be a number, a single word or a chemical formula.These answer options allow the students to enter exactly what has been set as the correct answer.The content of first year chemistry lends itself well to questions with such answers, because students learn to perform a large variety of simple calculations and become familiar with many simple chemical formulae.For numerical answers, the LMS allows a range of numerical answers to be marked correct.For example, if the correct answer is 10.2, the question can be written to accept anything between 10.1 and 10.3, or between 10.15 and 10.25, as specified in the question.Thus, if a student answers 10.19, they will be marked correct.In some cases, it is more appropriate to write the question such that only the exact answer is accepted, depending on the theoretical basis of the question and the sources of uncertainty.For text answers, it is possible to specify as many answers as necessary to be correct if there are possible ambiguities in how the answer should be typed.For example, if the correct answer is the name of the chemical compound 1,2-dimethylbenzene, the common name of this compound, ortho-xylene, must also be specified as a correct response unless the students are aware that only formal names are acceptable.
The use of the LMS allowed each question to be written with multiple variations to form a pool for each question type.In this project, around 15 variations were written for each question.For questions with numerical answers, the question variations were generated reasonably easily using Excel.Figure 1 shows a screenshot of a question generator that was designed by the project team.
On the page that is shown, the question requires determination of the percentage yield of a reaction.To obtain the correct answer, the students must first write a balanced chemical equation for the reaction.They must then determine the limiting reagent and maximum theoretical yield before they can determine the percentage yield obtained in their version of the question.For qualitative chemistry questions, the questions must be individually written, which is also time consuming.It is particularly difficult to ensure that the difficulty of all questions is comparable for non-numerical answers; this may impact on the reliability of the assessment.An illustration of this is if the question requires the identification of an organic compound.Small compounds such as methanol and ethanol might be considered easier to identify than more complex molecules such as 2-pentanone.This limits the number of possible different questions in the pool.
Once the questions have been written, the question pools themselves must be written within the LMS.Because the text of each question within a pool is virtually identical, it is relatively quick to make 15 copies of the question and then edit each question individually to input the variable values and the correct corresponding answer(s).An example of the resulting pools is given in Figure 2. It is worth noting that the LMS does have the ability to itself randomly generate questions, given a range of values for a variable.This facility has been used in other disciplines at the author's institution (Dr G. Isoardi, personal communication, April 4, 2011).However, for the specific questions used in this project, the function was not used because we wanted to be sure of the exact numerical values that the students would be asked.
The content covered, answer type and marks available for each question in the two assignments are given in Table 1.Within the LMS, each student then received a randomly generated assignment consisting of one question out of each pool.An example of the appearance of the assignment as seen by the students is shown in Figure 3.The students had several weeks from when the assignments were available on the LMS to complete and submit their assignment.They were encouraged to print out their own assignment and to work in groups if desired to determine how to solve the questions.
The instructions provided with each assignment are as follows: 1.
The Assignment is on the BlackBoard site under "Assessment", "Assignment Number".Once you click "OK" to begin the Assignment for the first time, you will get your Assignment.Every student will have a different Assignment.

2.
You can open the Assignment and look at the questions as often as you want.It is strongly suggested that you print it out to work on it.You may save your answers as you go and return to complete the Assignment later.You can also change saved answers.

3.
All answers are one of the following: a number, a chemical formula, a single word ("yes", "no", "increase", "decrease") You do not need to provide units with any answer.For chemical formulae, you do not need to write subscripts or superscripts, but you must write brackets where required.Do not leave any spaces in your chemical formulae or the software will not recognise your correct answer.e.g.iron (II) nitrate is to be written Fe(NO3)2 4.
For numerical answers, be careful with the number of significant figures.Providing too many significant figures is wrong, just as providing not enough significant figures is wrong.Exponential notation should be entered as follows: 3.4 x10 -6 is written 3.4E-6 into the BlackBoard assignment.

5.
Not all of the questions are worth the same number of marks; more time-consuming questions (e.g.you may need to plot data or solve an equation) are worth more marks.The Assignment is marked out of 100, and it is worth 5% of your final mark for SCB131.6.
When you have finished and have answered all the questions, click on "submit".YOU CAN ONLY DO THIS ONCE.Once you submit your answers, you will receive your marks and a list of which questions you got right and wrong.7.
If you have any technical problems whilst using BlackBoard, ring 3138 4000 or email ithelpdesk@ qut.edu.auimmediately and report it.8.
Assignments may not be submitted after the due date.Assignments close at 5 pm on the due date.It is your responsibility to ensure that you attempt the Assignment before the due date; if you miss the Assignment for any reason you will not be given another opportunity to submit it.9.
It is suggested that you attempt the Assignment from a QUT computer.Failure to submit the Assignment due to an inability to access the Assignment from a non-University computer is NOT an acceptable excuse.10.Keep your receipt upon completion of the Assignment in case there is any problem with your mark.
11.It is strongly recommended that you not leave it until after 4.00pm on the due date to attempt the Assignment in case you experience any technical difficulties.You will still be expected to attempt the Assignment before the original deadline.

Student Feedback on Initial Implementation
The tool was used for the first time during the first iteration of this unit in semester 2, 2008.In the first implementation, there were no practice questions and the student feedback strongly indicated that the students found this difficult.Of the 61 written responses within the end of semester Student Experience Survey to the question "What aspects of this unit are most in need of improvement?" 10 addressed the online assignments.Those (unedited) comments from the students relating to the assignments were: • The online quiz was pathetic i got all the answers right but I didn't get the marks due to rounding error which wasn't taught into get depth.• Online assessment -no credit given for staged working out and small margin for error ie right/wrong answers • Assignments-often extremely difficult and online submission seemed unfair as marks could be lost very easily.• I think that assignments need to be mark by persons.Submitting on line is like a 50/50 chance.
As well as we don't have mark for our working out etc. • The online assignments were a complete waste of time and did not accurately assess student's knowledge or abilities.It seems amazing to me that I can spend days working out a particular calculation, and lose the ENTIRE 20 marks (out of 100) because I put the Ph at 8.88 instead of 8.85??? • The online assignment format was horrible, many many student lost a large amount of marks for answers the computer did not register because of rounding at different points ( the amount of error she gave was .000001for a question with only 3 significant figures.)• the assignments were not particularly well thought out or very helpful in the end.It was difficult to determine where I had gone wrong in the questions I answered incorrectly & since part marks were not available for correct working etc the was exaggerated.
• The online assignments need some attention.I have done online assignments before and it was a much better system.I realise that chemistry requires a high level of accuracy but it was a little frustrating to get marked completely wrong for rounding errors.• The online assignment definitely needs to be re-evaluated as I was not happy with getting a question wrong when my answer was only .005 off the correct value and I didn't receive the notice regarding carrying sig.fig's all the way through til later.

• Assignment on blackboard is in dire need of review!!!
Only one student response relating to the assignments was received among the 57 responses to the question "What were the best aspects of this unit?" as follows: • Online Assessment was a good inclusion to the unit In semester 1, 2009, a different first year chemistry unit at the same institution that normally uses paper assignments trialled a similar online assignment for one of the three assignments in that unit.Students were surveyed regarding which assignment format they preferred; 61 out of 67 respondents preferred the paper assignments.The following (unedited) comments regarding the online assignment were received from those who preferred the online format: • This assignment was much better because feedback was given straight away in preparation for the final exam • It's good as I didn't have to worry about units or making sure my working out was legible.It was also easier as I didn't have to hand it in manually • Easy to complete -different questions for every student which was good.
• Easy to summit, no mass on the paper, save the paper for ecosystem.
• It's good to have an assignment that's different to everyone else's.It's also easier to submit!• It lets the students complete the assignment to their full extent rather than cheating off the paper assignment.• They are easy to work on, if I need to write questions out I still can, it's easy to save and try again later.It's great getting a mark back straight away!• I do like online assignments, only because you get your grade instantly.
Among the students in that unit who preferred the paper format, the overwhelming reason was that part marks were awarded for working, so even if their final answer was wrong they would receive some marks for that question.Students understandably were very upset at losing all the marks for a question simply because they put too many digits in their answer (for example).It is clear from these comments that the instant feedback of the online assignments is seen as a great benefit compared with hand marked assignments, for which there is usually a delay of around two weeks between submission and receiving their marks.It is worth noting that for these large classes, the hand marked assignments do not include detailed individual feedback, although detailed worked solutions are provided.

Further Implementation of the Assessment Tool
In response to the negative feedback from students regarding being given credit for their working, for the second iteration of this unit in semester 2, 2009, an extra set of each pool of questions was written, which required a moderate input of time.Extra numerical questions can easily be generated, but questions involving chemical compounds are more difficult, because it is important that all the compounds be real substances and that the questions be of similar difficulty.There is no overlap between the practice pools and the pools of questions for the summative assignments.The students were then given access to formative assignments based on the new pools.Students were able to attempt the practice assignments as often as they wanted.
Provision of the formative assignments vastly improved the student feedback on the summative assignments, as seen from the following end of semester Learning Experience Survey feedback, where four out of 57 responses to "What were the best aspects of this unit?"addressed the online assignments: • I really liked the practice assignments.They were very helpful in helping me to determine the correct way to give the answer.• The prac assignment was awesome, great to be able to check your reasoning before submitting the assignment was a huge bonus.• Assignment practices were good • The assignments were the best aspect of the unit.
However, five out of 59 responses to "What aspects of this unit are most in need of improvement?"addressed the online assignments, again with the complaint that part marks are not awarded for working, and no marks are awarded if there is a small error in a numerical answer.The range of acceptable answers can be further widened to address this, as well as reminding students that their working on the formative assignments results in better outcomes in the summative assignments.
The practice assignments are very useful to the academics involved, because it is possible to look up individual student attempts on the formative assignment.All details can be accessed, including what answers the student gave, how many attempts they made and when the attempts were made.Thus, if a student queries an aspect of the summative assignment, it is possible to check whether the student took advantage of the formative assignment before submitting their summative assignment.
It is possible to directly compare the outcomes of students in the two assignments over the two years, without (2008) and then with (2009) the formative assignments; Table 2 contains this data.The questions were not changed and the student cohort was similar each year.The marks for Assignment 1 over the two years were significantly different; t(401) = 3.64, p < .001,with the students that had access to the formative practice assignments in 2009 receiving higher marks.In contrast, the marks for Assignment 2 were identical over the two years.This is most likely due to the types of responses in the two assignments (Table 1).Assignment 1 requires mostly calculated numerical answers, for which the students can check their procedure using the formative Assignment.By comparison, most of the answers in Assignment 2 are chemical names, formulae and whole numbers.The underlying theory to determine the correct answers for these questions is more conceptual, and access to practice questions apparently makes less difference.
The concept of sustainable assessment has been developed by Boud (2000, p. 152).He states that for assessment to be sustainable, students should be able to "identify whether they have met whatever standards are appropriate for the task in hand and seek forms of feedback from their environment (from peers, other practitioners, from written and other sources) to enable them to undertake related learning more effectively." By attempting this formative assessment and checking their results, students are able to determine whether they are meeting the standard for the assessment task before submission of the summative assignments.Thus, provision of the practice assignments was very important to make the assignment assessment sustainable, and enables the students to learn more effectively.

Weaknesses of the LMS for Online Assessment of Chemistry
There are several technical problems with the use of the LMS to administer the questions, which diminish the value of this form of assessment compared with written and hand marked assessment.These relate to the input of numerical values and units and are explained in detail here.
In first year science, one concept that students are learning is the extremely important concept of measurement uncertainty, which encompasses error, accuracy and reporting significant figures.Thus, they learn that it is incorrect to specify more decimal places on an answer than can be experimentally measured.For example, in a titration, the concentration determined could never be correctly written as 0.192549 molL -1 , because the equipment used simply does not allow that level of accuracy.Therefore, values must be given to the correct number of significant figures, such as 0.19 molL -1 .However, if the LMS accepts a range of answers (for example, 0.18 -0.20 molL -1 ), the response 0.192549 molL -1 would be marked correct by the LMS because it lies numerically within that range.This response would in fact lose marks in a hand marked assignment for giving excessive significant figures.To obviate this problem, it is possible to specify in the LMS no range of values, so that the students must give the exact correct numerical answer in order to be marked correct.However, in that case they may use a slightly different rounding technique or value for a physical constant at some stage of their calculation, and thereby legitimately obtain a slightly different final value.Then they are marked incorrect by the LMS, although for a hand written assignment, they would be given most or all of the marks.
A second problem with numerical answers within the LMS is that for the computer to recognise the numerical response as a value within the range specified as correct, the students cannot give units.Learning to give the correct units for a measurement (such as mL, molL -1 , bar etc) is a very important part of first year science, so not requiring units to be given in the answers is a significant weakness of the LMS with respect to the LOs.This can be avoided by having the students input their answer as text rather than having it be treated as a numerical value, because the text field can be anything specified.The problem then arises that the numerical answer and units must be given exactly as specified or the answer will be marked incorrect.For example, in the above titration, the answer could require the student to input 0.19 molL-1 to be marked correct.If the student types more or less spaces, or gives a slightly different value of 0.18 or 0.191 molL-1, they will be marked incorrect, although they have understood and correctly used all aspects of the theory.
In our current implementation, the units for numerical responses (such as mL, V, molL -1 ) are provided to the student as part of the question (as seen in Figure 3, question 3) and they are only required to input their numerical response, so that the value can be recognised by the LMS as lying within the correct range.Thus, students giving an excessive number of decimal places are not being penalised, and students are not required to know the units for their answers.As can be seen in the student feedback, at least some of the students recognise that this is an easier option than requiring them to input the units themselves, and in fact providing the units may act as a cue to prompt their response.This issue will be addressed in future implementation of the tool, so that students are also tested on their understanding of units.This can be done by using multipart questions, so that the units must be entered by students separately from their numerical responses.
A final, minor problem with any online assessment is that students occasionally make a genuine typographical error, for example inputting units when these are provided.It is relatively easy to correct individual marks if the students let their instructor know that this has occurred.With large student numbers, the time required to make the minor corrections increases, and there is always the risk that a student does not inform the instructor so they do not receive their due mark.The relatively small weighting of these two assignments (5% each) makes this possibility unimportant.

Conclusions
The tool developed in this project requires extensive resources to set up: around 6 hours per question for writing, inputting to the LMS and checking, a total of at least 120 person hours.However, once in place, it requires minimal effort each semester.In comparison, hand marking and data entry takes around 65 person hours per semester, so within two years this tool should be a worthwhile investment of time.Students appreciate that their classmates are not able to cheat and that they receive their results instantly upon submitting their assignments.The advantages of instant marking, automated mark entry, provision of unlimited practice questions and avoiding cheating due to randomisation outweigh the weaknesses described above.The questions asked in the assignments are aligned with the teaching activities and intended LOs of the unit, making them a valid form of assessment according to Biggs (2003).They are reliable, because the LMS is consistent in its marking.Sustainability is achieved through the use of the practice assignments, so that students are able to self-monitor their own progress (Boud, 2000).Thus, we have developed and used a form of assessment that is valid, reliable, sustainable, and takes advantage of the power of the modern LMS.It fulfills the requirements stated in the Australian Learning and Teaching Council document Assessment 2020 (Boud and Associates, 2010).

Figure 1 :
Figure 1: Screenshot of the Excel document used to generate numerical questions; the page shown is for Assignment 2, question 5

Figure 2 :
Figure 2: Screenshot of question pool for Assignment 2, question 2

Figure 3 :
Figure 3: Screenshot of part of Assignment 1 in the student view

Table 1 :
Content, answer type and marks for each question in each Assignment

Table 2 :
Summative assignment results without and with formative assignments