Evaluation of the pilot project

The pilot project was evaluated from a number of angles: the student evaluations through SETs and SEUs, the survey to seek students' responses to the integrated assessment process and the interview with the sessional teacher.

SEUs and SETs are part of the quality assurance process at QUT, offering a common approach to evaluate of teaching and study programs across the university. The instrument used in ITN338 comprised twenty questions, with ten focusing on the curriculum, learning activities and learning materials, and ten focusing on the academic staff member's approaches to teaching and her support for learning. Students were asked to indicate the level of agreement with the given statements using a five-point Likert scale (strongly disagree/disagree/ neutral/ agree/strongly agree with an option for NA (does not apply).

Two questions sought student responses about the intellectual focus of the unit and the learning objectives:

• I understand from the unit materials (e.g. unit outline, study notes, OLT materials, handouts etc.) what learning and skills I am expected to learn by studying this unit.

• The topics and content of this unit are clearly related to what I am expected to learn.

Four questions dealt specifically with items of assessment:

• I understand the requirements of the overall assessment program (e.g. minimum unit requirements).

• The assessment tasks are clearly related to what I am expected to learn.

• I have been provided with guidelines or criteria which give me a clear explanation of how individual assessment tasks will be marked.

• The teacher gives me feedback that helps me improve my learning.

17 of the 36 students enrolled in the unit completed the SEU/SET, a response rate of 42.5%. The responses to the questions on assessment were consistent, with the majority of students strongly agreeing or agreeing with the statements. While 94% of respondents agreed or strongly agreed that they understood the learning objectives, the lower figure of 76% felt that the topics and content had actually achieved these goals.

94% of students agreed (59%) or strongly agreed (35%) that they understood the requirements of the overall assessment program. Only one student (6%) disagreed and two students (12%) were neutral about the assessment tasks being clearly related to what they were expected to learn and that they had been provided with guidelines or criteria that clearly explained how individual assessment tasks would be marked. Responses to the questions about valuable support from the teacher through the feedback provided were also positive (88%), with 47% agreeing and 41% strongly agreeing. It is felt that a major shortcoming of the current SET/SEU practice at QUT is the condition of 'surveyitis' suffered by students, with the result that while they complete the quantitative measures on the survey instrument, there is little interest in providing qualitative feedback.

The separate survey instrument was therefore designed to capture more qualitative data about the students' experiences with the concept of integrated assessment tasks through the process of creating an online journal and the level of satisfaction with CRA in the pilot project. Basic demographic data was collected about the students' gender, age and enrolment status, which potentially allowed some correlation with data collected at other stages of the course. Again using a five-point Likert scale, students were asked to indicate the level of agreement with a number of statements. Each question also had space for students to provide comments about their response. The questions about the process of developing an online journal included:

• The process of creating an online journal was a valuable learning experience.

• The process of creating an online journal helped me develop an understanding of the discipline content of the unit.

• The OLT environment facilitated the development of the online journal.

• The process of learning to produce work in the genre of a journal article was a valuable experience.

• The process of peer review was a valuable learning experience.

• This unit has encouraged me to consider writing for professional publication in the future.

Responses to these questions are presented in Table 1. The following abbreviations are used:

SA     Strongly agree

A       Agree

N       Not sure

D       Disagree

SD     Strongly disagree

The process of creating the online journal was seen by students as a positive approach to learning:

"The process was challenging, yes, but a very valuable experience"

Only one student did not find the process a valuable experience. One third of students stated that the process did not necessarily help them understand the discipline content of the unit. However, as the comments provided by several of these students indicated that the process of research and writing the article for the journal was indeed valuable as a way to learn about the discipline content of the unit, it was felt the wording of the question could have been improved to more clearly state what was meant by 'the process of creating the online journal'. A couple of students who did not find it an effective learning process indicated that they did not like the content of the unit being developed by the students themselves, as they felt it lacked authority. Students had grave concerns about the value of the OLT system as the medium for publishing journal, with many of them expressing their dissatisfaction with the system. These concerns reflected the technical difficulties experienced at the beginning of the semester which presented the teaching staff with immense challenges to achieve the desired outcomes of publishing to the online system.

91% of the students appreciated the opportunity to write in the new genre of a journal article, resulting in 50% of the students agreeing that they would consider writing for professional publication in the future. Some students specifically appreciated the need to adapt the written article for an oral presentation:

"Having to prepare the material for different forms of communication made it a more valuable learning experience"

As a further 40% were unsure, the idea of writing for publication and for conference presentations could be taken up as career development activity to foster greater interest amongst library and information professionals.

The peer review process was also well-received, with 82% finding it a valuable process. Comments indicated, however, that the scheduling of the peer review activities needed to be improved, but the timeframe of the semester presented logistical problems. The academic staff would welcome a collaborative partnership with another institution teaching in the same field, to provide a more objective and anonymous approach to peer review. Nevertheless, overall it was felt that the process of learning through writing a journal article was successful, but that there was a definite need to improve the strategies for publishing the product itself.

The questions about the use of CRA in the unit were:

• Criterion referenced assessment helped me understand what I was required to do for the assignments.

• Criterion referenced assessment helped me understand the strengths and weaknesses of my work.

• I found the level descriptors for each criterion clear to understand so I could relate them to my own work.

Students were also asked if they had used CRA for assignments at other stages of their university studies, with the opportunity to comment on the perceived value of CRA as an assessment tool. Two additional questions asked about the overall level of satisfaction with CRA in the unit and whether CRA should be used in more units at QUT. Twenty-two students returned the questionnaire, representing a response rate of 61%. While 86% of the students agreed or strongly agreed that CRA had helped them understand what they were supposed to do for the assignments, some concerns were expressed about whether the process actually helped them understand the strengths and weaknesses of their work. The responses to the questions about CRA are presented in Table 2.

14 students (64% of respondents) had been exposed to CRA in earlier studies. These students highlighted the value of CRA in articulating what was expected of them in the assessment tasks, and the grade they could anticipate achieving. A number of students felt that a rudimentary form of CRA had been used in other units of the course, to give students a good idea of what was being assessed in the assignments, and that they were used to this approach. This was the first time that the level descriptors had been presented as a complete CRA matrix. One student indicated that the relative clarity or fuzziness of the criteria remained an important issue for them, but felt the wording of the criteria in ITN338 was clear and logical. Students felt it helped them better understand the feedback provided by the teacher, as well as serving as a stimulus for their own self-reflections on learning.

The comments provided by the students were valuable in developing a better understanding of their experiences with CRA. Positive comments about the value of CRA included:

"It gave me an indication of what is being assessed and how I should concentrate my efforts"

"Having designed/used CRA for many years, it was invaluable in helping me determine requirements"

"After you got received your grade back, the criteria were helpful in judging what the marker has assessed"

"The criteria served as a stimulus to the reflections on my learning"

Students who responded negatively to the CRA process expressed their concern about the "amount of assessment" rather than about the "process of assessment". One student apparently did not relate to the concept of CRA at all. In responding to the questions about CRA helping understand assignment requirements, the student commented:

"This alarms me as I have no idea to what this refers"

To the question about the clarity of the level descriptors, the student declared:

"Oh dear-what have I missed here?"

While this feedback was an isolated incident, given the amount of discussion in class about the pilot project itself as part of the Teaching Fellowship, and the description and rationale presented in the study guide, it is interesting to find the student had "missed" it all.

It was found that, overall, the students responded positively to the use of CRA in the unit. While there were a few concerns about some perceived discrepancies with the criteria, the feedback helps the academic staff scrutinize the wording of the criteria and to consider how to reduce the fuzziness and enhance the clarity of the standards of achievement.

The interview with the sessional staff member offered the opportunity to consider the value of CRA from the perspective of the educator. Benefits were identified for the teacher in the classroom as well as in the marking of assignments. The criteria sheets themselves provided opportunities to focus on the work to be completed by the student in the individual assessment items and to stimulate classroom discussion on the assessment product within the context of the learning process. The range of criteria across the different assessment items encouraged discussion about desired learning outcomes across the spectrum of discipline knowledge and generic capabilities. The fact that students had considerable choice in selecting a topic for their journal article meant that the criteria in themselves offered common ground for exploring the expectations of students in their assignment work.

The semi-structured interview with the sessional teacher focused on four key questions to explore how it felt to have marking criteria to guide the marking process; what the actual marking process was; whether there was a need to deal with student queries as a result of the marking criteria; what changes should be made to the marking criteria. The sessional teacher found that the CRA process supported her own learning as a new lecturer, helping her to better understand the process of marking assignments and to develop her own confidence in assessing the work of others, particularly in terms of accountability, by being able to align her "gut instinct" with "professional judgement". The level descriptors made it easy to identify the relevant grade for individual pieces of work and to justify the marks or grades awarded, so that the marking criteria provided firm support for the "fairness" in the decisions she made. She felt it was very important to discuss the criteria with students early in the semester to ensure that the assessment process, the desired learning outcomes and the expectations for academic standards were clearly understood.

Generally speaking, academic staff have found that CRA reduces the number of challenges to assessment, i.e. students questioning the marks and grade awarded for a piece of assessment. In ITN338, there was only one incident of the marks being challenged, which was resolved satisfactorily through a discussion of the stated criteria for examples of the practical application of the theories presented, which had not been addressed in the student's work. The sessional teacher concluded that she had found using CRA in the unit a very positive experience that had helped her own professional development.

اخر الاخبار

اخبار العتبة العباسية المقدسة

قسم الشعائر يواصل تحضيراته لتنظيم عمل المواكب الحسينية في زيارة الأربعين

قسم شؤون المعارف يناقش تحضيرات المؤتمر العلمي الأول لمركز تراث البصرة

قسم الشؤون الفكرية يقيم دورة تطويرية لملاكات هيأة المساءلة والعدالة في الأرشفة الإلكترونية

المجمع العلمي ينظم مجلس عزاء في قضاء الهندية

المزيد

الآخبار الصحية

نصائح عملية للحد من التعرق المفرط في الصيف

أطباء يحذرون: مكمل غذائي شائع قد يدمر الجهاز العصبي

تحذير طبي من تأثير جانبي "غير متوقع" لحقن التخسيس

حمامات الثلج قد تكون خطيرة وأحيانا مميتة.. ما السبب؟

المزيد

الآخبار التكنلوجية

أكبر جدارية شمسية في العالم تدخل موسوعة غينيس.. مشروع ضخم

تسارع دوران الأرض يثير القلق.. وتحذيرات من "كوارث محتملة" ؟

"سوذبيز" تطرح أكبر قطعة من المريخ على الأرض في مزاد علني

أوبن أيه آي تعتزم إطلاق متصفح إنترنت يعمل بالذكاء الاصطناعي

المزيد

مواضيع ذات صلة

Blending Process with Product: Using Assessment to Drive Learning Through the Creation of an Online Journal Conclusions

Assessment as the driver for learning in ITN338

Methodology

Criterion referenced assessment

The learning context

Blending Process with Product: Using Assessment to Drive Learning Through the Creation of an Online Journal In September 2003, Queensland University of Technology (QUT) endorsed a new assessment policy to introduce criterion referenced assessment (CRA) in

Innovative Assessment and Learning in a Problem Based Environment Conclusions

Assessment of Accomplishments

Enhancing Teaching

Discussion; Enhancing Learning

Instructors

Professional Behavior