Topic outline

  • General

     ALTC logo  Assessment Moderation Toolkit

  • Topic 1

    About the Toolkit

    Shade White Balance by Nick Treby on Flickr.com

     

    This toolkit is designed to provide resources to Australian and partner institution teaching staff in transnational education (TNE) to promote good assessment moderation practice. TNE is referred to as education “in which learners are located in a country different from the one where the awarding institution is based” (UNESCO & Council of Europe 2001, p. 1 cited in McBurnie & Ziguras 2007, p. 22). Transnational programs are delivered in a variety of arrangements including franchising, twinning, joint awards, partner-supported delivery, and branch campus. This toolkit is aimed at academic staff in all forms of TNE.

    The toolkit is based on key principles identified by research under an ALTC project on moderation for fair assessment in transnational learning and teaching. The research involved a literature review, 85 interviews with academic and professional staff in transnational programs (59% of the interviewees were transnational partner staff), and 103 responses to an online survey (71% of the respondents were Australian academics). The project has found principles that would support good practice in moderation of assessment in transnational education. The project's paper on comparability and equivalence won the Best Paper Award at the Australian Quality Forum conference (AuQF 2010).

    • For a four minute overview of this toolkit you can take the guided tour (PPSX, 6.50 MB - Note: start the slideshow for audio)
    • For a brief overview of the research underlying this toolkit click here (PDF, 247 KB)
    • For a one page summary of the online survey results click here (PDF, 216 KB)
    • For a list of tools offered in this toolkit including research based narratives, research papers,  surveys, checklists, documents, audio clips of direct quotes from interview transcripts (read by actors) and weblinks see Toolkit Resources.

    Sections of this toolkit are:

    The toolkit also contains a glossary of terms in transnational education and a literature review (PDF, 564KB) of moderation of assessment in transnational education. It is encouraging that emerging themes (PDF, 333KB) in the literature resonate strongly with the research findings of this ALTC priority project on TNE Assessment Moderation.

    The development of this toolkit is a collaborative effort of the project team with support from the project reference group. Your feedback on this toolkit is greatly appreciated.

    Support for this project has been provided by the Australian Learning and Teaching Council Ltd, an initiative of the Australian Government Department of Education, Employment and Workplace Relations. The views expressed in this publication do not necessarily reflect the views of the Australian Learning and Teaching Council.

    Creative Commons License

    Unless otherwise noted, content on this site is licensed under the Creative Commons Attribution- Noncommercial-ShareAlike 2.5 Australia Licence

     

    • Topic 2

      Moderation and its meanings

      There are a variety of understandings and practices around moderation of assessment. Our research indicates that the meanings of moderation include:

      • Consistency in assessment and marking
      • Process for ensuring comparability
      • Measure of quality control
      • Process to look for equivalence
      • Maintaining academic standards to ensure fairness
      • Part of quality assurance

      The view held by most is that moderation is all about marking:

      "I think moderation is where you’ve got different groups of students and you have to make sure they’re marked with equal fairness." Australian academic

      For others, moderation also extends to ensuring that assessment is both valid and 'fair' in the first place:

      "Moderation is not a summative process, it's right from the beginning." Transnational partner academic

       Clip Art Listen to what an Australian academic says about moderation (mp3, 494KB). This is a direct quote from an interview transcript read by an actor.

      This project interprets moderation of assessment more broadly than just a quality control measure around marking work. This is because marking alone and reviewing allocated grades does not guarantee quality assessment. The project sees moderation as processes and activities that occur before assessment is implemented (i.e. quality assurance), as well as those that occur after assessment is undertaken by students (i.e. quality control). The project proposes viewing moderation as having three phases (PDF, 274KB) with non-linear feedback loops: 

      • assessment design and development
      • implementation, marking and grading
      • review and evaluation.

      For definitions of moderation and it's surrounding practices, see the glossary.

      For one interpretation of this approach, see the moderation process flowchart (PDF, 252KB) which shows the three phases of the moderation process.

      • Topic 3

        Streamlining moderation policy and processes

        The aim of moderation is to ensure that assessment is fair, valid and reliable. Moderation is especially needed when assessment involves large units or multiple markers, occurs on different campuses, is subjective or differs across students. Policy and process structure informs moderation practices in most cases in the ‘domestic’ context while additional policies and guidelines may operate in transnational contexts. Our research indicates:

        • The language in some current transnational moderation policies is that of control and audit
        • Moderation in practice tends to focus on 'post' processes with institutional variability
        • Assessment regimes are rarely negotiated among TNE teaching teams with a lack of shared understandings
        • Delays occur in marking and moderation feedback with lack of specificity in feedback
        • Australian staff consider marking by some partner staff as lenient, with loose referencing and different attitudes to plagiarism, where nobody fails. There is also concern that partner staff provide only summative (mark) feedback and teach to the exam

        While partner staff want to be trusted, recognised as professionals, included in curriculum and assessment decision making, Australian staff want to retain control to ensure standards are maintained, and satisfy quality audit resulting in discontent:

        "We have a high level of capable, upwardly mobile students who, from time to time, score very well.  We find that often we get, ‘Please reduce your marks across the board by 5%’. What does some lecturer sitting in Australia know about my cohort?" Transnational partner academic

        Clip ArtListen to what transnational partner academics say about the need for trust (mp3, 828KB), need to see Australian marking (mp3, 502KB), post hoc scaling (mp3, 194KB) and delays in moderation (mp3, 583KB). These are direct quotes from interview transcripts read by actors.

        Clip ArtListen to what Australian academics say about the gap between policy and practice (mp3, 692KB), trust (mp3, 210KB), and need to respect transnational partner academics (mp3, 843KB). These are direct quotes from interview transcripts read by actors.

        Our research suggests considerations for policy makers to streamline moderation policy (PDF, 173KB) that emphasise the need for joint partnership amongst the transnational teaching team with institutional support. Effective moderation practices identified are:

        • Develop a mature academic partnership between the Australian and transnational academics (working together for three or more teaching iterations of the subject)
        • Build rapport, have teaching team approach, meet each other face-to-face
        • Co-develop assessment items, negotiate curriculum and consider local contexts
        • Use marking criteria, discuss standards and variations in marking, cross mark a selection of assignments, seek peer feedback from the teaching team on performance of subject co-ordinator
        • Avoid post-hoc adjustment of marks/grades
        • Develop technology to assist moderation processes and train Australian and transnational staff in use of related technology

        The resources below cover moderation processes in general and suggestions for improving moderation in transnational contexts. They can assist you to streamline moderation policy and processes:

        • Topic 4

          Managing cultural issues

          The impact of culture in the moderation of assessment in the transnational context has a number of dimensions. Transnational teaching and professional teams conduct assessment work across national and organisational cultural boundaries. Our research found that the main challenges due to cultural issues are:

          • Differences in communication style between 'Australian culture' and other cultures, with Australian academics being perceived as being more direct by some transnational partner staff
          • Preconceived ideas about differences in learning styles and approaches of Australian and transnational students
          • Differences in attitudes towards plagiarism with Australian academics placing greater emphasis on academic writing and referencing
          • Lack of awareness of how implicit cultural perspectives impact on assessment tasks and moderation processes.

          Clip artListen to what transnational partner academics say about cultural issues (mp3, 780KB), learning styles (mp3, 527 KB, language issues (mp3, 652KB), exams (mp3, 546KB). These are direct quotes from interview transcripts read by actors.

          Clip ArtListen to what Australian academics say about cultural issues (mp3, 289KB) and plagiarism (mp3, 505KB).  These are direct quotes from interview transcripts read by actors.

          Some good practices to handle cultural issues identified by our research are:

          • Contexualise pedagogy and assessment where appropriate 
          • Establish supportive mutual relationships
          • Provide opportunities for interaction
          • Use technology to support communication, e.g. Skype, ZOOM
          • Take cultural issues into account in procedures and regulations.

          Our survey found a strong correlation between satisfaction with communication and satisfaction with moderation. An overview of the survey results (PDF, 216KB) is available.

          Resources in this section of the toolkit can assist you in managing cultural issues.

          • Topic 5

            Strengthening Teaching Teams

            Development of a community of practice in transnational programs ideally requires the input of all staff. The expertise, local knowledge and student engagement capabilities of partner organisation staff are invaluable for developing successful transnational programs. Our research found that transnational teaching teams face a number of challenges including sharing trust and control, and sharing knowledge. There are also differences in pedagogy and assessment practice.

            In one institution the exam is provided to the transnational academic staff member on the day:

            "That's very demeaning, you know? Does that mean because I'm a local instructor I'm less qualified, or I have less integrity?" Transnational partner academic

            Good practices identified by our research include:

            • Structured interaction, i.e. have an agreed time, medium and frequency for communication
            • Good communication using a number of modes, e.g. teleconference and e-mail
            • Good rapport and relationships built over time and based on trust
            • Discussion of moderation and enhancing expertise through professional development
            • Informal collegiality, e.g. meet in person before teaching starts.

            "If you trust someone you can openly admit if you've made a mistake." Australian academic

            Clip ArtListen to what transnation partner academics say about moderation policy (mp3, 367KB), rapport (mp3, 988KB), desire to be involved (mp3, 412KB), and the need to communicate (mp3 435 KB). These are direct quotes from interview transcripts read by actors.

            Clip Art What Australian academics say about the need for structured interaction (mp3, 574KB), need for  trust (mp3, 210KB), and the need to respect transnational partner academics (mp3, 843KB). These are direct quotes from interview transcripts read by actors.

            Resources in this section of the toolkit can assist you in strenthening teaching teams:

            Professional Development

            This section can help you and your teaching team engage in professional development activities around TNE assessment and moderation.

            Information to help you in your role as teacher or unit coordinator? See below for a Professional development plan for the Individual Teacher. This program steps you through a series of readings and activities that will enable you to critically examine your own practices in transnational unit delivery, and assessment and plan for more successful outcomes.

            Information to guide a review of moderation practices in a transnational unit or program with multiple academics or tutors in several locations? See below for a Professional development plan for a Team. This program provides a plan for engaging the teaching team in activities and readings that will foster discussion and critical evaluation of existing assessment processes and plan for more successful outcomes.

            Information that can support a review of moderation practices on a broader scale of a university faculty or school? See below for a Professional development plan for a School. This program steps you through a general plan for collecting information about assessment and moderation practices in transnational programs and collecting views across the School and subsequently, planning for changes. Opportunities for feedback from the School are included.

          • Topic 6

            Research Links

            To get a greater insight into the findings of this project see the research papers of this study:

            TNE processes and practices are starkly under-represented in the literature on the internationalisation of higher education. A literature review (PDF, 564KB) and emerging themes (PDF, 333KB) around moderation of assessment in transnational education have been completed in this project. While there exists a growing body of assessment research , the process of moderation of assessment in higher education remains relatively unknown. Follow the links below to leading academic journals with suggestions of research areas relevant to assessment moderation and transnational education. Full access to the journals below is usually available through the library pages of instiutions that have hard copy or online subscription:

            Some additional sources for research on transnational higher education:

             

            • Topic 7

              Glossary of terms

              The need for a glossary of transnational educational (TNE) terms has been identified by the International Education Association of Australia (2006). For a  collaboratively authored glossary of TNE terms see Glossary. This glossary is publicly accessible and aims to be a sustainable site of engagement with issues associated with TNE. A printer friendly version of the glossary can be accessed by clicking on the printer icon in the top right hand corner of the screen when displaying the glossary.

            • Topic 8

              Toolkit Resources

              The resources offered in this toolkit include checklists, project findings, narratives, research papers, surveys and online resources.

              Checklists

              Documents

              Project Findings

              Presentations

              Narratives

              Research Outcomes

              Clip ArtAudio clips of transnation partner academics about (direct quotes from interview transcripts read by actors.)

              Clip Art Audio clips of Australian academics about (direct quotes from interview transcripts read by actors.)

              Surveys

              Online resources

              • Topic 9

                Narratives

                These fictional narratives are stories that draw heavily on the research findings of this Project. The people in them are fictional - they represent no one in particular. Many of our research participants may recognise themselves or their situation because the situations described are common to many.

                Narrative 1: Policies and processes (PDF, 326KB)

                Narrative 2: Assessment comparability (PDF, 313KB)

                Narrative 3: Moderation and assessment practices (PDF, 318KB)

                Narrative 4: Moderation and assessment practices (PDF, 396KB)

                Narrative 5: Professional development (PDF, 259KB)

                Narrative 6: Cultural issues (PDF, 256KB)

                • Topic 10

                     Focus                                                                                              Activities for Self

                  Exploring understandings about assessment and moderation  

                   What do we know about…                                                                 Complete Survey 1

                  • assessment?                                                                             Read  Research Paper 1 and reflect
                  • moderation?                                                   

                   Are our understandings consistent…                                                  Document key points

                  • within the team?                                                                       
                  • across borders?

                  with policy or guidelines?

                   

                  Mapping existing moderation practices in transnational programs

                   What do we currently do?                                                                 Complete Survey 2

                  • Do we have policies or guidelines?                                           Complete brief checklist
                  • Who knows about them?
                  • Are they followed—how do we know?                                      Read Research Paper 2 
                  • Are policies reviewed?                                                            

                                                                                                                             Identify issues

                   

                  Examining expectations about assessment in transnational programs

                  What are our responsibilities?

                  • What does comparable mean?                                               Complete Survey 3
                  • What makes assessments comparable?                                 
                  • Who determines standards, and how?                                   Read Case Study 1 and propose a solution

                                                                                                                            Read Research Paper 3

                   

                  Building effective transnational relationships

                  •  How do we communicate?                                                                                  Read Case Study 2
                  • How do we come to understand others’ views and positions on assessment?   Read Research Paper 4 
                  • Who makes decisions?                                                                                         Plan for change as required 
                  • Trust—what does it mean?