The case for program level assessment design

Image generated using Flux AI (Dev)

The below is excerpted from a more in-depth Teaching@Sydney article on program level assessment design.

The University’s two-lane approach’ to assessment focuses attention on the legislated need to ensure our graduates have met the learning outcomes of their courses in a world where generative AI is ubiquitous and can mimic many of the products of ‘traditional’ assessments. This calls for assessment design at the program-level, as much as possible, with lane 1 assessment in which the use of AI is controlled validating attainment at relevant points throughout and at the end of each student’s study.

 

Is program level assessment the same as ‘programmatic assessment’?”

No – the term “programmatic assessment” is deliberately not used in the assessment principles in the Coursework Policy or Assessment Procedures. Programmatic assessment is a detailed area of scholarship with existing and extensive usage, in medical education Links to an external site. in particular. Programmatic assessment is associated with detailed assessment planning to enable longitudinal data to be regularly captured and used to inform educators and learners on their progress (Van der Vleuten et al. 2012 Links to an external site.). For our programs which are competency based, such as dentistry, medicine and veterinary science, a fully programmatic approach may be warranted. In most of our degrees, though, it is not.

 

What are the advantages for students and educators of program-level assessment?

• Holistic design: assessments designed exclusively at a unit level can lead to a siloed approach, leading to shallow learning (Tomas & Jessop, 2018 Links to an external site.; Whitfield & Harvey, 2019 Links to an external site.) and difficulties in supporting the development of each individual student. A focus on the outcomes at the disciplinary level such as the majors in liberal studies degrees increases their coherence, and the skill and knowledge development in the contributing units (Jessop & Tomas, 2017 Links to an external site.).
• Assurance of learning for all students: by focussing on validating PLOs through cycles of introduction, development and assurance, it is much easier to demonstrate that these are met for each student whatever their route through the degree. We award degrees, after all, on the basis of attainment of the PLOs (Charlton & Newsham-West, 2022 Links to an external site.).
• Assurance of learning and academic integrity: as discussed below, advances in technologies such as generative AI make validating achievement increasingly difficult, time consuming, stressful and intrusive. It is pedagogically and practically better to purposefully plan, oversee and conduct secured assessments at selected points in the program.
• Increased utility of feedback: with PLOs introduced, developed and assured through selected points in their studies, students can use feedback and feedforward to improve on subsequent tasks (Boud & Molloy, 2013 Links to an external site.).
• Coherent development and support for students: by focussing on the stages of their program, the learning and support for each student can be designed to be a coherent journey (Boud & Molloy, 2012 Links to an external site.; Jessop, Hakim & Gibbs, 2013 Links to an external site.). This could include deployment of diagnostic assessment of mathematics or language ability at the start of the program and the selection of suitable progression points such as the end of first and second year. In turn, assessment and feedback design can purposefully develop students into self-regulated learners.
• Volume of assessment and workload: designing and planning assessment which focusses on the program-level learning outcomes can reduce the number of required assessments and assessment load:
  • As discussed below, the significantly increased difficulty in securing assessment in an age where ubiquitous generative AI technologies can mimic humans and traditional assessment outputs means that such assessment will become even more time consuming and needs to be designed more deliberately and deployed more selectively.
  • Planning assessment at the program level reduces stress, and deadline and marking bottlenecks. Analysis of assessment load for students with typical enrolment patterns in our degrees shows just how the assessment becomes congested when it is planned only at the unit level.
  • By reducing siloing of learning and assessments in units of study, there is less likelihood of repetition as reported by students.
  • By concentrating supervision into lane 1 assessments, there is the possibility of reduced marking loads and academic integrity administration.

As Harvey, Rand-Weaver & Tree (2017) Links to an external site. put it:

A program-level approach to assessment at Brunel University reduced the “summative assessment burden by 2/3

 

The role of program-level design in assuring learning

The assessment principles in our Coursework Policy follow the TEQSA ‘Assessment reform for the age of artificial intelligence’ Links to an external site. and the Group of Eight principles on the use of generative artificial intelligence in referring to program-level assessment. These all in turn follow from the Higher Education Standards Framework Links to an external site. and from accrediting bodies that require demonstration of learning across a course.

Assessment principle 5 in the Coursework Policy states that “Assessment practices must be integrated into program design”. This principle requires that:

1. assessment and feedback are integrated to support learning across units of study, courses, course components and year groups;
2. assessment and feedback are designed to support each student’s development of knowledge, skills and qualities from enrolment to graduation;
3. students’ learning attainment can be validated across their program at relevant progression points and before graduation;

Program-level assessment design thus requires alignment of assessment across units (a), with deliberate consideration of individual development and validation at relevant points in the course as well as before graduation. Because of the effect of generative AI on many of our existing assessments, principle 6 also requires this is performed in a “trustworthy way” with “supervised assessments [that] are designed to assure learning in a program”.

“Program” here includes both courses and their components, such as majors and specialisations. The requirements of principle 5, though, mean that validation at regular points must occur. In degrees like the BSc, BLAS, BA and BCom, these points could mean once a student has completed the degree core, the first year or the second year. The points at which it is most useful to validate learning, through lane 1 assessment, are best decided by the course and component coordinators.