Yee-King, Matthew and Fiorucci, Andrea. 2025. 'Deploying language model-based assessment support technology in a computer science degree: How do the academics feel about it?'. In: IEEE Global Engineering Education Conference (IEEE EDUCON). Queen Mary University of London, United Kingdom 22 - 25 April 2025. [Conference or Workshop Item] (In Press)

No full text available

Text IEEE_educon_25_LLMs_in_BScCS.pdf - Accepted Version Permissions: Administrator Access Only until 22 April 2025. Download (269kB)

Abstract or Description

We present two contrasting case studies wherein we used large language model (LLM) technology to support critical elements of our work in the context of a large scale online undergraduate computer science degree. Firstly we used semantic embeddings to identify student-student collusion in exam answers. Secondly we used LLMs to generate starter drafts for exam question papers. We gathered academic staff responses to the two systems through structured interviews. We describe and use a novel, LLM-powered inductive thematic analysis methodology to tag and identify themes in the interviews. All analysis was carried out on locally hosted language models.

We identified 26 themes, some shared across the two systems, others unique. The academics were largely comfortable with the use of LLM technology in assessment, the exam generator system helped to kick-start exam writing and the collusion detection tool found otherwise invisible cases. Academics emphasised the need for human oversight of such systems, but were prepared to use them as they perceived that they improved the efficiency of exam processes.

Item Type:	Conference or Workshop Item (Paper)
Additional Information:	“© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.”
Keywords:	LLM, education technology, thematic analysis
Related URLs:	https://2025.ieee-educon.org/ https://github.com/yeeking/llm-thematic-... https://ieeexplore.ieee.org/xpl/conhome/...
Departments, Centres and Research Units:	Computing
Dates:	Date Event 16 December 2024 Accepted 22 April 2025 Published
Event Location:	Queen Mary University of London, United Kingdom
Date range:	22 - 25 April 2025
Item ID:	38206
Date Deposited:	23 Jan 2025 12:08
Last Modified:	23 Jan 2025 16:02
URI:	https://research.gold.ac.uk/id/eprint/38206

View statistics for this item...

Edit Record (login required)