LLM detection software

The aim is to provide markers with a simple, automatically generated report indicating the likelihood that unusual processes were involved in producing the work. Outputs include a JSON report (a detailed log of evidence, not intended for initial review by markers) and a .txt summary for markers and module leaders.

The method is to automate detection of unusual editing behavior in Word submissions using embedded metadata. The scope applies to single student .docx final submission or zip files containing many .docx files. The method is to extract metadata from each document, run heuristic checks, and generate a summary conclusion.

Student instructions: Students must draft their work in Microsoft Word. All drafts and the final document must be kept in a single folder. At submission, students must zip all drafts and upload the zip file, while also uploading the final document unchanged. Students must not use “Save As” or create new documents externally in a way that overwrites metadata.

Automated processing pipeline: First, unzip the student’s submission. Then, for each .docx file, extract metadata from docProps/core.xml and docProps/app.xml. Next, evaluate the data using heuristic checks. Aggregate the findings into a JSON report (optional). Generate a plain text conclusion file containing a single line: “no evidence of unusual behaviour” or “evidence of unusual behaviour”. Finally, append the .txt file to the submission archive or post results to Canvas.

Metadata fields: The system uses Created, Modified, TotalEditingTime, RevisionNumber, and LastModifiedBy. These fields are used to infer editing duration, revision activity, and authorship consistency.

Output: The JSON report is a full structured record intended for review by integrity officers if needed. The TXT summary is a single-line verdict for markers or LMS integration. Example outputs include “no evidence of unusual behaviour” and “evidence of unusual behaviour”.

Integration: In Canvas, the summary.txt or JSON file can be attached to the submission record for staff visibility. Alternatively, the summary.txt can be kept in the same folder as the student essay for audit purposes. The final document should remain uploaded separately outside the zip file so markers can use the native SpeedGrader interface without disruption, allowing the metadata check to run without any input from the marker.

Limitations: Metadata can be altered using third-party tools, so this system is heuristic only. It does not perform text-level analysis or AI content detection. It is intended for triage rather than disciplinary proof.

Implementation status: Fully deployable today with minimal requirements, needing only a quick script and no administrative privileges.

Future changes planned

At the moment, the software simply uses the metadata as provided by Word (and .odt standard). This provides a basic foundation as to number of saves made, editing time, author's name, etc. This is not, nor is intended to be proof of plagiarism, or LLM use. Instead, it is designed to act as a triage system; to funnel students with unusual writing process metadata to be looked at more closely, and perhaps undergo a misconduct viva. The formal viva process with the expertise of the lecturer must be the final decision process; software cannot decide whether plagiarism took place, but it can help markers to reduce their workload. However, there is a further step that can be taken to improve this workflow.

When a document is written in Word, the content of the document itself is written to document.xml, which is an xml-coded plain text file, preserving several features not apparent in the final document as viewed. For example the text: 'This is a sentence':-

      <w:p>
        <w:r><w:t>This is a sentence.</w:t></w:r>
      </w:p>
      

Where the sentence is edited normally- 'This is a longer sentence', Word will usually rewrite the entire paragraph:-

      <w:p>
        <w:r><w:t>This is a longer sentence.</w:t></w:r>
      </w:p>
      

However, if the text is copy/pasted from an outside source, the end result will often be considerably more messy:-

      <w:r><w:t>This is a </w:t></w:r>
      <w:r><w:t>longer </w:t></w:r>
      <w:r><w:t>sentence.</w:t></w:r>
      

or even:-

      <w:r><w:t xml:space="preserve">This is a </w:t></w:r>
      <w:r><w:rPr><w:lang w:val="en-US"/></w:rPr><w:t>longer </w:t></w:r>
      <w:r><w:t>sentence.</w:t></w:r>
      

or even:-

      <w:r><w:t>This is a </w:t></w:r>
      <w:r>
        <w:rPr><w:b/></w:rPr>
        <w:t>longer</w:t>
      </w:r>
      <w:r><w:t> sentence.</w:t></w:r>