Insights

I was sensoring 50 exams with AI help.

In the spring of 2023 I used AI as a tool to evaluate approx. 50 student tasks on 30-50 pages each. That's how I did it.

Karl Philip Lund and AI

In the spring of 2023 I used AI as a tool when I evaluated approx. 50 student tasks on 30-60 pages per page. task.

Other sensors indicated in advance that they spent from 30-60 minutes evaluating a task or up to 50 hours in total on all the tasks.

Using AI, I estimate that I can reduce the time of the evaluation per problem to average around 15 minutes. The bad tasks took less time. The good tasks I dedicated a lot longer! I would actually claim that the use of AI in the assessment process has increased the quality and led to a more fair assessment for students.

Here's the process I followed:

A. Find the right AI tool B. Define basic quality signals and assess the whole C. Ask the document questions using AI D. Manual assessment

Here's the story of how I went forward:

A. Find correct AI tool

I spent a lot of time finding the right AI tools to censor student tasks.

It shows up new apps every week, and it was hard to find a good alternative. Finally, I found Humata, a ChatGPT for documents. The service is developed by a former Stanford student and used by educational institutions worldwide. Humata claims that using the tool can make research and learn 100x faster! Humata privacy and document data are not used to train models. All data sent to/from the model is deleted after 30 days.

In Humata, I can upload documents and chat with the documents. Although it took time to learn how to use the tool effectively, it's definitely worth it when I can evaluate a problem of 10-15 minutes, compared to a minimum of 30-60 minutes per problem manually!

B. Define basic quality signals and assess the overall

Before I started using AI in the censorship process, I was sent a task that several sensors had considered independently of each other to be A. I analysed the A-task to identify the relevant quality signals. The idea was that I could use these quality signals to evaluate the remaining problems. (I thought, at the point, that I could give A's to all of the problems that looked like the A's problem.)

During my preparatory work, I benefited from my expertise and experience in search engine optimisation (SEO). The principles of good search engine optimisation can be used as a starting point for assessing the quality of any document, including a student submission.

A well-structured and well-organised document with relevant headings, relevant keywords and other quality signals makes it easier for the reader to understand its content - thereby increasing the chance of the submission being given a higher character.

Just as a website should be optimised to rank high in search engines, a student submission should be optimised to communicate clearly and effectively with the reader (the teacher or sensor).

Basic quality signals in the A task:

1. Title: Describes the title of the document content? 2. Table of Contents: Does the index provide a good overview of the contents of the document? Is it too short or too long? Three. Introduction/summary: Does the task contain a short introduction that gives the reader a short introduction to what the task is about? 4. Headings: Does the document contain good descriptive headings that make it easy for the reader to scan the document? Is the document logically structured? 5. Figures/pictures: Does the document contain visual elements that make reading the document easier? Are the visual elements unique (developed) and well-designed throughout the document? Are there any relevant captions describing the contents of the figures? 6. Tables: Are there good tables that are easy to read? Are the clearly defined column headers? Seven. Formatting: Are the paragraphs of the document formatted and organised in a way that makes reading easy? Eight. Spell Error: Student submissions should not contain spelling errors. Spell errors are easy to avoid by using spellchecking and getting a person to read through and correct spelling errors Nine. The source references: Does the document refer to the syllabus and other credible sources? (e.g. A source containing "gclid" indicates that the student has referred to a paid Google ad. Paid advertisements are not usually credible sources)

When you scan through a long document, you get a quick impression of quality by considering the content against the quality signals.

It is fully possible to manipulate a sensor by adapting the task to satisfy known quality signals. I compare this manipulation (or influence) with the way companies work to improve the location in Google. If all students know about these signals, it will raise the quality of the submitted tasks and save sensors worldwide too much frustration.

I also recommend everyone to get acquainted with Pyramid's Principle which is basically also about good structure and communication.

So to the use of AI!

C. Ask the document questions using AI

After defining basic quality signals and evaluating the whole of the documents, the way went on to "chat" with the documents.

In the beginning, I hoped that the right AI tool would do the whole job for me, i.e. evaluate the problems and give me grades, for example "based on the assessment criteria and the task set, what character should I give the task?"

I was thoroughly wrong!

It's not that simple.

I asked Humata the reasons for help writing good prompts:

"Hey. I'm testing out Humata. You have uploaded 48 reports written by students. I ́m thinking to use Humata to water the quality of the reports. Do you have any suctions for the prompts that I can ask in order to get a quick overview of the quality of the reports?"

I got the following response:

"Hi Karl, this is a novel and creative use case. In don't have any suctions for the evaluation to bulk in the moment. Although, it would be great to water the papers on a one-by-one basis. If there is a specific criterion for evaluation you can ask it that. For example, "Does this paper expressin X in relation to Y?"

For instance, do the student discuss [X]'s role in the formation of [Y] and give reasons to support their conclusion?"

I then asked the author the following question:

"I would like to do the following: Upload a student submission that is considered to be an "A" submission, high quality based on the grading criteria by human evaluators

The I would like to use this two degree other likes.

Is this a potential use case for Humata?"

I got the following response

"Hi Karl, yes, this is a potential and very creative use case for the multi-document analysis mode."

Meg:

"Can you suck a few prompts that could be used to test this?"

Answer:

"Since this is such a new use case I don't have any partial prompts that come to mind. If you have criteria that you used to deem the paper an "A", the den you can turn such criteria into questions in which to ask other documents. That is the first thing to come to mind.

Another experimental strategy is to ask Humata to generate the prompt given your critieria. Although I would third carefully with this direction."

I tested many samples before the tip from the Humata Gründer led me in the right direction.

I read the assessment criteria for the exam and then tried to convert the key criteria into sensible promps.

This was an a-ha experience for me.

I tested various samples.

e.g. criterion:

"Use the curriculum and what you've learned throughout your vacation"

I turned to:

"Has the student used the following syllabus: Iversen, Aalton, Hanlon, Furu"

This prompt should be relatively easy to answer and I was surprised when Humata did not respond correctly. When I asked Humata about the reason, I was told that the AI model should be able to respond exactly to the prompt, but that the system is being developed and that they would improve the model based on my input.

I then experienced Humata working well to extract relevant extracts from the documents based on my promts.

In fact, I felt that the better the task was, the better the Humata managed to answer! For the Humata founder, this was not a surprise:

"Better submissions will perform better with Humata because there is more context from the writing to pull from."

I was on the lookout! I wrote 7 prompts that referred to different parts of the submissions. For each submission I used the same prompts. I systematically reviewed each problem, read through the texts and thus got a good overview of the submissions. The method also made it easier to compare documents with each other!

D. Manual evaluation

After assessing the fullness of the documents and using Humata to assess different professional parts of the tasks, the way went on to make a manual assessment of a limited part of the text. Reading through a selection of text quickly gives you an impression of the style of the text in the rest of the document. In this section, it is about assessing the general flow in the language of the student.

Next Step

AI is undoubtedly exciting when it comes to student tasks assessment. With the right AI tool and a set of basic quality signals, you can quickly evaluate the quality of documents using AI to extract relevant extracts.

Combining AI and manual assessment can reduce the time and effort required to correct a large number of tasks, making it more effective for both students and teachers. So let's keep exploring how AI can be used in education to maximize the benefits while minimising the constraints.

Questions and answers

Question and Answer

Have students agreed to share their bachelor's problems?

"I don't see any particular privacy issues here. As regards copyright, the educational institution must be able to take copies necessary for censorship without consent. Ref also study regulations." https://twitter.com/VSondersrod/status/1673991917749645318 Vebjørn Søndersred Head IPR avd Lawyer Ræder, jus, GDPR, copyright civil process trademark marketing privacy

An examination answer is clearly a work of spirit. So, as you agreed to divide, under certain conditions. Is it then OK to send it to an unconsent third party?

"I don't see any particular privacy issues here. As regards copyright, the educational institution must be able to take copies necessary for censorship without consent. Ref also study regulations." https://twitter.com/VSondersrod/status/1673991917749645318 Vebjørn Søndersred Head IPR avd Lawyer Ræder, jus, GDPR, copyright civil process trademark marketing privacy

How does Google assess documents and why is that relevant?

Google's algorithms evaluate millions of documents (web pages, articles, etc.) every day. High quality content ranks high in search results. The principles behind how Google evaluates documents can be transferred to an assessment of a school task. Read more about how Google assesses content https://developers.google.com/search/docs/fundamentals/creating-helpful-content

Newspaper articles on this matter:

Disclaimer: When I started the assessment, I decided not to care about the students using AI to write the problems. I also understand that using AI to sensationalize tasks can be controversial, but I actually think that the quality of the evaluation is getting better!

PS. This article is written by me, but I have used ChatGPT to improve some sentences;)

What next?

Let's find the move that matters most for your online store.

Book an informal chat