OpenAI has published a study examining the effect of using ChatGPT study mode on learning outcomes. They looked at two subjects that students were studying: Neuroscience and Microeconomics, and the results were mixed, with only students in Microeconomics showing gains over traditional methods.
What is particularly interesting from this paper, however, is OpenAI’s claim that existing methods of measuring learning gain are insufficient and therefore they have invented a new measure that they called the Learning Outcomes Measurement Suite to measure learning gain when using AI. I may just be a bit of a cynic, but if you have to invent a new measure to show that your product works, then I am going to have questions about the validity of that new measure and the intentions behind it.