AI Chatbots and Student Learning: Insights from Johns Hopkins

Published Date : 23/10/2025 

A recent study by Johns Hopkins University investigates the role of AI chatbots in middle and high school classrooms, revealing both the strengths and challenges of integrating these tools. 

As students settle into the new school year, one question looms large: How will artificial intelligence tools like ChatGPT affect their learning? Seeking answers, a team from Johns Hopkins recently introduced a chatbot into a classroom of middle and high school students to act as a co-tutor and study the impact.

The pilot study included 22 students enrolled in the Johns Hopkins Center for Talented Youth's (CTY) online course Diagnosis: Be the Doctor. It involved two virtual classrooms; both were taught by the same instructor and organized similarly, except for one key difference: Students in one classroom had access to a large language model designed to act like a coach, asking Socratic-style questions as students worked through medical case studies.

“We wanted to use it as a potential way to supplement a teacher in the classroom, so it would be another opportunity for the students to learn,” said Kathryn Thompson, CTY's director of research and the study's lead author.

The research team was interested in the potential advantages the chatbot provided for students and teachers. They were inspired by the new innovations in generative artificial intelligence, one of them being these chatbots that can interact with students. Oftentimes, other researchers or authors will talk about the possible advantages chatbots can provide, like on-demand, personalized feedback to students, and how they lessen the load for teachers, so they don't have to spend as much time on things like grading or creating individualized lesson plans.

Their findings, recently published in the Journal of Advanced Academics, highlighted that successful AI use in classrooms depends on careful design, structured teacher guidance, and clear expectations for students. Study co-authors included Kimberley Chandler and Emily Delinski of the Center for Talented Youth; and Johns Hopkins Whiting School of Engineering computer scientists Daniel Khashabi, Candice Morgan, and Benjamin Van Durme.

The team found no significant differences in final assessment scores between the two groups of students. They also found that many students used the chatbot in unexpected ways—asking it for information rather than engaging in the back-and-forth coaching it was designed to facilitate. Study results highlight the importance of teaching students how to use AI tools effectively, suggesting that AI literacy will be essential if generative AI is to have a meaningful impact in education.

The project was conceived in 2023, when public use of AI was starting to surge. “We could see the industrialization of the technology, and we were eager to see how it was going to impact education,” said Khashabi, a study co-author who is an assistant professor in JHU's Department of Computer Science. “We connected with CTY because they're a pioneer in K-12 education and using technology in K-12 classrooms. Really the whole idea was, why don't we do the simplest thing, which is take a chatbot, give it some guardrails for safety, offer it to students, and see how they choose to use it? We didn't even know if they were going to use it or not.”

The team designed the study around CTY's Diagnosis: Be the Doctor online course because its learning objectives could be associated with critical, higher-order thinking skills, rather than students' ability to recall facts. The students learned various anatomy and physiology concepts in the course, which they applied while reviewing case studies and crafting diagnoses. Students were told they could use the chatbot—embedded directly within the course's online learning platform using a plug-in feature—for assistance throughout the course.

The chatbot was designed to provide on-demand, inquiry-based feedback as students submitted their diagnoses. For example, when a student deduced that a patient had leukemia, the chatbot—“Dr. Smith”—prompted the student to look more closely at a blood smear, to consider platelet counts, and to inspect the appearance of white blood cells in their assessment.

Some students used the chatbot as intended, engaging with it in a back-and-forth discussion about their patient assessment and why they arrived at a particular diagnosis. Others asked it for background information, for example, “What are the symptoms of polycythemia?” Occasionally, students simply asked it for the correct answer. The chatbot was designed to redirect those requests, encouraging students to review course materials and consider alternative diagnoses.

While the chatbot had to be fine-tuned along the way to work out some kinks in its intended interactions with the students, the researchers found that humans also needed more direct guidance on how to interact with the AI to yield greater learning outcomes. Additional instructor training may be one key in helping to guide students' use of a chatbot. According to the report, structured, hands-on professional development may be more effective than the self-paced AI training that was provided for the teacher in this study.

“Students also need training on effective ways to interact with large language model chatbots if we hope to improve student learning,” Thompson said. “Teaching students artificial intelligence basics, like what large language models are and their advantages and disadvantages, are important, but we believe we could see improvements by showing students useful ways to interact with the chatbot, how to critically evaluate its responses, and how it can make them a better and more reflective learner.” 

Frequently Asked Questions (FAQS): 

Q: What was the main goal of the Johns Hopkins study on AI chatbots in classrooms?

A: The main goal was to explore how AI chatbots could supplement teacher instruction and enhance student learning in middle and high school classrooms.

Q: What were the key findings of the study?

A: The study found that while there were no significant differences in final assessment scores between students with and without AI chatbot access, the chatbot's effectiveness depended on careful design, structured teacher guidance, and clear expectations for students.

Q: How did students use the AI chatbot in the study?

A: Students used the chatbot in various ways, including engaging in back-and-forth discussions, asking for background information, and sometimes requesting direct answers. The chatbot was designed to redirect requests for direct answers and encourage critical thinking.

Q: What were the challenges identified in the study?

A: Challenges included the need for better training for both students and teachers on how to effectively use AI chatbots, as well as the need to fine-tune the chatbot's interactions to better align with educational goals.

Q: What is the importance of AI literacy in education according to the study?

A: The study highlights the importance of teaching students about AI, including its basics, advantages, and disadvantages, to ensure they can use AI tools effectively and critically evaluate their responses. 

More Related Topics :