Recognizing Images and Extracting Useful Inferences by Asking Large Language Models Simple Questions

Laboratory exercises are an essential component of engineering education while the increasing trend towards distance learning presents unique challenges in replicating hands-on experiences. A number of AI-driven solutions have been proposed to facilitate remote laboratory exercises, however the emergence of Multimodal Large Language Models offers novel possibilities for visual recognition in remote settings. Vision AI, a subfield of artificial intelligence, enhances LLM capabilities by allowing them to process visual data through tasks like image recognition and segmentation, making it particularly relevant for use in educational applications. This work evaluates the integration of Vision AI into LLMs like OpenAI's GPT-4 and Anthropic's Claude 3.5 Sonnet, examining their ability to recognize and understand images from laboratory devices such as displays, gauges, and control panels. Our study focuses on the estimation of the performance of GPT-4 and Claude 3.5 Sonnet in laboratory-related image recognition tasks, with results indicating similar high text recognition accuracy (92% for GPT-4 and 91% for Claude 3.5). Despite these successes, challenges persist in spatial awareness and object identification, which are critical for accurate interpretation of complex lab environments. These findings highlight the potential of Vision AI to support remote laboratory exercises, improve accessibility for students in geographically distributed settings, or students with disabilities, and enhance interactive learning tools in STEM education. Future work will focus on refining these capabilities through custom LLM development, advanced prompt engineering, and multimodal approaches, aiming to create more versatile and effective educational technologies for remote and hybrid learning environments.