Multi-modal AI Chatbot for Text and Image Interaction

Abstract

The past few years have seen a rise in chatbot applications, although the majority of them are limited to text-based interactions. In practice, users often use visual media, like screenshots or photographs as an effective means of presenting information. The paper provides a multi-modal chatbot that is able to handle textual inputs, analyze uploaded images, and create new images based on descriptive questions.

Gemini API was chosen as the foundation of the system because it has an intrinsic support of both textual and visual input, and because of its available developer documentation. Streamlit was used to construct the front-end interface, and it can be deployed quickly without involving hefty computing resources. The testing proved that the system reacted faithfully to usual text queries, generated meaningful interpretations of diverse uploaded pictures such as diagrams, screenshots and photographs, and generated visually coherent pictures of simple descriptive prompts.

The main goal was not to create a production-scale AI platform, but to investigate whether it was possible to integrate text chat, image analysis, and image generation in one, lightweight interface.