Deploying and evaluating a conversational agent using LLMs for academic library reference
Megan Fitzgibbons·Aviva Majerczyk·Francisco Berrizbeitia·Joshua Chalifour·Olivier Charbonneau·Yara Stouhi
Purpose This study has two aims. First, we sought to implement a RAG-based GenAI system capable of answering reference questions. Second, we aimed to develop an evaluation protocol to assess the chatbot by means of comparing implementations that use three different LLMs. An evaluation rubric was piloted to gauge its viability as an assessment tool. Design/methodology/approach The RAG-based chatbot uses a two-step approach. First, in response to a query, the system retrieves relevant documents fr