Flui was at Yonder’s 2024 hackathon event. We worked with one of the hackathon teams to develop and evaluate the prototype of the Flui Wiki Bot, a tool aimed at improving our internal knowledge management capabilities by providing semantic search and accurate question answering. This tool addresses the challenge of information fragmentation by leveraging cutting-edge open source Large Language Models (LLMs), Embedding models, and Vector Databases as part of a Retrieval-Augmented Generation (RAG) application.
Context and Objectives
The project’s core objective was to create a Wiki Bot by implementing an effective RAG pipeline using open source tools while running everything within our own secure cloud environment where we can limit access to ensure the protection and integrity of our confidential/proprietary data.
The Wiki Bot was designed to handle natural language queries using a primary LLM which can be selected from available models. The process:
The following diagrams provide a good overview of the most basic RAG pipeline architecture:
Image 1: RAG Highest Level Architecture
Image 2: RAG Less-Abstract Overview
Technical Overview
Image 3: SageMaker Studio
Experience at the Hackathon
The hackathon environment was intense and exhilarating. As part of the SageMakers team, we tackled complex tasks such as configuring AWS SageMaker, embedding documents into a Vector DB, and integrating token streaming. The team skilfully adapted to the quirks of the newly released Llama 3 LLM to successfully implement a RAG pipeline crucial for the Wiki Bot’s functionality. The Yonder office was abuzz with energy, and teams experienced many highs and lows in learning new skills and unlearning old misconceptions.
We encountered and overcame several challenges; including devising a workaround for a release day bug in the Llama-3 model that caused it to endlessly ramble to fill the context window.
Reflections on the Hackathon
The hackathon was an invaluable learning experience, expanding our technical expertise and problem-solving skills. The rapid development of a functional prototype under an extremely tight deadline highlighted the importance of agility and collaboration. Also, there were loads of delicious Romanian food, unlimited energy drinks, and coffee (or “holy water” as it’s known at Yonder).
Expanded Scope
Once we met our initial prototype scope, we added some additional functionality. Inspired by how capable Meta’s open source Llama-3 was, we introduced a tab to our prototype enabling colleagues to use an AI Assistant and a separate tab for open source LLMs fine-tuned for natural language to SQL tasks, enhancing support for client queries and business intelligence solutions. The intention was to add functionality which could provide immediate benefit to colleagues at Flui, given the amount of time it would likely take to further refine the RAG pipeline would exceed the time we had left at the hackathon.
Value Brought Back to Flui
The Flui Wiki Bot prototype is fully owned by Flui Technologies and is ready for further development and deployment. It will be utilised by colleagues as part of a limited trial to assess its value. While the RAG pipeline needs more investment and development to extend beyond a single document, the newly added features such as the AI Assistant and the SQL DB Bot are ready to provide immediate value. Yonder has agreed to package the prototype into Flui’s GitHub repository to facilitate redeployment, and all the Jupyter notebooks are saved locally on the Flui OneDrive.
Further development of this prototype will not only enhance our ability to manage and utilise internal knowledge but also demonstrate Flui’s commitment to data security and technological innovation.
When deployed, the AI Assistant and SQL BI Bot could provide immediate benefits in areas governed and prohibited by our corporate policy regarding using confidential/proprietary information with consumer AI tools like ChatGPT.
Future Directions and Improvements
Short-term goals include enhancing the embedding and retrieval processes by experimenting with different models and chunking approaches to optimise the quality of responses. Long-term ambitions focus on implementing caching mechanisms for faster response times and ensuring scalability through AWS Elastic Load Balancing.
Closing Thoughts
This hackathon not only demonstrated our team’s dedication and skill in developing sophisticated systems within stringent timelines but also set the stage for ongoing enhancements and innovations at Flui Technologies. With these developments, we aim to further leverage cutting-edge AI solutions to transform our business operations and decision-making processes. As always, the true test of our prototype will be how it performs and how useful colleagues find it in practice.
– Jason Jackson, Product Owner, Flui R&D
Request further information to see how our solutions can support your business.