Introduction
We work with permission, which is our bread and butter. LLMs are not. But recently we have been thinking a lot about how to build secure LLM chatbots, taking into account permission considerations. This post will not go in-depth on how to build one using Oso Cloud and filtering the list, but rather as an introductory post to LLM chatbots, how they work, and best practices for building them.
What are LLM chatbots?
LLMs, or large language models, are advanced AI models trained on large amounts of data to understand and produce human-like language by predicting the most likely word sequences based on text. These models support a wide range of applications, including content creation, text completion, translation, sentiment analysis, and code generation.
One of the most common uses of LLMs is in chatbots – artificial intelligence-based tools that simulate human conversation. Chatbots are widely used for customer support, information retrieval, and task automation, and interact with users via text or voice.
LLMs and chatbots are inherently interconnected, as LLMs serve as the underlying technology – natural language understanding and generation (NLU) capabilities – that powers many modern chatbots (such as ChatGPT, Claude, Gemini, etc.). When a user enters a query, the chatbot sends it to the LLM, which processes the text, takes into account context using attention mechanisms, retrieves relevant knowledge (potentially augmented by retrieval-augmented generation (RAG) or external APIs), and produces an appropriate response.
This architecture allows chatbots to maintain conversation history and context across multiple exchanges. This enables them to understand complex queries, linguistic nuances, and provide accurate answers or perform tasks.
By leveraging LLMs, modern chatbots achieve greater fluency, coherence, and contextual awareness, making them highly effective for applications such as virtual assistants, interactive customer support, and personalized content generation.
Examples of LLM chatbots:
- ChatGPT (OpenAI): A general-purpose AI chatbot based on OpenAI's GPT architecture, designed for a variety of text-based tasks such as answering questions, generating content, and providing personalized assistance. Its core strength lies in its ability to handle a wide range of topics and produce human-like responses, using the transformer-based GPT model to understand and produce general-purpose language.
- Claude (Anthropic): An AI chatbot built by Anthropic with a focus on AI alignment and safety, designed to prioritize ethical considerations while providing helpful responses. Unlike other models, Claude emphasizes prudent behavior and alignment, ensuring that responses comply with safety guidelines and ethical standards.
- Google Gemini (formerly Bard): An AI chatbot integrated into the Google search ecosystem that uses Google’s advanced language models to help with contextual, real-time information. This chatbot stands out with its deep connection to Google search and the ability to retrieve information seamlessly within the search environment.
- Meta AI: An AI-powered chatbot built into Meta’s social media platforms, such as Facebook and WhatsApp. The service focuses on personalized responses, recommendations, and assistance tailored to the social and messaging context, leveraging Meta’s proprietary language models tuned for these platforms.
How do LLM chatbots work?
LLMs use deep learning techniques, with a particular focus on transformer architecture, to process and generate text. These models are trained on a wide variety of data sets, including books, websites, and other sources, allowing them to learn patterns, grammar, contextual relationships, and semantic structures within the language. At the core of their functionality is an attention mechanism that allows the model to focus on relevant parts of the input text and understand long-term dependencies and contextual nuances. This mechanism, combined with other advanced techniques such as Retrieval Augmentative Generation (RAG), allows LLMs to produce highly coherent and context-aware responses by dynamically paying attention to relevant information, both in the input and from external knowledge sources.
Attention
At the core of the converters is a mechanism called attention that allows the model to weigh the importance of different words in a sentence relative to each other. This enables the model to capture long-range dependencies and understand the context of words beyond their immediate neighbors. Rather than processing words individually, attention helps the model understand how each word interacts with other words in the broader sentence structure.
Training an LLM involves processing billions of parameters (weights) that are tuned over time to minimize prediction errors. The model is trained on large text datasets and learns to predict the next word in a sequence, improving its ability to produce coherent, contextually relevant text as it processes more data.
When generating text, the model uses the patterns it has learned to predict the most likely continuation of a particular phrase. Each potential next word is scored based on its relationship to previous words, and the model selects the word that is the most likely based on the learned context.
For example, in the sentence «The cat sat on …,» the attention mechanism helps the model prioritize «mat» over less important words like «bus» or «computer» because it recognizes that «mat» matches the context and meaningfully completes the sentence.
Thus, LLMs process text by encoding input through layers of attentional mechanisms, refining predictions through training, and generating responses by selecting the most likely next words based on the text they have learned.
Augmented generation recovery
RAG (Retrievable Augmented Generation) is a technique that combines information retrieval with generative models. In a RAG system, when the model receives a request, it first retrieves relevant information from an external source (such as a document or database) using a search mechanism. It then uses a generative model (usually based on a transformer architecture) to produce a response that incorporates both the original input and the retrieved information.
In RAG:
- BFinding: The model queries an external knowledge base or document collection to find relevant text.
- Production: A transformer-based model, such as GPT or BART, then produces a response based on the original input and the retrieved information.
While the transformer architecture feeds the RAG generative part, RAG itself refers to the process of augmenting the output with external, retrieved knowledge. This enhances the model’s ability to produce more informed, contextually accurate, and up-to-date responses, allowing it to utilize a wider range of information beyond its pre-trained knowledge.
For example, imagine a user asking an AI-powered assistant, “What are the latest developments in quantum computing?” Without real-time internet access, a generative model alone might struggle to provide an up-to-date answer. In a RAG system, however, the model can first search a relevant database of recent research papers or news articles about quantum computing. It then generates an answer that includes the latest findings from these sources, ensuring a more informed and accurate answer.
Therefore, by augmenting its generative process with external knowledge in real time, RAG improves the model's capability, making it more effective in answering specific or dynamic questions.
How do LLM chatbots learn?
Unsupervised learning
LLMs learn through unsupervised learning, where they are trained on large amounts of text data to predict the next word in a sequence. This training process involves feeding the model large, unlabeled datasets. Since there is no explicit human annotation, the model learns patterns, grammar, context, and relationships between words simply by recognizing patterns and structures in the text.
At the core of the learning process is gradient descent, an optimization algorithm that adjusts the model parameters (weights) over time to minimize prediction errors. The model starts with random weights and, as it processes each text input, predicts the next word based on the text. When the prediction is incorrect, the model updates its weights to reduce future errors. This iterative process is repeated billions of times on huge data sets, enabling the model to produce consistent, text-appropriate text.
During training, the model learns:
- NHow: Rules governing the structure of language.
- Semantics: The hidden meaning in words and phrases.
- Context: How words interact with each other in different situations, which allows the model to handle long-term dependencies and nuances.
A practical example of unsupervised learning is customer segmentation in marketing. Suppose a company has a large dataset of customer behavior, but no predefined categories such as «high-value customers» or «regular buyers.» Using k-means clustering, an unsupervised algorithm, the model can automatically group customers based on their purchasing behavior (e.g., high spenders, occasional buyers). These segments can be the basis for more targeted marketing strategies.
Fine-tuning under supervision
While LLMs are initially trained using unsupervised learning, they are often fine-tuned using supervised learning to improve their performance on specific tasks. In this step, the model is trained on a smaller, labeled dataset that provides the correct output (e.g., a classification label or answer to a question). This enables the model to learn more task-specific patterns and improve its accuracy for specific applications, such as translation or question answering.
For example, after performing unsupervised customer segmentation, an e-commerce company might want to classify customer reviews as positive, negative, or neutral to optimize email campaigns. The company could fine-tune a pre-trained model using a labeled dataset of customer reviews with sentiment labels. This fine-tuning allows the model to specialize in understanding sentiment and improve its ability to classify future reviews specific to the company’s products.
Learning transfer
LLMs also use transfer learning, where knowledge gained from one task is applied to other, often unrelated, tasks. This enables models to be effective in different scenarios without the need for explicit training for each specific task.
Building on the sentiment analysis example, suppose a company wants to expand its model’s capabilities to handle a wider range of customer questions, including customer service questions. Rather than starting from scratch, the company can use transfer learning to apply the sentiment analysis model’s understanding of customer sentiment to a new task, such as answering customer questions.
For example, a company could adapt the tuned sentiment model for a customer service chatbot. The chatbot can now understand the emotions of customer feedback and respond intelligently based on those emotions. If a customer expresses frustration with a late delivery, the model uses its knowledge of the emotion and context to create a compassionate and helpful response. This approach significantly increases the chatbot’s ability to help customers without requiring extensive new training.
Building an Internal Chatbot: Why and How
Why: Why should we build an internal chatbot?
- Shorten sales cycles: An internal chatbot can automate the generation of personalized outbound emails by retrieving customer data from CRM systems, helping the sales team produce relevant text messages faster.
- Improved customer support: This chatbot can integrate with existing knowledge bases and support ticket systems to provide fast and accurate answers to support teams, reduce response times, and improve customer satisfaction.
- Training new employees: Chatbots can automate the onboarding and hiring process by providing instant access to internal documents, policies, and frequently asked questions, helping new employees learn company processes effectively.
- Identifying documentation gaps:
This chatbot can identify unanswered questions or requests for information that are not covered in current documentation and help highlight areas that need updating.
How-to: How to build a chatbot?
- Defining scope and role-based access:
Specify the purpose of the chatbot – whether it’s for sales, support, or training – and implement role-based access control (RBAC) to ensure that users only receive information relevant to their roles. For example, a support representative might have access to detailed internal resources, while a general employee could access more basic information. - Technology selection and integration:
Use transformer-based models (e.g. GPT) for natural language understanding and integrate them with internal APIs, databases, and knowledge bases. Use RAG to get relevant, real-time information for answers. - Implementing authorization mechanisms:
Integrate your chatbot with a permissions service (like Oso) to enforce permissions based on user roles and attributes. This ensures that sensitive company data, such as sales figures or employee performance, is only accessible to authorized personnel. - Training and fine tuning:
Train the chatbot using domain-specific data and integrate internal documents and knowledge bases. Fine-tune the model using supervised learning to ensure accurate responses in context, while ensuring the model knows which information to keep hidden based on user permission. - Testing and monitoring:
Regularly test the bot with internal users to identify performance issues and unauthorized access attempts. Continually update the bot's knowledge base and adjust the model to reflect changes in company processes, policies, and security protocols.
Four Ways to Build an LLM Chatbot: Pros and Cons
Pre-trained models and fine-tuning
A common approach to building an LLM-based chatbot is to use pre-trained models such as GPT or BERT and then fine-tune them on domain-specific datasets. Pre-trained models are trained on large corpora, enabling them to process natural language effectively. Fine-tuning these models involves training them on smaller, more specific datasets for a particular use case, such as customer support or internal documentation. This approach allows the chatbot to leverage the extensive linguistic capabilities of the pre-trained model while still focusing on the features of the target domain.
Recovery Augmented Generation (RAG)
RAG combines the retrieval of relevant information from external sources with the generative capabilities of LLMs. In a RAG-based chatbot, when a user submits a query, the model first queries an external knowledge base or database for relevant information. The retrieved information is then passed to the generative model, which uses it to generate an answer. This approach improves the accuracy of answers by grounding them in real-time data, making it particularly useful for applications that require up-to-date information. The main advantage of RAG lies in its ability to generate contextually accurate answers without requiring extensive training on domain-specific datasets.
For example, if Oso’s internal chatbot uses a RAG-based approach to answering developer questions, it can query the latest internal documentation as well as GitHub repositories before generating an answer to ensure that developers receive the most up-to-date and accurate answers about Oso’s permission system. Integrating real-time data retrieval with the generative model enables more accurate and tailored answers, reducing the need for frequent updates or retraining of the chatbot.
However, implementing RAG systems can be complex, as they require integrating external data sources and managing retrieval processes. In addition, ensuring the quality and relevance of data is essential to maintaining the effectiveness of the system.
Law-based systems
Rule-based systems operate on a predefined set of rules, where a chatbot follows specific patterns or decision trees to generate responses. This approach is typically used for tasks with a defined scope, such as answering frequently asked questions or executing specific commands. Rule-based systems do not require large-scale training data, which makes them less resource-intensive than LLM-based systems. However, they are limited in handling unexpected user input or complex conversations, as they can only respond according to the rules initially defined. Over time, they may require maintenance to keep up with changing user needs or business requirements.
In Oso, an example of a rule-based chatbot might include predefined questions such as «Show me the scope of this migration project» or «Where in the code do we define the feature?» The chatbot responds based on these fixed rules. While this approach works well for well-defined questions, it cannot handle more complex or unexpected inputs and requires referral to a human agent or updating the ruleset.
Combined approaches
Hybrid approaches combine the strengths of LLMs, RAGs, and rule-based systems to overcome the limitations of each method. For example, a chatbot might use LLMs to process and understand natural language input, use RAGs to retrieve relevant information from an external source, and use rule-based logic for specific tasks or to execute structured workflows. This approach allows the chatbot to handle both dynamic, context-sensitive queries and tasks that require predefined, rigid responses. However, hybrid systems introduce complexities in design and maintenance, as they require integration across multiple components and ensure seamless interaction between different methods.
API-based solutions
API-based solutions enable a chatbot to interact with external systems to retrieve data, trigger actions, or perform specific functions. This approach is particularly useful for chatbots that need to access live data, perform transactions, or integrate with other software systems. For example, a chatbot can call an API to retrieve customer data or process a payment request. While API-based solutions extend the capabilities of a chatbot beyond simple conversational tasks, they also create dependencies on external services. These dependencies can create potential points of failure if the external APIs break, have limited access, or undergo changes that break compatibility with the chatbot. Additionally, using external APIs raises security and data privacy considerations, especially when dealing with sensitive or personal information.
Chatbot Security Risks and Best Practices
When building an LLM-based chatbot, it is essential to consider potential security risks that could compromise both the integrity of the chatbot and the privacy of sensitive data. One of the main risks is delusion, where the LLM produces false or misleading information. This can be dangerous, especially in high-stakes areas such as customer support or internal documentation, where incorrect advice can lead to confusion or even financial loss.
Permissions play a crucial role in securing a chatbot, especially when it comes to restricting access to sensitive information. Without proper access controls, there is a significant risk that unauthorized users may gain access to information they shouldn't have access to.
For example, if appropriate access restrictions are not applied, a chatbot could inadvertently reveal details of an employee’s personal performance. Similarly, if the chatbot lacks safeguards to prevent unauthorized queries, confidential information, such as a colleague’s family history, could be exposed.
Other potential horror stories could include:
- An employee who is looking for income information from other employees to compare with their own income.
- An employee who is searching for health insurance information but accidentally notices a coworker's medical condition, such as breast cancer, and asks about it.
- A team searching for a document outside of their department inadvertently gains access to sensitive company financial data.
- A lower-level employee who attempts to retrieve confidential project details from the senior team and gain access to strategic plans or confidential negotiations that they are not authorized to see.
- A user searching for information about a specific company policy but accidentally receives old or sensitive versions of the policy that are outdated or classified.
- A contractor who gains access to internal documents containing intellectual property (IP) or trade secrets that are not related to their temporary role.
Conclusion
As a result, building secure LLM chatbots requires a careful balance between advanced AI techniques and strong authentication mechanisms. By integrating augmented retrieval, leveraging pre-trained models, and implementing strong role-based access controls, businesses can ensure that their chatbots are not only efficient, but also secure, delivering personalized and contextually accurate interactions.
To learn more about how to build your LLM chatbot and secure it using authentication, join the O'Reilly webinar SuperStream: Retrieval-Augmented Generation in Production.
