What You’ll Build
A complete RAG system with:- A customizable knowledge base containing your documents
- Vector embedding and intelligent retrieval capabilities
- An interactive chat interface for querying your documents
- Automated testing for continuous quality assurance
- Analytics for performance monitoring
Knowledges functions as a RAG-as-a-Service offering, creating a collaborative environment where Data Scientists, Data Engineers, Developers, and Business Teams can efficiently work together. The platform features a unique methodology called Gen.AI Test-Driven Building, ensuring everyone operates from a single source of truth.
Prerequisites
Before starting this tutorial, make sure you have:- Access to the Knowledges product (self-hosted or Cloud version)
- Basic understanding of prompting concepts
- Documents to integrate (web pages, PDFs, Word documents, PowerPoint presentations, etc.)
Step 1: Understanding the Knowledges Interface
Let’s familiarize ourselves with the Knowledges product’s interface and main components:1
Access Knowledges
Log in to your Prisme.ai account and navigate to the Knowledges product
2
Explore the Navigation
The interface consists of several key sections:
- Agent: Configure AI models, select embedding models, and set parameters
- Test: Configure and manage automated tests for your AI
- Analytics: Monitor KPIs like token usage, generated answers, and daily usage
- Help: Access embedded product documentation
Step 2: Creating Your RAG Project
Now, let’s set up your first RAG project in Knowledges:1
Access Your Dashboard
Navigate to the Knowledges dashboard
2
Create a New Agent
Click the “Create Agent” button to start building your RAG agent
3
Configure Project Settings
Enter a name and description for your project that reflects its purpose
Step 3: Understanding the RAG Architecture
Knowledges uses a Retriever-Augmented Generation architecture that combines document retrieval with language generation: This architecture allows your agent to:- Process and store documents as vector embeddings
- Retrieve relevant document chunks based on user queries
- Generate contextually accurate responses using LLMs
- Provide sources and citations for transparency
Knowledges is designed to be model-agnostic, ensuring compatibility across various LLMs for both embedding and generative tasks. This flexibility allows you to switch models as needed without rebuilding your entire knowledge base.
Step 4: Integrating Documents
Let’s add documents to your knowledge base:1
Access Document Integration
In your project, navigate to the document integration section
2
Upload Documents
Click the “Add Document” button and upload files from your computer, or provide URLs to web content
3
Configure Document Processing
Adjust the document processing settings:
- Text Splitter: Choose between static (fixed-size chunks) or dynamic (content-aware) segmentation
- Chunk Size: Set the optimal size for document segments
- Overlap: Configure how much content overlaps between chunks to maintain context
4
Process Documents
Start the document processing, which includes:
- Document loading and parsing
- Text extraction and cleaning
- Segmentation into chunks
- Vector embedding generation
- Metadata extraction
Step 5: Testing Your RAG Agent
Now that your documents are integrated, you can start querying your knowledge base:1
Access the Chat Interface
Click on the “Chat” button to access the interactive interface
2
Ask Questions
Enter questions related to your documents to see how the AI responds
3
Review Sources and Chunks
Examine the sources and chunks used to generate each response
Knowledges provides full transparency by showing which document chunks were used to generate each response. This feature allows you to verify the AI’s reasoning and ensure accuracy.
Step 6: Implementing Test-Driven Building
A unique aspect of Knowledges is its support for Test-Driven Building (TDB), which ensures your RAG agent performs consistently over time:1
Access the Testing Module
Navigate to the “Test” section in your project
2
Create a Test Suite
Click “Add Test” to create a new test suite for your RAG agent
3
Define Test Cases
Add test cases with:
- Questions: Queries that users might ask your agent
- Expected Answers: The correct information that should be provided
4
Configure Test Settings
Set how often tests should run (daily, weekly, etc.) and whether they should trigger automatically
5
Run Tests
Execute the tests to evaluate your agent’s performance
6
Review Results
Analyze the test results, which include:
- Prompt and Context: The exact prompt and context used for generating responses
- Generated Response: The AI’s actual response
- Response Evaluation: Rating of response quality (poor, correct, good)
- Context Evaluation: Rating of context relevance and accuracy
Business experts can manually assess responses, context, and instances of hallucination. We recommend running tests after each modification to the model, prompt, AI parameters, chunk size, image inclusion settings, self-query configuration, and other adjustments.
Step 7: Advanced Customization Options
After setting up your basic RAG agent, you can explore advanced customization options:- Using Builder Product
- Using External Code
1
Access the Builder
Navigate to the Builder product on the Prisme.ai Platform
2
Implement Custom Logic
Create automations with Python or NodeJS using frameworks like LlamaIndex or Langchain
3
Connect to Knowledges
Copy the automation URL and go to Advanced > API & Webhook in Knowledges
4
Enable Webhook
Enter your automation URL and subscribe to events like questions asked, documents added/deleted/updated, and test runs
Step 8: Handling PII and Safety Controls
For organizations dealing with sensitive information, Knowledges offers options for privacy and safety:1
Implement PII Controls
Use the Webhook functionality to detect and handle personally identifiable information
2
Configure Data Anonymization
Decide whether to anonymize personal data based on your use case
3
Add Custom Safety Controls
Integrate specific safety measures tailored to your organization’s compliance requirements
Step 9: Monitoring and Maintenance
Keep your RAG agent performing optimally with regular monitoring:1
Review Analytics
Check the Analytics section to monitor usage patterns, token consumption, and response quality
2
Set Up Alerts
Use the Webhook feature with Builder to implement alerts on platforms like Slack, Teams, or Jira
3
Regular Maintenance
Update your knowledge base as new documents become available or existing information changes
4
Performance Optimization
Based on analytics and test results, adjust settings to improve response quality and efficiency
Best Practices for High-Performance RAG
To maximize the effectiveness of your RAG agent, consider these recommendations:Knowledge Architecture
Design a well-organized structure for easy access and interpretation by AI systems
Data Quality
Ensure data is accurate, well-formatted, and relevant—this accounts for ~80% of success
Hybrid Technologies
Consider combining NLU for tag extraction with LLMs for generation to improve efficiency
Continuous Optimization
Regularly monitor response times, accuracy, and retrieval effectiveness
Wrapping Up
By following this tutorial, you’ve created a powerful RAG agent that can access and leverage your organization’s knowledge without requiring any coding. Your AI can now provide accurate, contextual responses based on your documents, with full transparency about its sources and reasoning. The Test-Driven Building approach ensures your agent maintains high performance over time, while the flexible architecture allows you to adapt and extend its capabilities as your needs evolve.Next Steps
Document Classification
Learn how to automatically classify and organize your documents with AI
AI Contact Routing
Create an intelligent contact form that routes inquiries to the right department
Webhook Integration
Build advanced integrations between your RAG agent and other systems
Website to RAG
Turn website content into a comprehensive knowledge base for your RAG agent