Retrieval-augmented generation (RAG) is the standard way to put an AI assistant on top of company documents. Instead of relying on whatever a large language model absorbed during training, a RAG system first searches your own content (policies, contracts, manuals, support articles) and then asks the model to answer using only the passages it found, with references. For a business leader, the questions that matter are what it is good at, where it fails, and what it takes to trust the answers.
What is RAG in AI, in plain terms?
RAG is an open-book exam for an AI model. The model is handed the relevant pages before it answers, instead of answering from memory.
The term comes from a 2020 research paper by Patrick Lewis and colleagues at Facebook AI Research. The idea has since become the default design for "ask our documents" tools, and it sits behind the grounded answers in many workplace assistants and help desk products.
It exists because language models have three limits that matter to a business. They do not know your internal information. Their knowledge stops at a training cutoff. And when they do not know something, they can still produce a fluent, confident answer. Retrieval addresses the first two directly and makes the third easier to catch.
How does retrieval-augmented generation work?
Two processes run: one prepares your documents in advance, and the other runs every time someone asks a question.
Preparing the documents (indexing):
- Connect the sources: SharePoint, Google Drive, Confluence, a help center, exports from a database.
- Extract the text, using OCR for scans, and keep metadata such as title, date, owner and who is allowed to see it.
- Split each document into chunks of a few paragraphs, keeping headings attached so each chunk makes sense alone.
- Turn each chunk into an embedding, a list of numbers representing its meaning, and store it in a searchable index such as pgvector in PostgreSQL, OpenSearch, Pinecone or Qdrant.
Answering a question:
- Identify the user and what they are allowed to see.
- Search with both keywords (good for part numbers, names and clause numbers) and embeddings (good for questions phrased differently from the source).
- Rerank the top results and keep the best handful.
- Send the question and those passages to the model, with instructions to answer only from them, cite them, and say so when the answer is not there.
- Show the answer with links to its sources, and log the exchange.
RAG vs fine-tuning: which does a business need?
For answering questions from company knowledge, RAG, in almost every case. Fine-tuning teaches a model a behavior or a style; it is a poor way to teach it facts that change.
| RAG | Fine-tuning | Long context (include everything) | |
|---|---|---|---|
| What changes | What the model reads when it answers | The model's internal weights | What the model reads when it answers |
| Best for | Facts that change and need sources and permissions | Consistent format, tone, classification, specialist phrasing | A handful of documents or a one-off analysis |
| Updating knowledge | Re-index the changed document | Train again | Include the new version |
| Citations | Built in | Not available | Possible |
| Per-user access control | Filter at search time | Not practical | Manual |
| Cost profile | Indexing, plus model usage per question | Training runs, plus hosting | High per-question cost with large inputs |
Context windows are now large enough that, for a small collection such as a 50-page staff handbook, you can skip retrieval and include the whole document with every question. That is a legitimate choice. Retrieval earns its keep once you have thousands of documents, need per-user permissions, or want to control the cost of each question.
Where does AI on company documents work well, and where does it fail?
It works well for questions whose answers sit in a few passages of text. It struggles with counting, aggregating and anything that needs judgment across a whole archive.
Good fits:
- staff questions about HR, IT and expense policies;
- support agents searching manuals, release notes and resolved tickets;
- drafting answers to tenders and security questionnaires from previously approved responses;
- finding the relevant clause in a specific contract;
- product and specification lookups for sales teams.
Poor fits:
- "How many contracts expire next quarter?" That is a database question. Extract the dates into structured data and use a report, which our data analytics team would build instead.
- Tables and figures in scanned PDFs, unless extraction is checked carefully.
- Contradictory sources, such as three versions of the travel policy with no clear current one.
- Questions that need a decision rather than a lookup.
If staff mostly need to find the right document rather than a written answer, better enterprise search may deliver most of the value: consistent titles, metadata and archived old versions, with no generated text and no risk of invented answers.
Does RAG stop AI hallucinations?
No. It reduces them and makes them checkable, because every answer can point to its source.
Errors still happen in three main ways: search misses the right passage and the model answers from general knowledge; the model blends two passages into something neither says; or it misreads a number, a date or a "not". The countermeasures are practical. Test the "I don't know" behavior as carefully as the right answers. Show citations prominently so users check them. Restrict high-stakes collections to approved, current documents. Keep a human in the loop for legal, medical and financial outputs.
How do you evaluate a RAG system?
Measure retrieval and answers separately, against a fixed set of questions, and re-run the set after every change.
- Have subject experts write 100-300 real questions, each with the correct answer and the document that contains it. It is the most valuable time they will spend on the project.
- Check retrieval: for each question, did the right document appear in the top few results? If not, no prompt will fix it.
- Check answers: correct, supported by the cited passage, and declined when the sources do not contain the answer.
- Re-run the set whenever documents, chunking, search settings or the model version change.
Open-source tools such as Ragas can automate part of the scoring, but have people check a sample; automated judges make mistakes too.
What about security and data privacy?
The two big risks are showing people documents they should not see, and sending sensitive data somewhere it should not go.
- Permission-aware retrieval. Carry access rules from SharePoint or Google Drive into the index and filter every search by the user's identity from Microsoft Entra ID or Google Workspace. Test with a low-privilege account before launch.
- Where the model runs. Azure OpenAI, Amazon Bedrock and Google Vertex AI run models under your cloud agreement with regional options; our AWS vs Azure vs Google Cloud comparison covers choosing a platform and region.
- Instructions hidden in documents. A file can contain text like "ignore previous instructions". Treat retrieved text as untrusted input, not as commands.
- Logs. Questions reveal what staff are worried about. Protect them and set a retention period.
Firms that hold client documents, such as law and accounting practices, carry extra confidentiality duties; our page on professional services touches on secure document handling. Data residency expectations also vary by market; our UAE page, for example, outlines the federal PDPL and the separate DIFC and ADGM regimes.
What does a RAG project cost, and how long does it take?
A focused pilot on one document collection for one team often fits in six to ten weeks. Most of the effort goes into cleaning documents, syncing permissions and evaluation, not into the model.
Cost drivers, roughly in order of impact:
- the number and messiness of sources (a tidy help center versus twenty years of scanned files);
- connectors and permission syncing;
- expert time for the test set and reviews;
- model usage per question, which grows with the amount of text sent each time;
- ongoing work: re-indexing, monitoring and someone who owns the content.
A sensible pilot picks one team, one collection and one type of question; removes duplicate and superseded files; builds the test set with that team; then launches with feedback buttons and a weekly review. Expand only when the numbers justify it: accuracy on the test set, real usage, and time saved as reported by users.
How Meritbyte Technologies approaches RAG
Meritbyte Technologies is a Nepal-based web and software development company whose AI development practice builds document assistants and extraction tools. We set up evaluation before the user interface, and we will tell you when enterprise search or a structured report is the better answer.
Projects are scoped in a free conversation, with a fixed price for the first milestone (often the pilot), followed by two-week blocks with a demo at the end of each. Models, indexes and cloud accounts sit in your name, in the region you choose, with documentation at handover.
Frequently asked questions
Is our company data used to train the AI model?
With the business and API offerings of major providers, and with cloud platforms such as Azure OpenAI, Amazon Bedrock and Google Vertex AI, the default terms generally say your inputs are not used to train their models. Retention for abuse monitoring and the processing region vary, so read the current data terms for the exact service and plan you use, and record the decision.
Do we need a separate vector database for RAG?
Not necessarily. If you already run PostgreSQL, the pgvector extension handles vector search for many business-sized collections, and search engines such as OpenSearch and Elasticsearch support keyword and vector search together. Dedicated vector databases make sense at larger scale or with specific performance needs. Choose what your team can operate and back up reliably.
Can RAG work with scanned PDFs and spreadsheets?
Scanned PDFs need OCR first, and quality varies: tables, stamps and handwriting often extract badly, so check samples before indexing everything. Spreadsheets usually belong in a database or reporting tool, where exact figures can be queried, rather than being cut into text chunks. Many systems combine RAG for documents with structured queries for numbers.
How accurate is retrieval-augmented generation?
It depends on your documents and questions, not on a published benchmark. Well-scoped systems over clean, current documents can answer most routine questions correctly with citations, while messy or contradictory sources drag accuracy down quickly. Measure it yourself on real questions before launch, and keep measuring after every change to documents, settings or model.