How to Prepare Your Company's Data for a RAG or LLM Deployment

In a RAG project, the language model is usually not the hardest part. Far more problems come from outdated documents, conflicting sources, missing access controls, and data that nobody ever prepared for automated use.

You don't need to reorganize the entire company before your first deployment. But you do need to know exactly what problem you're solving, which data the system can use, and what it should do when it can't find a reliable answer.

Start with the question, not the data

Conversations about RAG deployments usually begin with sources.

We have SharePoint. A few thousand PDFs. Manuals on a network drive. Data in the ERP. Contact history in the CRM. Can we feed it all to AI?

Technically, we usually can. That doesn't mean we should.

If the entire company's documentation goes into the system without a defined purpose, the model gets more information but not necessarily more knowledge. Current procedures mix with archived ones. Instructions meant for different departments start competing with each other. Multiple systems provide different versions of the same information.

The result is an assistant that knows a lot, but you can't predict when it's answering correctly.

That's why the first step isn't connecting a vector database. It's defining a single process.

Not:

We want to use AI for company knowledge.

Rather:

A field technician asks questions about current repair manuals. The system uses only approved technical documentation, cites its source, and declines to answer when it doesn't find sufficient data.

That sentence contains the user, the task, the data scope, and the expected system behavior. That's already the beginning of architecture.

Practical tip: Write your use case as a single sentence starting with "The user asks about..." - if you can't formulate it, you're not ready for deployment.

Every use case needs different data

An assistant answering questions about internal procedures works with different sources than a system checking order status.

The first one uses documents, manuals, and the company knowledge base. The second should pull real-time data from the ERP or order management system. Trying to solve both problems with the same document database quickly leads to errors.

The same applies to other processes:

A system analyzing RFQs needs message content, client data, and lead qualification rules.
A product assistant needs the catalog, specifications, pricing, availability, and variant information.
A marketing agent can use Google Ads, GA4, Search Console, store data, and historical decision logs - this is exactly how our Ads Agent works, operating on live client data rather than static reports.
A system supporting animal breeding doesn't search for similar text paragraphs. It needs genealogical data, calculations, and relationships between records.

There is no single "AI knowledge base." There are specific data sets needed to perform specific work.

Companies usually have more data sources than they think

Data is rarely in one place.

Some knowledge lives in PDF and Word documents. Some in transactional systems. Some in emails, employee notes, spreadsheets, and applications built over the years.

During the first inventory, you'll typically find:

documents and presentations,
procedures and manuals,
ERP system (SAP, Oracle, Microsoft Dynamics),
CRM (Salesforce, HubSpot, Pipedrive),
SQL databases and APIs,
product catalogs,
websites and intranets,
ticketing systems,
spreadsheets,
emails and meeting notes,
industry-specific or custom-built applications.

At this stage, you don't need to import or index anything. You need to establish:

who owns each source,
how it can be accessed,
how often the data changes,
whether the source is still actively used,
who is authorized to view its contents,
whether the data is needed for the first version of the system.

The goal is a data map, not an organizational migration. In our audits, we regularly find that a significant portion of critical documentation sits in SharePoint folders or network drives that nobody actively accesses.

One piece of information, multiple versions

The hardest problem appears when you start comparing sources.

A product's price is in the ERP, the online store, and the PDF catalog. The complaint handling procedure has three versions. Technical specifications differ between the product card and the supplier's spreadsheet.

Each of these documents looks credible. But the language model doesn't know which takes precedence. If it retrieves two conflicting pieces of information, it may pick one, combine both, or generate an answer that matches neither source. And it will do so with full confidence.

That's why you need to designate a single source of truth for every significant type of information.

If inventory levels come from the ERP, the system doesn't look for them in documentation or in a catalog copy from last month. If the current terms of service live in the document management system, archived versions don't compete with it during retrieval.

In practice, you need to define:

the authoritative system for each data type,
the data owner,
update frequency,
conflict resolution rules,
archival labeling policies,
system behavior when current data is unavailable.

A single source of truth doesn't mean one system for the entire company. The ERP may be authoritative for pricing and availability, the CRM for contact history, and the approved document repository for procedures. What matters is that the hierarchy is explicit and documented before you write your first line of code.

Not everything that exists should go into RAG

During the audit, a temptation appears quickly: since we already have access, let's index everything.

That's usually a bad idea.

A large number of documents doesn't automatically improve response quality. It can, however, increase the number of duplicates, contradictions, and results that only appear related to the question.

Before including a source in the system, check:

whether the document is still in effect,
whether it has an owner,
whether you know when it was last updated,
whether the content is a copy of another document,
whether its scope can be clearly defined,
whether the user should actually have access to it,
whether the document is relevant to the chosen use case.

In a production system, fewer well-described sources are often more valuable than thousands of random files.

Messy data doesn't clean itself when you plug in a model

If the documentation is outdated, RAG won't make it current. If the same product appears under multiple identifiers, the model may not recognize they refer to a single item. If an old policy hasn't been marked as archived, it gets retrieved just like the current one.

If documents lack proper metadata and status labels, the LLM doesn't know which version of a policy is in effect. It may retrieve an archived document with the same confidence as the current one. That's not the model's fault. That's the data's fault.

Before deployment, you need to clean up at least the data covered by your first use case:

remove or flag duplicates,
separate active documents from archived ones,
standardize identifiers (same product = same code, no variations),
check completeness of key fields,
label publication and validity dates,
assign a document owner,
define a policy for incomplete records - do you reject, supplement, or flag them?

This doesn't mean a multi-month "data governance" program before starting the project. It means getting a specific scope of data to a state where you can test and iterate without guessing.

Metadata is part of the answer

Document content is only one layer of information. The system often needs to know:

when the document was created and whether it's still in effect,
which market it applies to,
what language it was written in,
which department it belongs to,
who approved it,
what its confidentiality level is,
which product, client, or project it relates to.

This information is stored as metadata. It allows the system to narrow search to current technical documentation for a specific market, rather than searching the company's entire knowledge base.

Metadata also helps resolve conflicts. If two documents contain different answers, the system can prefer the newer, approved one assigned to the correct region. Example: a document with status "approved" and an update date of 2026-07 wins over a draft version from 2025-01, even if the draft is semantically closer to the question. Without metadata, the system only sees text similarity - and has no basis for making a decision.

Access controls must work before the answer is generated

An AI assistant cannot bypass the access rules in place at the company. If an employee doesn't have permission to open a document in SharePoint, they shouldn't receive a fragment of it in a model-generated response.

Access control must operate at the retrieval stage. The system first determines which data the user is authorized to see, and only then selects context for the model.

Depending on infrastructure, this may require integration with:

the company's identity management system,
Active Directory or Entra ID,
application-level roles,
project-specific permissions,
access control lists assigned to documents.

Filtering the finished response is too late. The model should never receive content that the user isn't authorized to access.

For personal, medical, financial, or other regulated data, additional requirements around retention, logging, and processing location apply. These rules must be established before selecting a model provider and infrastructure - not after.

RAG doesn't always mean a vector database

The most common RAG variant relies on semantic search - documents are split into chunks, converted to vectors, and compared with the user's question. This works well for many document types and descriptive questions.

But not for every type of information.

If the user asks about invoice INV/2026/001, a serial number, or a specific product code - exact matching may matter more than semantic similarity.

If they ask about the current order status - the right answer may be a SQL query or an API call, not a document search.

If they want to analyze relationships between objects - you may need a graph, not a vector database.

That's why the retrieval layer can combine multiple methods:

semantic search (vectors),
full-text search (BM25),
metadata filtering,
SQL queries to transactional systems,
real-time API calls,
graphs and relationships,
dedicated computation functions.

Good architecture doesn't force all data into a single store. It selects the source and method based on the type of question.

Hybrid search: meaning and precision

In larger knowledge bases, semantic search is often combined with traditional full-text search.

Semantic search handles meaning well. The question "What is the supplier's liability for delayed delivery?" can lead to a document that uses the phrase "exceeding the delivery deadline," even though the words aren't identical.

Full-text search is better at finding:

product codes,
invoice and order numbers,
proper nouns,
part symbols,
industry abbreviations,
exact contract language.

The third layer is metadata filtering - narrowing the search set before the actual search begins. "Show only documents from the Finance department from the past year" - this dramatically increases precision and eliminates irrelevant results.

Combining these three methods finds semantically similar content without losing precise identifiers.

Chunking - where to split a document

The model shouldn't receive a multi-hundred-page document every time a user asks a question. Documents are therefore split into smaller pieces - chunks.

It sounds like a technical detail. In practice, the splitting strategy can completely change response quality.

If a chunk is too small, the system finds the specific sentence but loses the conditions and exceptions described a paragraph earlier. If it's too large, the model receives a mass of irrelevant content, and the key information disappears in the middle.

Three main approaches:

Splitting by document structure - chapters, headings, sections, and paragraphs as natural semantic boundaries. Works well for manuals, policies, technical documentation, and legal materials.

Splitting into smaller text fragments - shorter chunks improve precision for simple questions but require preserving information about the parent document and adjacent sections.

Overlapping chunks - some content is repeated between adjacent chunks so that sentences or conditions aren't cut at the wrong point. But overlap isn't a universal fix - it increases index size and can cause near-identical results to be returned.

The most important rule:

Don't choose your chunking strategy based on an article on the internet. Compare several variants using the questions your users will actually ask.

Technical documentation requires a different strategy than customer emails, and product catalogs require yet another. This is one of the areas where the implementation team's experience makes the biggest difference.

Choosing an embedding model

An embedding converts text into a numerical representation, allowing the system to compare the meaning of a question and a document. Thanks to embeddings, a question about "delivery delay" is linked to a document about "order fulfillment timelines."

The choice of embedding model matters, but a general ranking won't tell you how the model handles your technical documentation in your language, drug names, legal phrasing, or industry abbreviations.

Criteria that should guide the choice:

Language and domain - a model optimized primarily for general-purpose English text may perform worse on technical, legal, or medical terminology in other languages. You need to test this on your own data - don't assume the model "understands" your domain just because it scores well on an English benchmark.

Quality on your data - don't trust general benchmarks. Prepare a representative set of questions your users would actually ask and manually check which model retrieves the most relevant fragments. In a small pilot this might be a few dozen cases; in a higher-risk system, considerably more. Testing on your own data matters more than a vendor's leaderboard.

Cost and performance - embedding models differ in quality, cost, and speed. A larger model may score higher on a benchmark but doesn't guarantee better results on your company's data. If the system needs to handle hundreds of queries per minute, you'll need to balance quality with throughput.

Privacy and hosting - can your company's data go to a public cloud? Do GDPR requirements or industry regulations require hosting the model on-premise? This is a question to ask at the start of the project, not at the end.

Stability and versioning - if the provider changes the model without notice, your existing embeddings will become inconsistent with newly generated ones. Swapping an embedding model means reindexing the entire database - not a trivial operation.

A larger model doesn't automatically perform better. Quality is determined by the entire system: data, chunking, metadata, search strategy, reranking, and the types of questions being asked.

The "Lost in the Middle" problem

New models accept increasingly large amounts of text in their context windows. It's tempting to assume you can pass them all retrieved documents and let them pick the right answer on their own.

It doesn't work as well as the theory suggests.

Research on long-context utilization showed that models make better use of content at the beginning and end of the context, and struggle with information buried in the middle. If you feed 10 chunks and the correct answer is in the fifth one, the model tends to overlook it.

In practice, it's better to pass the model a few well-chosen fragments than dozens "just in case."

What helps:

precise metadata filtering,
better retrieval (hybrid),
limiting the number of results,
deduplicating output,
reranking - re-evaluating retrieved chunks and promoting those most relevant to the specific question,
splitting complex questions into multiple steps.

Reranking won't fix bad data or poorly designed search. But it can significantly improve context selection before it's passed to the model. Fewer chunks, better chosen - that's the key to accurate answers.

Data needs to stay current

A one-time document import is enough for a demo. A production system needs synchronization.

If policies, pricing, documentation, or product data changes, the update must reach the AI layer. If a source is deleted, its chunks can't remain in the index indefinitely.

You need to define:

how often data will be refreshed,
whether the system pulls the full dataset or only changes (delta sync),
what happens when a document is deleted from the source,
how sync errors are handled,
which version of a document is active at any given moment.

Frequency depends on the process. Documentation updated a few times a year doesn't need minute-by-minute sync. Product availability data may need to be read directly from the source system with every query.

Versioning is not optional

If the system gives a wrong answer, you need to be able to determine:

what question it received,
which sources it used,
what chunks it retrieved,
which version of the document was active,
which model it used,
what rules and prompts were in effect at that moment.

Without this, the team sees only the output but can't reconstruct the cause.

Versioning data, prompts, models, and retrieval configuration lets you distinguish a random error from a regression caused by a system change. That's one of the differences between a RAG demo and a system you can maintain in production.

Build test questions before you start building

Don't wait until the system is ready to start testing. At the beginning of the project, you can already collect questions that users actually ask.

These should include:

typical questions - the system's daily bread,
imprecise questions - the way people actually ask,
questions requiring information from multiple sources,
questions with no answer in the sources - the system must know how to decline,
questions about archived data,
cases with restricted access,
questions containing exact symbols, numbers, and identifiers.

For each question, it's worth defining:

where the correct answer can be found,
what information must appear in the response,
what the system must not make up,
when it should decline to answer,
which source it should cite.

Only with such a test set can you meaningfully compare chunking, embeddings, search methods, and reranking. A vendor's general benchmark won't replace testing on your data, your questions, and your use case.

Minimum data audit before deployment

Before you start building, answer a few questions:

Not every answer needs to be "yes" right away. The audit is meant to show where the gaps are and whether you can safely launch a first version on a limited scope.

Case Study: BLUP-FLOCK - when standard RAG isn't enough

BLUP-FLOCK is an ERP platform for managing poultry breeding stock, built on Laravel 11, Vue.js 3, MS SQL, and Redis, running on Kubernetes. The system doesn't work solely with documents. It must reconstruct family trees spanning five generations, calculate inbreeding coefficients, and support breeding candidate ranking using the BLUP algorithm.

Standard text-similarity search isn't sufficient here.

The data problem starts at collection

In pedigree poultry breeding, each bird is identified with an individual wing pin. Data is collected with handheld scanners - but the process itself depends on the type of farm.

On a cage farm, it's relatively simple: a scale rides between cages on a light platform with a computer. The bird is in the cage, the egg is easy to weigh and assign.

On a litter farm, hens walk freely and lay eggs in special trap nests. The person pulling a hen from the nest must also grab the egg - there's no room for weighing at the nest. We designed a handheld scale, but gravity defeated us. The solution turned out to be simpler: a worker walks with a scanner, and every 20 meters or so, electronic scales are mounted on the wall.

The result: twice as many eggs weighed compared to the old data collection method. Partly thanks to the new tools, and partly thanks to more consistent enforcement of the new process - technology alone isn't enough without changing the way people work. We still don't weigh all eggs - but as the professor from the agricultural university consulting on the project explained, in pedigree breeding you don't weigh every egg. What you need is a properly selected, representative sample.

Why this matters for RAG

This example shows something rarely discussed in articles about AI deployments: data quality in the system is limited by how data is physically collected. No embedding model will fix the fact that some eggs weren't weighed because the hen was walking on litter instead of sitting in a cage.

What was needed was an architecture combining structured data, genealogical relationships, a custom retrieval mechanism, deterministic calculations, an explainability layer, and result auditability. The language model is one component in such a system - it doesn't replace database queries, the BLUP algorithm, or the rules responsible for computational accuracy.

After streamlining the data collection process and the way data was made available, result quality improved without changing the model. The problem wasn't the LLM - it was how data was prepared and delivered.

That's an important lesson: sometimes data preparation means cleaning documents and metadata. Other times it means designing a physical data collection process on a litter farm.

See the full BLUP-FLOCK case study →

CommercePilot: product data as a special case

If your use case is a shopping assistant, product chatbot, or AI agent operating on an e-commerce catalog, product data must be as clean as technical documentation in manufacturing.

It rarely is.

CommercePilot scans an online store and checks the completeness and consistency of product data - automating part of this audit for publicly accessible e-commerce catalog data.

Common problems it detects:

Missing product identifiers (EAN/GTIN) - missing EAN codes make it harder to uniquely identify products, match offers, and use data across shopping platforms and agentic systems. This isn't just an SEO issue - it's a readiness issue for agentic commerce.

Incomplete descriptions and missing attributes - if a product card lacks dimensions, materials, or weight, the model can't generate a meaningful answer to a customer's question. Garbage in, garbage out.

Inconsistent categorization - the same product assigned to three categories signals that the data lacks a single source of truth.

Missing structured data - missing proper Schema.org structured data, most commonly implemented in JSON-LD, makes it harder for machines to unambiguously identify the product, price, availability, and specifications. It doesn't completely block catalog utilization, but increases dependence on less reliable page parsing.

If you're planning an AI deployment in e-commerce, a product data audit is one of the first steps. CommercePilot automates it for the storefront layer and gives you a clear picture of what needs fixing on the catalog side.

Check your store with CommercePilot →

You can swap the model. Data problems stay

Models will keep changing. A company can switch from one provider to another, use a cheaper model for simple tasks, or a more powerful one for analyses requiring deeper reasoning.

The model is one of the more replaceable elements of the architecture. Changing it still requires testing, prompt adjustments, and quality checks - but it's usually simpler than fixing data sources, access controls, and integrations that have been evolving for years.

What's much harder to fix:

no single source of truth,
unclear access controls,
conflicting documents,
no versioning,
integration with a system nobody maintains,
a process that was never clearly defined.

That's why data preparation isn't work you do "before the real AI." It is part of the real deployment.

Planning a RAG or LLM deployment?

We start with the process, the data, and the sources of truth. Only then do we select retrieval strategies, models, and the rest of the architecture.

We design RAG systems, data pipelines, and AI agents integrated with real documents, databases, ERP, CRM, and operational business tools.

See how we build production AI systems and LLM integrations →