Enterprise RAG on Salesforce Data: An Architecture That Holds Up

Short answer: retrieval quality decides whether RAG on Salesforce data works, not which model you pick. A frontier model given the wrong three records will answer confidently and wrongly, and swapping the model changes nothing. The three things that actually move the needle are what you treat as a chunk, how permissions are enforced during retrieval, and whether you measure retrieval separately from generation.

Why does RAG on CRM data fail more often than RAG on documents?

Because a CRM corpus breaks the assumptions that document RAG is built on.

• Records are short and repetitive. Ten thousand cases about the same product produce near-identical embeddings, so similarity search returns ten variations of one answer.
• Meaning lives across objects. The answer to "why did this customer churn" is spread over an account, three opportunities, a contract and a case thread. No single record contains it.
• The corpus mutates constantly. Opportunities move stage, cases close, contacts change role. A document corpus is mostly static; a CRM corpus is a moving target with an index behind it.
• Access is per user. Two people asking the same question must legitimately get different answers, which is not a scenario most RAG tutorials consider at all.

What is the right chunk when the source is a Salesforce record?

Field-by-field chunking is the most common mistake we are asked to fix. A field value is not a retrievable unit of meaning. Assemble instead.

1. Chunk at the level of a business event, not a field. For a case, that is subject plus description plus resolution plus the comment thread, rendered as one passage. For an opportunity, the record plus its close reason and key notes.
2. Denormalize the identifiers a human would use. Account name, product name, serial number, contract number. Embeddings do not resolve foreign keys.
3. Keep hard metadata alongside the vector. Object type, record id, owner, account, last modified date. Every filter you will ever need at query time has to exist as metadata, not as text.
4. Keep exact-match retrieval in the loop. Case numbers, SKUs and error codes are precisely where pure vector search is weakest. Salesforce's own hybrid search combines keyword and vector indexes and fuses the ranked results, reporting better performance than either alone (Salesforce Engineering).
5. Re-embed on change, not on schedule. Tie re-indexing to record change events, or your agent will confidently quote last quarter's status.

How do you make retrieval permission-aware without rebuilding sharing?

This is the requirement that separates an enterprise RAG system from a demo, and it has exactly one safe shape: filter before you retrieve, using the asking user's identity.

Three patterns, in descending order of how well they hold up.

• Pre-filtered retrieval. The query carries the user's identity, the retrieval layer restricts candidates to what that user can see, and ranking happens inside that set. Correct, and the only one that survives an audit.
• Segmented indexes. One index per audience, such as per region or per business unit. Workable when your access model is coarse and stable, unmanageable when it is not.
• Post-filtering. Retrieve broadly, then drop what the user cannot see. Do not do this. Restricted records still consume the top-k slots, so the answer silently degrades, and any ranking signal exposed to the user leaks the existence of the records you filtered out.

Whatever store you use, identity has to propagate all the way through: session, retrieval, prompt, and the audit log. Never ask the language model to enforce access. It is a text predictor, not a policy engine, and a prompt instruction is not a security control.

How do you know whether retrieval is actually good?

Most teams cannot answer this, which is why they keep changing models. Separate the two failure modes: either the right record was not in the context, or it was there and the model ignored it. Only the second is a model problem.

1. Build a golden set of 100 to 200 real questions from actual users, each labelled with the record ids that genuinely answer it.
2. Measure retrieval on its own with recall at k and mean reciprocal rank. If the right record is not in the top k, nothing downstream can save you.
3. Track context precision too. Padding the prompt with near-duplicate chunks costs budget and makes the model hedge.
4. Run the golden set in CI on every change to chunking, embeddings, ranking or filters, and treat a drop as a build failure.
5. Only then evaluate answers, with citations back to record ids so a reviewer can check the claim in Salesforce.

Teams that add this harness usually discover their retrieval recall was somewhere around half, and that no model upgrade was ever going to fix it.

What does a stack that holds up look like?

Native or external matters less than the retrieval contract. Data 360 supports vector search and retrievers over unstructured content such as knowledge articles, PDFs and transcripts (Salesforce Help), which keeps grounding close to the data and inside the platform's governance. An external vector store gives more control over chunking, hybrid ranking and cross-system corpora, which matters when the answer also lives in your ERP or your ticketing system.

Pick either, but write the contract down first: what is a chunk, what metadata every chunk carries, how identity is enforced, and what the golden set says good looks like. We build agentic RAG systems on this pattern across Salesforce and 70+ enterprise systems, and it is the contract, not the model, that we spend the most time on. More on how we work is at Tekunda.

FAQ

Does a bigger model fix bad retrieval?

No. If the answering record never enters the context window, model size is irrelevant. Fix recall first, then compare models.

Should I fine-tune instead of doing RAG on CRM data?

Rarely. CRM data changes daily and access is per user, so a fine-tuned model would be both stale and unable to respect sharing. Fine-tuning is for behaviour and format, retrieval is for facts.

How do I stop the agent leaking records a user cannot see?

Filter candidates by the asking user's access before ranking, and log every retrieval with the identity that made it. Prompt instructions are not an access control.

How often should the index be refreshed?

Drive it from record change events rather than a nightly job. In a CRM corpus, a stale index produces answers that are wrong in a way users find hard to detect.