Put the documents in order first
I would not build a knowledge base from the entire company folder. I would first assemble a short set of approved documents with an owner and a date, because search over a stale file makes the error sound convincing.
The mistake I would remove first: People try to fix a bad base with a long prompt. But if it holds two different answers to one question, the model will not know which is current until you add freshness rules.
What to prepare and what result to expect
- Outcome: The assistant answers from the right document version and can show where the rule came from, instead of confidently retelling an old file.
- Build a document registry: name, type, version, owner, update date, audience, access, and next review date.
- Keep source data and access rights separate from the output so you can audit what the knowledge base for the AI assistant did.
- Create tests for an exact question, a two-version conflict, a missing answer, and a restricted document.
Build a base you can keep updating
- Remove junk before upload
Do not add duplicates, drafts, or ownerless documents to the index. First choose one current source for each rule.
Проверьте: Every document has an owner and a revision date.
Если не сработало: Move unclear files to quarantine instead of deleting them indiscriminately.
- Add access metadata
For each record, store department, region, product, and access level. Search must filter the document before text reaches the model context.
Проверьте: A user does not receive a fragment of a document they cannot access.
Если не сработало: Start with one shared folder without sensitive data.
- Write answers grounded in a fragment
In the instruction, set the format: answer, document, version, what to do on conflict. Do not ask for quotes if the source never entered the context.
Проверьте: The answer includes a link or source name.
Если не сработало: Make the answer a draft and show the found fragments to a human.
- Run a question-driven review
Once a week, review unanswered questions, stale documents, and cases where an employee corrected the assistant. Update the source, not only the prompt.
Проверьте: Every error becomes a document change or a test change.
Если не сработало: Assign a base owner and a calendar review date.
Minimum document card
For each file store doc_id, title, source_url, owner, valid_from, valid_to, audience, access_level, and status: draft, approved, archived. Before search, filter out archived documents and documents the user cannot access.
Split long instructions by meaning, not every 500 characters. Keep product, region, and date in the fragment title so you do not mix a return policy for one country with a rule for another.
- Every answer needs a link to the source fragment.
- A question the base cannot answer should go into an update queue.
How to accept the base before launch
Build 20 test questions: ordinary ones, synonyms, conflicting rules, an expired document, and an out-of-access request. For each, record expected_source and expected_answer. That validates search accuracy and access control, not how “smart” the AI sounds.
A document needs an owner and a lifetime
Add `doc_id`, `title`, `source_url`, `owner`, `valid_from`, `valid_to`, `audience`, `access_level`, and `status = draft/approved/archived`. Before search, exclude the archive and documents without access rights. Split a long instruction by meaning, not randomly every 500 characters.
Build 20 test questions: ordinary, synonym, conflicting rules, expired document, and out-of-access request. For each, store `expected_source` and `expected_answer`. The base is ready when you can explain where the answer came from and who will update the document after an error.
How to know a source is safe to give AI
| Criterion | Question | Good sign |
|---|---|---|
| Input | What exactly enters the knowledge base for the AI assistant? | Build a document registry: name, type, version, owner, update date, audience, access, and next review date. |
| Action | What is the system allowed to do on its own? | Only prelisted actions, without access to the entire account |
| Check | How do you know the result is acceptable? | Create tests for an exact question, a two-version conflict, a missing answer, and a restricted document. |
| Failure | Where does an unclear case go? | Return “not found” and hand the question to the base owner—do not ask the model to fill the gap from general knowledge. |
What should change after setup
The assistant answers from the right document version and can show where the rule came from, instead of confidently retelling an old file.
Why the assistant confidently cites old rules
Indexing a folder of drafts and outdated versions.
Relying on a prompt instead of an access filter.
Not showing the user the answer source.
Updating only the vector store without an owner for the source document.
When a knowledge base becomes a separate system
You need a specialist when the base holds personal data, multiple access levels, dozens of sources, or must pass an audit.
What to test before connecting the team
Who is this approach for when working with a knowledge base for an AI assistant?
A knowledge base is not a folder of files. It needs current documents, clear owners, metadata, and a rule for what to do when no answer is found.
Where should you start if everything is still manual?
Build a document registry: name, type, version, owner, update date, audience, access, and next review date.
How do you check that the setup will not cause harm?
Create tests for an exact question, a two-version conflict, a missing answer, and a restricted document.
What should you do with an unclear result?
Return “not found” and hand the question to the base owner—do not ask the model to fill the gap from general knowledge.
