First collect answers you can trust
What I would do if a company said: “Connect AI to the whole Slack archive, email, and old tickets.” I would leave the bot twenty frequent questions from the current knowledge base and a hard handoff rule to a human.
The mistake I would remove first: The problem is not that the model is sometimes wrong. The problem is the team never agreed where AI must say “I don’t know” and who takes the conversation next.
What to prepare and what result to expect
- Outcome: The customer gets a fast answer to a standard question, and the team sees failed queries and knows what to add to the knowledge base.
- Build a knowledge table: question, short answer, rule source, update date, exception, and owner.
- Keep source data and access rights separate from the result so you can audit what AI answers on frequent questions actually did.
- Run tests for a normal question, an outdated rule, an ambiguous wording, and a refund request.
Launch FAQ with honest handoff for hard cases
- Assemble the source of truth
Move prices, timelines, refunds, and limits into one document or table. For each row list an owner and last review date.
Проверьте: A manager can find the source rule in a minute.
Если не сработало: Do not connect old decks and chats until you decide which document is canonical.
- Separate FAQ from actions
Allow AI to read rules and propose an answer, but put refund, order change, cancel, and discount on separate routes with confirmation.
Проверьте: AI cannot perform a sensitive action with a single text reply.
Если не сработало: Remove the tool and leave draft mode only.
- Add a phrase for unknown answers
In the instruction write: “If the answer is not in the knowledge base or depends on the customer’s conditions, do not invent. Say you will connect an employee and hand off the dialog.”
Проверьте: A test unknown question triggers escalation.
Если не сработало: Add a concrete trigger list: refund, complaint, legal question, VIP customer.
- Review failed questions weekly
Store the question, AI answer, employee correction, and failure reason. Once a week add only verified question–answer pairs.
Проверьте: Failures become knowledge-base changes, not endless prompt rewrites.
Если не сработало: Assign a knowledge-base owner and a short weekly review slot.
An answer base, not a chaos folder
Build a table with faq_id, question, answer, source_url, valid_from, valid_to, owner, and escalation_rule. Do not load old decks and chats “just in case”: the model will not know which document is canonical.
In the instruction write: “Answer only from the retrieved fragment. If there is no answer or the question is about refunds, complaints, money, or law—do not invent. Save the dialog and hand it to an employee.”
- Keep a short approved answer and a link to the full rule.
- Once a week review questions where an employee corrected the answer.
The test I would run before launch
Take a normal question, an outdated rule, an ambiguous wording, a refund request, and a request to call a human. For each, pre-record the correct outcome: answer, clarify, or escalate.
If an unknown question gets confident text, the problem is not model temperature. Remove the action, check retrieval, and add an explicit needs_human route.
Two workflows instead of one giant one
First: `Cron → Google Drive/Notion → Extract text → Split text → Embeddings → Vector Store`. Second: `Helpdesk/Webhook → Get ticket history → Classify → IF risk → Human handoff → Retrieve → AI draft → Validate → Send reply`.
In fragment metadata store `source_id`, `title`, `version`, `updated_at`, `valid_until`, and `product_area`. Before search filter out `archived` and documents the user cannot access.
{
"answer": "",
"source_ids": [],
"needs_human": false,
"handoff_reason": null,
"next_question": null
}Five tests before auto-reply
Take a normal question, an outdated rule, an ambiguous wording, a money refund, and a request to call a human. For each, set the expected result in advance: `answer`, `clarify`, or `escalate`. If the question is not in the knowledge base, the answer must be empty and go to `waiting_human`.
Add a limit: after two fruitless clarifications the ticket goes to an employee. Otherwise the bot will endlessly repeat “try again.”
Which questions go to AI vs an employee
| Criterion | Question | Good sign |
|---|---|---|
| Input | What exactly enters AI answers on frequent questions? | Build a knowledge table: question, short answer, rule source, update date, exception, and owner. |
| Action | What is the system allowed to do on its own? | Only pre-listed actions, without access to the whole account |
| Verification | How do you know the result is acceptable? | Run tests for a normal question, an outdated rule, an ambiguous wording, and a refund request. |
| Failure | Where does an unclear case go? | Reply honestly that an employee is needed, save the question, and hand the dialog to the queue without inventing a solution. |
What should change after setup
The customer gets a fast answer to a standard question, and the team sees failed queries and knows what to add to the knowledge base.
Why AI support starts annoying customers
Mixing current and outdated rules in one knowledge base.
Asking AI to answer every question confidently.
Not showing the employee the original dialog.
Trying to fix a bad knowledge base with a long system prompt.
When FAQ becomes a full support system
Bring in a specialist if support runs across multiple channels, uses personal data, or must perform operations in CRM and orders.
What to check on a test dialog
Who is this AI answers on frequent questions approach for?
Business AI support via FAQ works only when AI answers from an approved knowledge base, honestly says “I don’t know,” and hands hard cases to an employee without endless questions.
Where to start if everything is still manual?
Build a knowledge table: question, short answer, rule source, update date, exception, and owner.
How do you check the setup will not hurt you?
Run tests for a normal question, an outdated rule, an ambiguous wording, and a refund request.
What if the result is unclear?
Reply honestly that an employee is needed, save the question, and hand the dialog to the queue without inventing a solution.
