How to Test an AI Chatbot Before Launch: A Practical Method
A repeatable way to test an AI chatbot before launch: build a golden question set, score retrieval and grounding separately, and…
What an HR policy chatbot should answer, what it must never touch, how to handle role and location variants, and the numbers that prove it is working.
An HR policy chatbot earns its place the moment it stops your team answering the same leave question for the ninth time this month. However, it only earns trust if every answer quotes the clause it came from.
Indeed, that distinction runs through this guide. A generic assistant guesses politely, whereas a policy assistant retrieves, cites and knows when to stay quiet.
Overall, we cover what to connect, how to handle role and location differences, where the assistant must hand over to a person, and which numbers prove the project worked.

In most workplaces, policy questions cluster tightly. In most companies, twenty questions cover the majority of the volume, and leave, notice periods, reimbursements and holidays sit at the top.
Employees still ask a person, though, for two reasons. First, the handbook is long and the answer is buried in it. Second, they are not sure the version they found is current.
So the problem is rarely a missing document. Instead, it is a retrieval problem wearing a documentation costume, and that is exactly what an HR policy chatbot is built to solve.
There is a cost to the status quo as well. Every interruption pulls an HR generalist away from hiring, onboarding and casework, which are the parts of the job that need judgement.
In practice, five requirements separate a useful assistant from a liability. Miss one, and the first wrong answer will be the one everyone remembers.
Notice that none of these depend on the model. They depend on how you organise policies and permissions, which is the work most teams underestimate.

To begin with, start with documents that were written for everyone. The handbook, the leave policy, the travel rules and the benefits guides all qualify, because they describe entitlements rather than individuals.
Next, add the documents that vary by group. Shift rules, country supplements and grade-based benefits belong here, and they need role filters so the assistant answers correctly for the person asking.
Finally, keep individual records out of open chat. Salary slips, appraisals and medical notes describe one person, therefore they belong behind authentication in your HR system rather than inside a general assistant.
Clearly, this boundary is not only a privacy control. It also keeps answers accurate, since an assistant that mixes policy text with personal records produces replies that are hard to verify.
Generally, most wrong answers come from the right document read in the wrong context. The clause was accurate, yet it did not apply to that employee.
| Question | What changes the answer | What the assistant needs |
|---|---|---|
| How much leave do I have left? | Individual balance | A link to the HR portal, not a policy quote |
| How many casual leaves do I get? | Location and employment type | Role and country filters on retrieval |
| What notice must I serve? | Grade and contract | The current contract template for that grade |
| Can I claim this taxi fare? | Travel policy version and city tier | Effective-date checking and city rules |
| When is the next holiday? | Office location | The location holiday calendar |
As a result, the design question is not whether to connect the HR system. It is which questions genuinely need a record lookup, and which only need the right version of a policy.
To begin with, employee records count as personal data under India’s Digital Personal Data Protection Act. Consequently, an assistant that widens access to those records creates real exposure, even when nobody intended it.
Fortunately, the safe pattern is straightforward. Retrieval runs with the permissions of the person asking, and the assistant reads policy documents rather than personal files. Meanwhile, every query and every source used gets logged.
Additionally, two more controls help during review. Keep processing and storage inside the region your legal team expects, and set a retention window for conversation logs instead of keeping them forever.
Our guide to permission-aware chatbots and DPDP covers the architecture in detail, including how permission mirroring works at query time.
In our experience, restraint builds trust faster than coverage does. Some questions should never receive an automated answer, however well the assistant might phrase it.
Therefore, make these rules explicit in the configuration rather than hoping the model behaves. In addition, tell employees where the line sits, because clarity about limits raises confidence in everything else.
Above all, agree the measures before launch. Otherwise the review meeting turns into a debate about chat volume, which proves nothing.
| Metric | Definition | Healthy range |
|---|---|---|
| Cited answer rate | Replies that quote and link a policy | Above 95 per cent |
| True resolution | No repeat question from the same person within 48 hours | 40 to 65 per cent |
| HR hours returned | Repeat questions removed, multiplied by handling time | Rising for three months |
| Correct variant rate | Answers that match the asker’s role and location | Above 98 per cent |
| Escalation with context | Handoffs that carry the thread and sources | 100 per cent |
In addition, track answer quality by policy area. Usually one or two areas produce most of the errors, and fixing those documents lifts the whole system.
In short, thirty days is enough, provided you resist the urge to connect everything. Start with the questions HR already answers weekly.
| Stage | Work | Exit criteria |
|---|---|---|
| Days 1 to 7 | List the top 25 questions and find the clause that answers each one | Every question has a named source document |
| Days 8 to 14 | Clean the handbook, mark effective dates, remove superseded versions | One current version per policy, with owners |
| Days 15 to 21 | Connect sources, apply role and location filters, test the 25 questions | At least 23 answers correct and cited |
| Days 22 to 30 | Launch to one department with escalation and logging switched on | Gap list produced, first policies rewritten |
Start with one department first, and then add the rest. A department that adopts the assistant willingly becomes the reference story for everyone else.
In practice, four patterns cause most of the damage, and each one is easy to avoid.
None of these need new technology to fix. They need editorial discipline, which is why HR usually owns this project rather than IT alone.
Notably, the retrieval stack behind HR is the same one behind IT and customer support. Once permissions, connectors and evaluation exist, adding a second audience becomes configuration rather than a new build.
Many teams start with an assistant in Slack or Microsoft Teams, then extend it to HR. Others begin with HR, because the question list is small and the value is easy to measure.
In either case, the platform should support both. Intellowork connects your policy library, mirrors the permissions you already have, and keeps citations on every answer, so an HR policy chatbot and an IT assistant can share one foundation.
Naturally, the licence is rarely the interesting number. Most of the effort sits in content and configuration, especially in the first month.
| Cost area | Share of effort | Main driver |
|---|---|---|
| Policy cleanup | High | Duplicate handbooks and undated versions |
| Role and location rules | Medium | Number of countries, grades and contract types |
| Review and tuning | Medium | Weekly review of flagged and low-confidence answers |
| Model usage | Low | Question volume, which is modest for internal HR |
After launch, moreover, expect a steady hour or two each week. That small commitment keeps accuracy high and prevents the slow decay that ends most pilots.
Retrieval quality starts in the document rather than in the model. A clause written as one dense paragraph is hard for people and machines alike.
Broadly, four habits make a policy library retrievable. First, give every clause a heading that matches how employees phrase the question. Second, keep one idea to a paragraph. Third, state the effective date at the top of the page. Finally, name the owner, so somebody is accountable for keeping it current.
Similarly, tables help too, especially for entitlements that vary by grade or location. A short table answers the variant question directly, whereas prose forces the reader to work it out.
Furthermore, avoid cross-references where you can. A clause that says the rules follow the travel policy sends the assistant, and the employee, on a second search.
Typically, these edits take a fortnight for most handbooks. In return, they improve every future answer, and they make the handbook easier for new joiners to read on their own.
As a rule, the question mix is remarkably consistent across companies. Leave dominates, followed by money and time.
Because the mix is predictable, you can prepare for it. Cover these five areas properly and an HR policy chatbot will answer most of what arrives in its first month.
Meanwhile, keep the rest for later. A narrow assistant that is right beats a broad one that is occasionally wrong, particularly on questions about pay.
Of course, Indian workplaces rarely run in one language. Factory teams, field staff and support desks often ask in Hindi or in a mix of Hindi and English, while the handbook stays in English.
Happily, good multilingual retrieval solves this without translating your policy library. The question gets embedded in its own language, the search still finds the English clause, and the answer comes back in the language the person used.
Even so, two details matter here. The assistant should keep the citation pointing at the original document, because that is the version legal signed off. It should also handle mixed scripts, since many employees type Hindi words in the Latin alphabet.
Finally, test this properly before launch. Ask twenty real questions in each language and check that the same clause comes back every time.
It should not, at least in open chat. Policy answers come from group-level documents, while personal balances and payslips stay behind a login in your HR system.
About thirty days for one department, and most of that time goes into cleaning policies rather than configuring software.
The assistant should say so and route the question to a person. Afterwards, the gap appears in a report, so HR can write the missing clause once and answer it forever.
Yes. Multilingual retrieval lets employees ask in the language they prefer while the answer still cites the original policy document.
Compliance depends on the build, not the label. Keep retrieval permission-aware, log access, hold data in the expected region and set a retention period for conversation history.
HR operations, with support from IT. The weekly job is editorial, since most corrections involve rewriting a clause rather than changing a setting.
To begin with, write down the twenty-five questions your team answered this month. Find the clause that answers each one, mark the effective date, and delete every superseded version you find along the way.
Notably, that exercise alone improves your handbook. Then connect it, keep citations on every reply, and let an HR policy chatbot handle the repeats while your team keeps the casework. If you want the retrieval and permission layer ready-made, Intellowork is built for exactly this pattern.
A repeatable way to test an AI chatbot before launch: build a golden question set, score retrieval and grounding separately, and…
How to build an AI chatbot for Slack and Microsoft Teams that answers from your own documents, respects permissions, cites its…
An assistant that can read every document in your company is an incident waiting for a question. Access control, residency and…
Tell us what you are trying to ship. We will tell you what it actually takes — scope, sequence and the risks worth knowing about before you commit budget.