Define roles before testing answers
Start with the real user roles that will use the system: front-desk staff, analysts, clinicians, managers, contractors, executives, or support agents. For each role, list the document sets, records, fields, and actions that are allowed.
A permission test should ask the same question as different roles. The expected answer may be full response, partial response, refusal, escalation, or a request for a narrower question.
Test retrieval and generation separately
The retriever should not surface documents that the user is not allowed to access. The generator should also avoid leaking restricted content from snippets, citations, summaries, or source titles.
Score each test case on retrieved sources, citation safety, answer content, refusal behavior, and escalation. This makes it easier to see whether the failure is access control, chunking, metadata, prompt behavior, or UI display.
Include mixed-access and stale-source cases
Real knowledge bases contain overlapping policies, outdated files, draft documents, customer records, and restricted attachments. Permission testing should include cases where only part of the source set is allowed.
For healthcare and financial workflows, stale or restricted material can be more dangerous than no answer. The system should prefer a safe refusal or escalation over a confident answer from the wrong source.
Review logs with the source owner
Permission tests should produce a review trail: user role, question, retrieved source IDs, access decision, generated answer, citations shown, refusal reason, and reviewer notes.
A source owner should be able to update metadata, retire old documents, clarify ownership, and add new negative examples when a failure appears.