Deterministic Triage for Member Inquiries, Run on Your Own Machine

By the end of this tutorial you will have a working triage desk on your own computer: member inquiries go in, each one comes out labeled for the right department, and the same message gets the same label every time. You need Python 3.12, about two gigabytes of free disk, and the list of queues your inquiries go to today.

A chatbot can write a stunning answer to a member question on Monday and a different one on Tuesday. That is not a flaw in your prompt. In 2025, researchers testing LLM reproducibility found the same model and prompt varying by up to 9% in accuracy and 9,000 tokens in response length from GPU count, type, and batch size alone, even with greedy decoding (Yuan et al.). Sampling plus floating-point arithmetic moves the output while the input stands still. This is a teaching example, not a case study: the numbers come from a reference run on synthetic messages, with its own warnings attached.

Laya is a different kind of tool: a decision model that scores a fixed set of categories and picks the top one, with no token sampling, so reruns agree. It runs entirely on your machine, which settles the privacy question directly: member text never leaves the building. For what is safe to paste into cloud tools instead, our member data tutorial covers that separately.

Stage 1: set up the machine

1. Create a Python 3.12 virtual environment. Per the Python venv documentation: python3 -m venv .venv, then source .venv/bin/activate.

2. Install Laya 0.3.5 with the CPU-only PyTorch build. Install the CPU wheel first so pip never pulls a GPU runtime you will not use, then the pinned requirements.

python -m pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cpu
python -m pip install laya==0.3.5 transformers==5.10.0 huggingface_hub==1.32.0 safetensors==0.8.0 numpy==2.2.6

3. Download the pinned English checkpoint and verify the receipt. This pulls revision 1c5edc17a7acd8701df6fc341c0d179f1c62c982 of the public laya model repository into models/laya-english: 846,195,574 bytes, about 807 MB. Confirm the byte count before going further.

from huggingface_hub import snapshot_download
snapshot_download('convaiinnovations/laya',
                  revision='1c5edc17a7acd8701df6fc341c0d179f1c62c982',
                  local_dir='models/laya-english',
                  allow_patterns=['model.safetensors', 'rl_agent_config.json',
                                  'encoder/*', 'tokenizer/*'])

Stage 2: teach it your categories

4. Classify one inquiry and read the probabilities. The reference run below used a duplicate-payment message: the model chose billing at 0.9806 in about 300 milliseconds after a one-time load of roughly six and a half seconds.

import json, os
os.environ.update(HF_HUB_OFFLINE='1', TRANSFORMERS_OFFLINE='1')
import torch, laya
torch.set_num_threads(3)
agent = laya.load('models/laya-english', device='cpu')
questions = {'department': {
    'type': 'choice',
    'instructions': 'Which team should handle this message?',
    'criteria': {
        'billing': 'payments, duplicate charges, refunds',
        'technical': 'software errors, bugs, login problems',
        'sales': 'pricing or buying the product',
        'other': 'unclear request or none of these'}}}
result = agent.system_one(
    'I was charged twice. Please refund the duplicate payment.',
    questions)
print(json.dumps(result['answers']['department']['choice']))
print(json.dumps(result['answers']['department']['probabilities'], indent=2))

Reference output (excerpted):

"billing"
{
  "billing": 0.9806,
  "technical": 0.0062,
  "sales": 0.0061,
  "other": 0.007
}

A probability is not an accuracy score: 0.9806 means confidence, not 98% correctness.

5. Rewrite the choice question with your association’s categories. The criteria dictionary is the entire template: one plain-language description per queue. Fill it in with your real departments, and use the colleague from your prerequisites to settle borderline cases.

questions = {'department': {
    'type': 'choice',
    'instructions': 'Which team should handle this member message?',
    'criteria': {
        'dues': 'dues payments, duplicate charges, refunds, billing questions',
        'technical': 'portal login, password resets, website errors',
        'membership': 'joining, renewing, upgrading, member benefits',
        'other': 'unclear request or none of these'}}}

6. Triage a batch and watch the uncertain one. Run the same call over several messages and read every probability. In the reference run, a portal login error scored technical at 0.8643, while a group pricing question split sales 0.3655 against billing 0.3464: that message needs a person, and the next step makes that routing automatic.

Stage 3: gate every decision

7. Add the clarification gate. Route a message onward only when the chosen category is not other, the top probability reaches 0.80, and its lead over second place reaches 0.20; everything else goes to a human. In the reference run the refund case passed, while a vague “something strange on my account” message (technical 0.351 against other 0.242) failed and went to a person.

def gate(answer):
    probs = sorted(answer['probabilities'].values(), reverse=True)
    return (answer['choice'] != 'other'
            and probs[0] >= 0.80
            and probs[0] - probs[1] >= 0.20)

These cutoffs are illustrative, not calibrated for your inquiry volume.

8. Rerun the batch and diff the labels. Run the same messages again and compare every choice. In the reference run the refund message scored billing at 0.9806 in three separate runs, down to the fourth decimal. Nothing is sampled between runs, so the same input returns the same label.

Stage 4: draft, never send

9. Draft replies with a local model if you want them; the draft stays a draft. This step is optional. Point the script at a language-model server on your own machine, give it one policy per department (billing: ask for the order number, never promise a refund), and cap replies at two sentences at temperature 0. Every reference result is marked draft only, not sent; no code here sends anything.

10. Log every decision and keep a human on the send. Write one line per message: timestamp, chosen label, top two probabilities, gate outcome. Member text never left your machine, and the send button stays with a person who read the draft.

Check your result: same message, same label, every time

Rerun your batch twice and diff the choices; every line should match. Confirm uncertain messages route to a human, the offline flags are set before classifying, the log holds one line per message with the gate outcome, and nothing in your scripts sends mail or processes refunds. Anything missing sends you back to the step that owns it.

Mistakes that feel like progress

Reading 0.98 as an accuracy score is the first mistake: it measures confidence on synthetic demo data, and three messages prove nothing about your inbox. Lowering the gate until the uncertain case passes is the second: tuning uncalibrated thresholds means labeling real inquiries and measuring, not nudging numbers until the demo looks clean. Letting a draft send itself is the third: the design assumes a person reads every reply, and removing that person removes the safety the gate was built for. Pasting member mail into a cloud chatbot for the same triage is the fourth: it trades away this setup’s entire privacy win for nothing. Renaming your categories without rerunning the batch is the fifth: new descriptions mean new scores, so the determinism check in step 8 starts over.