Lab 1: Build a Support-Ticket Triage Pipeline

Every support team faces the same firehose: tickets arrive in plain language and must be routed — billing questions to finance, bugs to engineering, account issues to ops. In this lab you build an automatic triage pipeline from scratch, measure how good it is, and improve it through deliberate iteration. No machine learning libraries, no APIs: just the standard library and your own thinking.

CT concepts fused in this lab: decomposition (splitting triage into stages: data, rules, classify, evaluate), pattern recognition (finding the language patterns that distinguish categories — and the patterns in your classifier’s failures), and algorithm design (turning rules into a classification procedure, then redesigning it twice). Evaluation & debugging runs through the whole exercise.

You need: Python 3.9+ and a terminal. Create a file called triage.py; you will grow it step by step. Time: roughly 60 minutes.

Decompose the problem

Before writing code, split “triage tickets automatically” into parts that can be built and tested separately:

Stage	Responsibility	Testable on its own?
Data	Labeled example tickets — the ground truth	Yes: read them, check labels
Rules	Knowledge of what language signals each category	Yes: inspect for overlaps
Classifier	Procedure applying rules to one ticket	Yes: feed it single tickets
Evaluator	Score the classifier against ground truth	Yes: verify with a known case

This decomposition is the lab’s skeleton — each remaining step builds one stage. Notice what it buys you immediately: when accuracy is bad later, you will know which stage to blame.

Build the ground truth

Start triage.py with labeled data. Twenty tickets, four categories — small enough to read, large enough to score:

TICKETS = [
    ("I was charged twice for my subscription this month", "billing"),
    ("The app crashes every time I open the settings page", "bug"),
    ("How do I reset my password? The email never arrives", "account"),
    ("Please add a dark mode, my eyes hurt at night", "feature"),
    ("My invoice shows the wrong company name", "billing"),
    ("Export to CSV produces an empty file", "bug"),
    ("I cannot log in after enabling two-factor authentication", "account"),
    ("It would be great to have keyboard shortcuts for search", "feature"),
    ("Why did my payment fail? My card is valid", "billing"),
    ("The dashboard shows an error 500 after the last update", "bug"),
    ("I want to change the email address on my profile", "account"),
    ("Can you support importing data from spreadsheets?", "feature"),
    ("I need a refund for the annual plan I bought yesterday", "billing"),
    ("Uploaded images appear rotated on mobile devices", "bug"),
    ("My account was locked after too many login attempts", "account"),
    ("Add an option to schedule reports weekly please", "feature"),
    ("The price on the checkout page does not match the plan page", "billing"),
    ("Notifications stopped working since the crash yesterday", "bug"),
    ("Please delete my account and all associated data", "account"),
    ("A calendar view would make planning so much easier", "feature"),
]

Before moving on, do the human version of the task: read five tickets and note which words told you the category. That noticing is pattern recognition — you are about to externalize it into rules.

Version 1 — first match wins

Add your first classifier and evaluator:

RULES_V1 = {
    "bug": ["crash", "error", "fail"],
    "account": ["password", "login", "account", "email"],
    "billing": ["charge", "invoice", "refund", "payment"],
    "feature": ["add", "feature"],
}

def classify_v1(text, rules):
    text = text.lower()
    for category, keywords in rules.items():
        for kw in keywords:
            if kw in text:
                return category
    return "unknown"

def evaluate(classify, rules, tickets):
    hits = 0
    for text, expected in tickets:
        got = classify(text, rules)
        if got == expected:
            hits += 1
        else:
            print(f"  MISS: expected {expected:8s} got {got:8s} | {text}")
    print(f"Accuracy: {hits}/{len(tickets)} = {hits/len(tickets):.0%}")

print("=== v1: first match wins ===")
evaluate(classify_v1, RULES_V1, TICKETS)

Predict your accuracy, then run python3 triage.py. You should see 12/20 = 60% and eight misses.

Read the failures like data

Do not fix anything yet. The misses are a dataset — find the patterns in them:

Most misses say got unknown: your keyword lists are simply incomplete (“empty file”, “rotated”, “two-factor” match nothing).
One miss is worse than incomplete — it is wrong: “Why did my payment fail?” was routed to bug. The ticket contains both “fail” and “payment”, and because bug sits first in the dictionary, first-match-wins never even considers billing. Your algorithm design — not your keywords — caused this one.

Two failure patterns, two different fixes required. Diagnosing before patching is the evaluation habit this lab exists to build.

Version 2 — score, don’t race

Fix the design flaw first: instead of first-match-wins, let every category score, and take the best. Then enrich the keywords. Append:

RULES_V2 = {
    "billing": ["charge", "invoice", "payment", "refund", "price",
                "plan", "subscription"],
    "bug": ["crash", "error", "broken", "fail", "empty file",
            "stopped working", "rotated"],
    "account": ["password", "log in", "login", "locked", "profile",
                "delete my account", "two-factor"],
    "feature": ["add", "would be great", "option to", "view would",
                "can you support", "shortcuts"],
}

def classify_v2(text, rules):
    text = text.lower()
    scores = {cat: sum(kw in text for kw in kws)
              for cat, kws in rules.items()}
    best = max(scores, key=scores.get)
    return best if scores[best] > 0 else "unknown"

print("=== v2: scoring classifier ===")
evaluate(classify_v2, RULES_V2, TICKETS)

Run it: 19/20 = 95%. One stubborn miss remains:

MISS: expected feature  got billing  | A calendar view would make planning so much easier

Zoom in. Why would a calendar request look like a billing ticket? Because "plan" in "planning" is True — substring matching found a billing keyword inside another word. Your pattern matcher recognized a pattern that is not really there.

Version 3 — match words, not fragments

Require word boundaries using the standard library’s re:

import re

RULES_V3 = {cat: list(kws) for cat, kws in RULES_V2.items()}

def classify_v3(text, rules):
    text = text.lower()
    scores = {}
    for cat, kws in rules.items():
        scores[cat] = sum(
            bool(re.search(r"\b" + re.escape(kw) + r"\b", text))
            for kw in kws)
    best = max(scores, key=scores.get)
    return best if scores[best] > 0 else "unknown"

print("=== v3: word-boundary matching ===")
evaluate(classify_v3, RULES_V3, TICKETS)

Run it — and enjoy the surprise: still 95%, but the miss moved:

MISS: expected bug      got unknown  | The app crashes every time I open the settings page

Your fix regressed something. With boundaries enforced, crash no longer matches crashes. This is the most valuable moment in the lab: every change can break what already worked, which is why you re-run the whole evaluation after every change — never just the case you were fixing. Add the missing form:

RULES_V3["bug"].append("crashes")

Re-run: 20/20 = 100%.

Stress-test your 100%

A perfect score on data you tuned against proves little — you may have memorized your twenty tickets rather than learned the categories. Write three fresh tickets you have never used, predict the output, then check:

NEW_TICKETS = [
    ("The mobile app freezes on the login screen", "bug"),
    ("Charge appeared on my statement after I cancelled", "billing"),
    ("Would love an integration with my calendar app", "feature"),
]
print("=== hold-out check ===")
evaluate(classify_v3, RULES_V3, NEW_TICKETS)

Expect imperfection (“freezes” is in no keyword list — and notice the first ticket contains “login”, pulling it toward account). That gap between tuning-set accuracy and fresh-data accuracy is the core idea behind train/test splits in machine learning — you have just rediscovered why they exist.

What you actually practiced

You decomposed a fuzzy task into testable stages; you extracted linguistic patterns into explicit rules twice — once from tickets, once from your own failures; and you redesigned an algorithm when measurement proved the design wrong. The loop you ran — measure, diagnose, change one thing, re-measure everything — is the same loop used to tune trillion-parameter models. Only the classifier got fancier; the thinking is identical.

Going further

Priorities: add an urgency field (e.g., “locked”, “charged twice” are urgent) and route to a queue — a second classifier composed with the first.
Tie-breaks: max silently prefers earlier dictionary order on ties. Make ties explicit: return "needs_human" when two categories tie — an honest classifier knows when it does not know.
Weighted rules: let strong signals (“refund”) count more than weak ones (“plan”). You are now one step from naive Bayes.
Scale test: generate 200 tickets by combining fragments, and see which rules stop generalizing.
Connect the concepts: revisit Foundations for the theory behind decomposition and pattern recognition, and Lab 3 to apply the same evaluation loop to AI outputs.

Lab 2: Simulate an Epidemic