Guide
What Is Natural Language Processing? A Ticket Routing Example
Follow a natural language processing project from a message to a proposed support queue, with a small example you can score by hand.
By the Space Office team · Published · Updated · 16 min read
Natural language processing is the field of computing that helps machines work with human language. It includes tasks such as classifying a message, extracting a date, or drafting a reply. In this guide, a bicycle shop labels messages that ask to book or change a repair visit. In a scoped Space Office pilot, Lithium can implement a baseline, Carbon can check the results, and Hydrogen reviews the evidence before a human owner decides how to use it.

Key takeaways
- Define the output first. A queue label, an extracted date, and a generated reply call for different checks.
- Write a policy for ambiguous and multi-intent messages before deciding what correct means.
- Compare a simple rule and a candidate on the same held-out messages.
- Inspect precision, recall, review workload, and what happens when a suggestion is wrong.
- Run a shadow pilot and keep staff in control of queue changes.
What does natural language processing do?
A repair shop can receive the same request in several forms. A customer might write “Can I bring it Tuesday?”, “Any slots after work?”, or “Need to move tomorrow’s tune-up.” A routing system has to work with wording that changes while the intended action stays similar. Natural language processing, often shortened to NLP, supplies methods for turning these messages into inputs a computer can use and for producing outputs such as labels, extracted values, translations, or responses. AWS introduction to natural language processing
Start with the shape of the answer the workflow needs. For routing, the result can be one approved queue label and a short reason attached to the message. For extraction, the result could be a time mentioned by the sender and the words that support it. For search, the goal may be to return relevant approved repair policies. A drafted reply has a different job because it introduces wording that a person may read as a promise. You can combine tasks once their responsibilities and failure behavior are clear.
Scroll horizontally to view all columns.
| Task | Possible output | What to check |
|---|---|---|
| Classification | Booking, service question, or review | Does it match the written queue policy? |
| Extraction | “Tuesday” and the matching text span | Is the date explicit? Which Tuesday does the sender mean? |
| Search | Relevant excerpt from an approved repair policy | Does the excerpt address the current request and apply to this shop? |
| Generation | Draft follow-up asking for a preferred date | Does it preserve the facts and avoid confirming an unverified slot? |
Some approaches use rules, such as checking whether a note contains a particular phrase. A conventional text classifier can learn associations between features and labels. One introductory representation counts words or phrases, which can be inexpensive to inspect but may lose word order and finer context. A language model can also propose a label from examples. Choose a baseline and evaluate against actual messages the shop expects to receive. Jurafsky and Martin on text classification and naïve Bayes
Define the queue and its exceptions
In the fictional shop, booking means the sender asks to arrange or move a repair visit. A service question asks about a service or price without asking for a visit. The review queue holds messages that do not give staff enough information to choose the next owner safely, along with exceptions such as a cancellation. This policy assigns all messages to a named destination. It also lets you evaluate one useful binary question across the three queues. Did the system suggest booking when the written policy calls for booking?
Agree on whether you are labeling a single message or the whole thread. “That works for me” is difficult to label when an earlier message contains the proposed slot. Use a short thread excerpt if staff would see it and the data policy permits it. For this example, the unit is the first relevant request in a short thread, and each specimen in the worksheet has the facts the reviewer is allowed to see. Keep replies or duplicate copies of the same thread together when assigning examples to test sets.
Set a rule for competing intents. A customer who asks about the price and requests a visit can go to booking, where staff can answer the pricing question before confirming anything. A customer who asks whether a cancellation cleared and mentions a new day goes to review if the existing booking state is unresolved. If two staff members disagree, record which sentence led them to different results. Update the policy before comparing tools.

Write the category policy so a new employee can apply it without looking at a model answer. Put the actions that carry the most risk in the policy itself. In this shop, booking suggestions never promise a free bay or a particular time. The repair calendar remains the authority. If a message asks about moving an appointment, the person handling it checks the existing reservation and open times before responding. Include the difference between selecting a queue and acting on the request in any implementation brief.
Review the labels on a handful of examples independently. If one staff member reads “Is Tuesday still open?” as a booking request and another reads it as a general question, agree on what a request for availability means. Decide whether the phrase is enough only within a repair thread. Otherwise, either collect the missing context or reserve a review category. Revisit the test set when the policy changes and record which labels changed. This avoids scoring a method against a policy that staff no longer follow.
Build a baseline and prepare a fair comparison
- 1Write the allowed labels and include one straightforward example and one borderline example for each.
- 2Remove records the task does not need, such as payment details and unrelated personal information. Give each thread a stable test ID.
- 3Develop on one set of examples. Reserve a later or separate batch for the final check and keep duplicate or related threads together.
- 4Apply a simple baseline first. This example flags a booking if the lowercased note contains the literal substring “book”, “appointment”, or “slot”.
- 5Score an alternative on the same cases and record its exact version. Preserve a review outcome for low-confidence or conflicting requests.
The literal rule is deliberately small enough to reproduce. It matches “booking” because that word contains “book.” It can also match a question about a previous booking or a sentence refusing an appointment. That weakness is useful evidence about what context the next method needs. When no substring matches, record “no booking suggestion” and leave final routing to staff. The two methods are compared on their booking suggestions only.


Score the worked ticket-routing example
The worksheet contains 16 invented messages. Six should go to booking under the written policy. The baseline emits seven booking suggestions. Four are correct, three are false alarms, and it misses two actual booking requests. A second column contains hand-authored candidate outputs that illustrate a possible result. No classifier was trained or measured here. The candidate suggests booking six times, five correctly. It misses one booking and sends three messages to review in total.
Scroll horizontally to view all columns.
| Count or rate | Literal rule | Illustrative candidate |
|---|---|---|
| Correct booking suggestions | 4 | 5 |
| Incorrect booking suggestions | 3 | 1 |
| Missed actual bookings | 2 | 1 |
| Other messages without a booking suggestion | 7 | 9 |
| Precision | 4 ÷ 7 = 57.1% | 5 ÷ 6 = 83.3% |
| Recall | 4 ÷ 6 = 66.7% | 5 ÷ 6 = 83.3% |
Precision asks how many booking suggestions staff can expect to be correct in the test set. Recall asks how many actual requests the method catches. Both rates use the booking label as the positive category. A “review” output for an actual booking counts as a missed booking suggestion, even if staff eventually route it correctly. Track review volume separately to see the operational effect. Google on classification, precision, and recall
Download the ticket routing worksheet. Recalculate the baseline by searching the message column for the three substrings. For the candidate, count the rows where the proposed label and human label both read booking. Then read the missed case about dropping off a bike Thursday. The candidate sends it to review, so notice how the work moves to a person even when the system declines to suggest booking.


Look at the exact errors before using the overall scores in a proposal. The substring rule marks “I do not want an appointment” as positive. The phrase appears but the sender explicitly rejects the action. It misses “Could I bring my bike in Tuesday?” because the vocabulary does not match. The second example shows why adding one keyword at a time can become hard to maintain. A new term might catch one request while unexpectedly matching unrelated notes. Keep a few examples for each reason in your test plan.
Calculate the number of missed bookings as six minus the true positives, since the eligible test set has six actual bookings. Calculate false alarms as all booking suggestions minus the true positives. If you also report accuracy, the literal rule gets eleven of sixteen booking versus nonbooking decisions right. The illustrative candidate gets fourteen of sixteen. Those scores collapse service and review into nonbooking, so they cannot tell you how many questions reached the correct desk. Use a full queue-by-queue table when assessing all three labels.
Choose when to send something to review
Ask staff which mistake costs more. A false booking suggestion sends a question to staff who may need to reassign it. A missed booking may sit in a slower queue while the customer waits. The shop can tolerate some manual review if staff can clear it each day. Measure the number of queued reviews, the time they sit there, and the reason staff changed each suggestion. A better percentage can still produce an unworkable backlog at a different traffic volume.
A score threshold can move the balance between false alarms and missed requests. A score of 0.8 does not automatically mean an 80% chance that the suggestion is correct. Choose any threshold with development evidence and keep the final batch separate. If the system can abstain, test the review route and what happens if that queue is unavailable. Give staff a usable fallback and keep the original text with the suggestion. Google on classification thresholds

Before applying a new rule, run it on an authorized batch while staff continue their existing process. Compare proposed labels with policy-reviewed labels rather than assuming every current assignment is correct. Check common and uncommon requests, messages with spelling errors, and the languages the shop actually handles. An English test set supports conclusions about its own examples. Collect additional approved cases before extending the system to other message populations.
A useful error log has the message ID, approved label, proposed label, and one reason for the mismatch. For example, “Can I move it to Friday?” may require a prior booking reference. If context was genuinely absent, improve the review route. If the necessary information was present and the system missed it, add an independent test and examine the model or rule. This difference helps staff decide whether to change inputs, the label policy, or the method.
Set an operational stop condition before a pilot. For example, the owner might stop the trial if the review queue cannot be cleared by closing time or if an important class of requests repeatedly goes to the wrong desk. Watch counts by day as well as the average. A holiday message with an unusual volume spike can exceed a staffing limit even when the percentage of uncertain cases looks normal.
Collect feedback where people already work. A reviewer can choose the final label and one reason, such as missing thread context, wrong queue policy, language unsupported, or incorrect inference. Keep timestamps for the suggestion and the decision. When staff can distinguish a bad label from a long queue delay, the team can decide whether to improve the classifier, adjust staffing, or change how messages are presented.
Prepare a scoped review with Space Office
Give Hydrogen the queue policy, sanitization rules, allowed language scope, and a fixture of sample messages. Ask for the substring baseline and a review report that includes counts and representative errors. Lithium, the Full Stack Developer, can implement the approved routing rule in a scoped script or integrate it with a staging application. Carbon, the Senior QA Engineer and Test Strategist, can verify labels, check review cases, and reproduce the counted outcomes. Hydrogen coordinates the work and reviews the results against the brief. Space Office workflow
The human support owner resolves disagreements, defines acceptable queue delays, and approves any change to ticket handling. Supply files or limited test access through the agreed process. The scope can remain a file-based exercise until the owner chooses a pilot. If the shop later wants draft replies, define a separate acceptance check for generated content and human approval before sending. Space Office specialist roster
Start with a label staff can verify
Natural language processing is most useful here when a proposed label makes the next staff decision easier and its mistakes are visible. The bicycle shop can start with one booking label, a simple baseline, and a review queue. Check the confusing messages before celebrating a percentage. Expand only after staff can reproduce the result, clear the review queue, and agree on what to do when the system cannot tell what the customer wants.
Sources and review notes
Reviewed against the linked primary sources on the checked dates. Search features and citations are selected by their respective platforms; no placement is guaranteed.
Frequently asked questions
What is natural language processing?
Natural language processing is the field of computing that develops ways to work with human language. It includes classification, information extraction, language search, translation, and generation. The method and evaluation depend on the output needed.
Does NLP require a large language model?
Rules and conventional statistical classifiers can perform many language tasks. A large language model is another option. Compare results on the same approved examples and include the cost and time of review.
What is the difference between routing and reply drafting?
Routing proposes a label from a defined set. Drafting produces text that may introduce claims or promises. Routing tests compare decisions with the queue policy. Drafting reviews compare claims with approved evidence and inspect the message before use.
How do I score messages routed to review?
Report the number sent to review separately. If an actual booking goes to review, include it among missed booking suggestions for recall. Track whether staff ultimately route it correctly and how long the review takes.
How many messages should I test?
Choose enough approved cases to represent the labels, languages, and exceptions the team expects. A small worksheet teaches the math and exposes policy gaps. Use representative, independent examples to estimate how the method handles real volume.