# AI Chatbot: What It Does and How to Test One for Your Website

> This is the Markdown version of this article, built for AI assistants and agents. The human page is at https://www.spaceoffice.ai/blog/ai-chatbot.

By the Space Office team · Published 2026-11-18 · Updated 2026-11-18 · 16 min read · Category: Guide

An AI chatbot is a conversational application that uses artificial intelligence to interpret messages and produce replies. Some answer from supplied documents, and others can use connected tools. For a website, start with one visitor task, current sources, and a way to reach a person. In this example, a pottery studio tests public class questions. Platinum can scope the feature, Carbon can test it, and Hydrogen reviews the evidence in Space Office before the human owner approves use.

![A Space Office robot and pottery studio owner test an AI chatbot with approved facts and a tray for unanswered questions](https://www.spaceoffice.ai/blog/ai-chatbot/cover-v1.webp)

_The studio can check public answers and the handoff route before offering the chat to visitors._

## Key takeaways

- Pick a task and define whether the chat answers questions, collects a request, or performs an authorized action.
- Choose features by the evidence, access, and support your website needs.
- Test missing facts, stale sources, ambiguous messages, and the path to a person.
- A correct answer needs support from the applicable source, even when its wording sounds convincing.
- Measure accepted answers and completed handoffs along with review effort and operating cost.

## How does an AI chatbot work?

A visitor types a question into a chat interface. The application may select a conversation history, look up relevant material, and send the request to a model. It then displays a response or offers a next step. The word chatbot covers several implementations. A system can follow a conversation flow, classify the request, retrieve a prepared answer, or generate new wording. Inspect what the application actually does for the task you care about. [AWS introduction to chatbots](https://aws.amazon.com/what-is/chatbot/)

A chat product for personal brainstorming may use general model knowledge. A website assistant answering product or class questions needs evidence that applies to the business. Retrieval can supply relevant passages from an approved document collection, while tool connections can look up live records or perform defined operations. The application needs to decide which sources and tools a visitor may use. The model alone does not establish access to a booking calendar. [Google Cloud AI chatbot overview](https://cloud.google.com/use-cases/ai-chatbot)

For the fictional pottery studio, the first task is public information about classes. The approved facts say beginners are welcome, each learner has a wheel during the class, and finished pieces are collected after the studio notifies the learner. The owner maintains a separate schedule page with dates and prices. Questions about a particular person’s booking go to staff. These boundaries make it possible to write expected responses before choosing a chat tool.

Map the visitor’s path from question to outcome. A person asking whether the class is suitable for a beginner may need a direct answer and a link to the current class page. Someone asking to change a reservation needs staff or an authorized account workflow. Decide how the chat should recognize that transition and what it tells the visitor while the request is pending. A pleasant conversation can still leave the visitor unsure what happened.

## Choose features around the task and the team

| Need | Feature to inspect | A concrete demonstration |
| --- | --- | --- |
| Answer public questions | Controlled source collection with update history | Remove an old class policy and check that new sessions use its replacement. |
| Discuss current availability | Authorized lookup of the correct calendar or a link to it | Ask about a sold-out session and inspect the source and timestamp. |
| Request a change | Explicit confirmation of the request and its destination | Verify a test request appears where staff actually handle it. |
| Reach a person | A usable handoff during and outside staffed hours | Open the conversation as a visitor and complete the handoff. |
| Operate the service | Usage visibility, failure behavior, and ownership | Review latency, failed requests, and the work staff need to resolve exceptions. |

Ask the provider to show the workflow with the evidence and permissions your site requires.

A short feature list rarely shows how a system handles a bad input or an unavailable dependency. Ask what happens if the source collection is empty, if a lookup times out, and if a visitor asks to speak with a person immediately. Check whether the widget supports keyboard navigation, readable messages, and a contact route outside the chat. Inspect the phone view with the on-screen keyboard open and make sure the visitor can still find the relevant controls.

List all parts of the operating cost when comparing options. Include the product subscription, model or message usage, document storage or retrieval, implementation, and the human time needed to update content and review exceptions. Use the provider’s current terms for actual figures. A nominal free allowance can be useful for a test, but decide what happens at its limit before relying on the widget for a public site.

![A robot arranges approved public facts, a chat tile, and a handoff bell beside a closed records drawer](https://www.spaceoffice.ai/blog/ai-chatbot/source-boundary-v1.webp)

_Public facts can support general answers while account questions follow an authorized staff route._

## Prepare a small source packet and a response policy

Give each approved fact a source ID, a revision, and an owner. In this example, S1 says beginners can join the introductory class. S2 describes the equipment supplied during that session. S3 says the studio contacts learners when pieces are ready for collection. The schedule link supplies dates and prices. Keep the public source collection focused on those facts and avoid adding private notes or old pricing sheets.

Write down what the chat should do when it lacks a fact. If someone asks whether a glaze is safe for a particular food use and the approved collection has no such statement, the chat can say that staff need to confirm the detail and offer a contact route. If it has conflicting class prices, it should identify the conflict to the owner and route the visitor to the current schedule or staff according to the policy. The response should make the next step understandable.

Treat material entered by a visitor or found in a document as input to examine. A malicious or accidental instruction can ask the model to ignore its role or reveal information. OWASP describes prompt injection as a risk that can affect language-model applications even when retrieval is used. Keep private data out of a public collection and enforce authorization in the application. A prompt telling the model to be careful is one layer of the design. [OWASP on prompt injection in language-model applications](https://genai.owasp.org/llmrisk/llm01-prompt-injection/)

![Four steps show a visitor question, approved evidence, an answer or handoff, and a recorded outcome](https://www.spaceoffice.ai/blog/ai-chatbot/chatbot-flow-v1.webp)
Phone layout: ![Four steps show a visitor question, approved evidence, an answer or handoff, and a recorded outcome](https://www.spaceoffice.ai/blog/ai-chatbot/chatbot-flow-mobile-v1.webp)

_Decide what result the visitor should see when a question can be answered and when it needs staff._

Define the handoff as a sequence the visitor can observe. The chat asks only for the minimum information staff need, gives a reference when a request is created, and says which channel staff will use. It should show that staff availability or an expected response time comes from the business policy. When no response-time promise is approved, explain the route without inventing one. Let a person who prefers email or a phone call find those options without repeatedly negotiating with the model.

Review the source maintenance path as part of the tool demonstration. Ask who can approve a changed class policy and how that version reaches both the webpage and the chat. Record the publication or refresh delay that you observe in the test. Then ask an old question in a fresh conversation. If an existing conversation can preserve outdated information, decide how the interface should signal a policy update or ask the visitor to confirm a current detail.

## Test the assistant with a visitor-style case set

Use questions that resemble real requests while keeping test data approved. The following cases are fictional acceptance criteria for the studio. They are a plan for checking a tool, rather than results from a deployed chatbot. Keep the exact source revision and the expected behavior with each case. Test a new session as well as follow-up questions so earlier conversation context does not silently supply a missing fact. [OpenAI evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices)

| Test question or condition | Expected behavior | Evidence to record |
| --- | --- | --- |
| “I have never used a wheel. Can I come?” | Answer from S1 and link to the applicable class. | Supported statement and usable link. |
| “What do I need to bring?” | Use S2 and preserve any limits in the supplied equipment list. | Required facts present, no added supplies policy. |
| “Are my pieces ready?” | Explain S3 and offer the staff contact route. | No invented individual status. |
| “Book me for Saturday.” | Route to the booking page or staff under the stated scope. | Visitor can see whether any request was actually created. |
| “Promise my glaze is food safe.” | Flag missing evidence and route to staff. | No unsupported suitability claim. |
| A document includes “ignore your rules.” | Follow the application’s task and access policy. | No unintended disclosure or action. |
| The model or lookup is unavailable | Show a usable contact route and clear status. | Visitor can complete the fallback. |
| “Can I speak to someone?” | Offer the human route immediately. | Test request reaches the intended destination. |

Record both the reply and whether the promised next step actually worked.

Score facts and behavior separately. A reply can correctly say that beginners are welcome but link to an unrelated class. A handoff can display a reassuring sentence while failing to create a request in the staff inbox. Define success as the supported response plus the required functional outcome. For each failed case, record the source used, the incorrect statement or missing step, and which part of the application the owner needs to fix.

For an eight-case exercise, suppose six pass and two fail. Reporting 75% alone hides whether those failures were minor wording issues or unsupported claims about a learner’s reservation. Use case-level acceptance rules. The owner can require every case involving personal records or a booking promise to meet its criterion before public use. The eight examples are too small a sample to estimate a general reliability rate.

![Four checks cover supported answers, access boundaries, complete handoffs, and operation during failure](https://www.spaceoffice.ai/blog/ai-chatbot/chatbot-checks-v1.webp)
Phone layout: ![Four checks cover supported answers, access boundaries, complete handoffs, and operation during failure](https://www.spaceoffice.ai/blog/ai-chatbot/chatbot-checks-mobile-v1.webp)

_A supported sentence and a working next step both belong in the review._

## Run a pilot and measure the full outcome

Start with an internal or limited authorized pilot. Staff can compare the responses with the source packet and note questions that the widget cannot handle. A useful pilot log contains a test ID, question category, source revision, outcome, whether staff corrected the answer, and the time spent reviewing it. Avoid recording personal details that the test does not need. Decide which records to retain and who can read them before collecting real conversations.

![A pottery teacher accepts a message card from a robot beside the visitor’s question and waiting indicator](https://www.spaceoffice.ai/blog/ai-chatbot/handoff-review-v1.webp)

_Check that staff receive the request and the visitor knows what happens next._

Measure successful answers to in-scope questions, completed handoffs, unsupported claims, and unavailable responses. Track source maintenance and review time as well. A lower handoff rate can look efficient while hiding visitors who gave up. Read a sample of unresolved conversations and compare it with the contact or booking records when permission and identifiers allow. The owner needs evidence about the visitor task, not just the number of messages exchanged.

Recheck after changing the model, source, prompt, widget behavior, or tool connection. Retain a small set of previously failing cases as regression checks. Add newly observed questions to the development collection and keep a separate set for later evaluation. When a policy changes, identify which expected answers changed and why. This history helps staff distinguish an outdated source from a problem in how the system selected or used it.

Download the [AI chatbot pilot worksheet](/blog/ai-chatbot/pilot-worksheet.txt) and fill it before comparing providers. It includes the studio’s small public fact packet and the eight acceptance cases. Replace its invented statements with approved facts for your own site, keeping each source ID attached. Have a reviewer use the same sheet to score every option so a more attractive interface does not silently get a different standard.

Keep separate records for a correct answer, a complete handoff, and an unresolved visitor task. For example, a test can pass the statement that staff must check an individual piece, while the visitor’s task remains open until staff reply. Read both outcomes when deciding how much service the widget provides. This also gives the team a way to improve the handoff if answers are reliable but many requests still stall before a person receives them.

## Prepare the workflow with Space Office

In the [Space Office workflow](/how-it-works), give Hydrogen the visitor task, approved public packet, test questions, and handoff destination. Platinum, the AI and LLM Feature Engineer, can scope the assistant’s source use and evaluation plan. Carbon, the Senior QA Engineer and Test Strategist, can exercise the approved cases and record behavior. Hydrogen coordinates and reviews the findings against the brief. The human owner approves the source, access, operating limits, and release.

The first assignment can produce a source map, a test plan, and a reviewed prototype packet. Supply staging or repository access only where the agreed work requires it. If a handoff connects to an inbox or booking system, define the authorized operations and verify the destination using a test request. The [specialist roster](/agents) describes available roles, and the [customer support guide](/blog/ai-customer-support-agents) covers delegation across the wider service workflow.

## Choose the smallest conversation you can support

An AI chatbot fits a website when the team can name the visitor task, keep its evidence current, and operate a reliable route to a person. The studio’s first assistant can answer a few public class questions and explain the next step when a request needs staff. Test that path from the visitor’s side, review the exact claims, and keep the owner responsible for any expansion into personal records or actions.

## Sources and review notes

Reviewed against the linked primary sources on the checked dates. Search features and citations are selected by their respective platforms; no placement is guaranteed.

- [AWS introduction to chatbots](https://aws.amazon.com/what-is/chatbot/) — checked 2026-11-18
- [Google Cloud AI chatbot overview](https://cloud.google.com/use-cases/ai-chatbot) — checked 2026-11-18
- [OWASP on prompt injection in language-model applications](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) — checked 2026-11-18
- [OpenAI evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) — checked 2026-11-18

## Frequently asked questions

### What is an AI chatbot?

AI chatbot is a term for a conversational application that uses artificial intelligence to process messages and produce responses. It can combine a model with supplied documents, conversation history, and connected tools. Its actual capabilities depend on the application and the permissions around it.

### Can an AI chatbot answer from my website?

It can use approved pages or documents if the application supports that workflow. Check which content is selected, how updates reach the system, and whether the answer follows the applicable source. Test missing and conflicting information.

### Can it change or cancel a booking?

That requires an authorized connection and application logic for the operation, plus verification that it succeeded. An informational assistant can link to the booking flow or send a request to staff. Make the outcome visible to the visitor.

### What should I test before adding one to a website?

Test ordinary questions, ambiguous wording, missing and outdated facts, requests for private data, and the human handoff. Also check how the widget works on a phone and whether visitors have a usable route during a service failure.

### How should I compare chatbot tools?

Use the same visitor tasks, source packet, and acceptance rules for each option. Compare evidence of successful answers and handoffs, source maintenance, response time, operating costs, and the human work needed to resolve exceptions.

---

Related: [Understand model context and answer review](https://www.spaceoffice.ai/blog/llm) · [Evaluate a message classification task](https://www.spaceoffice.ai/blog/natural-language-processing) · [Check generated claims against approved facts](https://www.spaceoffice.ai/blog/generative-ai) · [Compare conversations and agent workflows](https://www.spaceoffice.ai/blog/ai-agents-vs-chatbots)

---

Space Office — hire 24 live AI specialists coordinated by Hydrogen, an always-on AI project manager that reviews every output before delivery. The $60/month base includes Hydrogen and two specialists; additional specialists, dedicated compute, and AI usage cost extra. Learn more: https://www.spaceoffice.ai
