AI Chatbots

Lesson 14 gave you a one-shot AI button: click, wait, read the answer, done. A chatbot is different — the user keeps typing, the model keeps replying, and every reply depends on what was said before. In this lesson you turn the notes app into a chat with my notes assistant: streaming responses, a persona you control, and a real conversation the model actually remembers.

Part 1 · Why chatbots are different

The single biggest mental shift: the model has no memory. Between two API calls it forgets everything. A chat that "remembers" the last ten turns only remembers because you resend the last ten turns on every call. Memory is your job, not the model's.

Part 2 · The system prompt is your product

The system prompt is the first message in every conversation and the user never sees it. It sets the persona, the rules, the tone, the format, and the guardrails. A generic model plus a strong system prompt beats a fancy model with a weak one — every time.

• Name the role. "Research assistant" > "helpful assistant."
• Scope the knowledge. "Only answer from the notes provided." Prevents freelance hallucinations.
• Pin the format. Length, tone, citation style — decide once, in the system prompt.
• Set the fallback. Tell the model what to do when it doesn't know. Silence is worse than "I don't have a note on that."

Write a system prompt for a notes-chat assistant. It should answer only from the notes I pass in, cite the note title in italics, refuse politely when the notes don't cover the question, and keep replies to 3 short paragraphs max.

Part 3 · Streaming — why it feels 10× faster

A 400-word answer might take 6 seconds to generate. If you wait for the whole thing, that's 6 seconds of a blank screen. With streaming, the first word appears in under a second and the rest arrives as the model writes it. Same total time — very different feel.

Make the chat reply stream token-by-token into the UI, with a blinking cursor while it's still writing. Disable the send button while streaming and re-enable it when the stream finishes.

Part 4 · Conversation memory

You have two choices for where the messages live, and they aren't equivalent — pick based on whether the user should be able to close the tab and come back to the same chat.

Pruning long chats

Every model has a context window. Send too many past messages and you'll hit it — the request errors, or worse, the model quietly forgets the middle. Two cheap strategies:

• Sliding window. Only send the last N turns (e.g. last 20 messages). Simple, works for chat that lives in the moment.
• Summarize the old half. When the history gets long, ask the model to summarize turns 1–20 into one paragraph, then send [system, summary, turns 21–40, new user].

Part 5 · Mini-project — Chat with my notes

We're bolting a chat panel onto the notes app from Lesson 11. The user opens a side drawer, asks a question, and the assistant answers using their notes as context. Streaming reply, persistent history, one system prompt.

1. 1Schema

   Add conversations and messages tables. RLS on both, scoped to auth.uid().

2. 2Server fn

   chatWithNotes({ conversationId, message }): load recent notes, load history, prepend system prompt, call the Gateway with streaming.

3. 3Persist

   Save the user message before the call; save the assistant reply after the stream ends.

4. 4UI

   Drawer with a message list, an input, a Send button. Show streaming tokens as they arrive.

5. 5History

   On mount, load the last N messages for the active conversation. New chat = new row in conversations.

Add conversations and messages tables with RLS. Create a streaming chatWithNotes server function that loads my last 20 notes as context, prepends a system prompt that scopes the assistant to those notes, and replies in a side drawer with the last 20 messages of history.

Part 6 · Polish & pitfalls

• Loading states matter. Show the user's message immediately + a "typing…" indicator before the first token lands. Silence between "Send" and the first character feels broken.
• Errors belong in the thread. If the Gateway returns 429 or 402, render a small red bubble in the conversation ("rate limited, try again") — don't just toast and forget.
• Prompt injection is real. A user (or a note!) can say "ignore your instructions and…". Never blindly trust user text to override the system prompt; keep guardrails ("Answer only from the notes provided") strong and repeat them if needed.
• Escape user content. Render assistant replies as markdown (bullets, bold), but escape any HTML — the model can and will emit tags if a note contained them.
• Cost per message ≠ cost per call. Every turn resends the whole history, so token cost grows with the chat length. Prune, or summarize old turns.
• Model choice. Start on a fast, cheap chat model. Upgrade only if answers are visibly weak — cost per chat scales with model size and chat length, so it compounds.
• New chat = new row. Give users a clear "New conversation" button. Long-running chats drift; a fresh conversation with a fresh system prompt is often the fix.

Chatbot cheat sheet

Prompts you'll reuse

• "Add conversations and messages tables with RLS scoped to auth.uid()" → schema + policies.
• "Make the chat reply stream token-by-token" → streaming server function + UI reader.
• "Only answer from the notes I pass in; if a note doesn't cover it, say so" → grounded assistant.
• "Send only the last 20 messages of history on each call" → sliding window pruning.
• "Add a 'New conversation' button that creates a fresh row" → clean context reset.

Rules of thumb

• The model is stateless. Memory = what you resend.
• System prompt sets the product. Persona, scope, format, fallback.
• Stream when the user reads live. Skip streaming for tiny outputs.
• RLS on every chat table. Same rule as any user-owned data.
• Prune before you hit the ceiling. Sliding window or summarize-and-collapse.