Imagine you're building a customer support tool for a SaaS company. You fire up the OpenAI API, send a user's question, and get back a perfectly reasonable answer — except it's written like a philosophy essay, it reveals internal pricing logic the company considers proprietary, and it completely ignores the specific tone guide in the brand handbook. The model is smart enough. The problem is you haven't given it the right instructions, in the right structure, at the right time.
This is where conversation design comes in. The Chat Completions API — the foundation of ChatGPT and tools like it — doesn't just accept a single question and return an answer. It accepts a structured conversation thread made up of multiple messages, each assigned to a specific role. How you assemble that thread is one of the most consequential decisions you'll make when building with LLMs. Get it right, and the model behaves like a focused, reliable specialist. Get it wrong, and you're fighting the model on every call.
By the end of this lesson, you'll understand exactly how the Chat Completions API structures conversations and why, and you'll be able to build well-designed message threads from scratch.
What you'll learn:
openai Python library installed (pip install openai)Before you write a single line of prompt text, you need a mental model of what you're actually sending to the API.
When you call the Chat Completions endpoint, you're not sending a string of text. You're sending a list of message objects. Each message object has two fields: a role and content. The role tells the model who is speaking, and the content is what they said.
Here's the simplest possible call:
from openai import OpenAI
client = OpenAI() # uses your OPENAI_API_KEY environment variable
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "user", "content": "What's the difference between a data lake and a data warehouse?"}
]
)
print(response.choices[0].message.content)
That works, but it's the API equivalent of walking up to a stranger and asking them a technical question with zero context. The model will answer, but it has no idea who you are, what you need the answer for, what level of detail is appropriate, or what tone to use.
The power of the Chat Completions API comes from building a richer, more deliberate thread. To do that, you need to understand the three roles.
Think of a conversation in the API as a script, and the roles as the characters in that script. There are exactly three:
system — This is your backstage director. The system message sets the stage before any real conversation begins. It tells the model what it is, how it should behave, what it can and can't talk about, what tone to use, and any domain-specific facts it should treat as ground truth. The user never sees this message (in a well-built application, anyway). It's purely instructional.
user — This is the human in the conversation. User messages represent inputs from the person interacting with your application. This is where questions, commands, and data live.
assistant — This is the model's voice. When the model responds, its output is labeled as an assistant message. Here's the key insight: when you're building a multi-turn conversation, you include previous assistant responses in the thread so the model can see what it already said. This is how context is maintained.
Let's see all three in action:
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": "You are a senior data engineer helping junior analysts at a fintech company. Explain technical concepts clearly and concisely. Use plain language, avoid jargon where possible, and always give a practical example. Do not discuss competitor products."
},
{
"role": "user",
"content": "Can you explain what database indexing is?"
}
]
)
That system message changes everything. The model now knows its audience (junior analysts), its style (clear, concise, practical examples), and its constraints (no competitor discussion). The same underlying model produces a very different output with that context than without it.
The system prompt is the single highest-leverage piece of text in your application. A thoughtful system prompt can eliminate entire categories of problems. A vague one forces you to patch issues downstream forever.
Good system prompts cover four things:
1. Identity and role — Who is the model, and what is its job?
You are a SQL assistant for Meridian Analytics, a business intelligence consultancy.
Your job is to help data analysts write, debug, and optimize SQL queries.
2. Behavioral guidelines — How should it communicate?
Always explain your reasoning before giving a final query. When a user's question is
ambiguous, ask a clarifying question rather than guessing. Keep explanations under
150 words unless the user asks for more detail.
3. Domain constraints — What should it know or avoid?
The company uses BigQuery as its primary data warehouse. All SQL examples should
use standard BigQuery syntax (not MySQL or PostgreSQL syntax). Do not write queries
that use SELECT * — always specify column names explicitly.
4. Output format — What shape should responses take?
When providing a SQL query, format it in a fenced code block. Follow the query
with a brief plain-language explanation of what it does.
Put it all together:
system_prompt = """
You are a SQL assistant for Meridian Analytics, a business intelligence consultancy.
Your job is to help data analysts write, debug, and optimize SQL queries.
Communication style:
- Explain your reasoning before providing a final query
- Ask a clarifying question if the user's request is ambiguous
- Keep explanations concise (under 150 words) unless asked for more detail
Technical constraints:
- All SQL must use BigQuery syntax
- Never use SELECT * — always name columns explicitly
- When referencing table names, use the format `project.dataset.table`
Output format:
- Present SQL in a fenced code block with the sql language tag
- Follow every query with a brief plain-language explanation
"""
Notice this isn't a wall of rules — it's organized, scannable, and specific. Models respond well to structure within the system prompt itself. Headers and bullet points are completely valid inside a system message.
Tip: Be specific rather than abstract. "Be helpful and accurate" means nothing. "If you're not confident about a specific BigQuery function, say so and suggest the user verify in the official documentation" is something the model can actually act on.
Here's the thing that surprises most people new to the API: the model has no memory between API calls. Every call is completely stateless. If you want the model to remember what was said two messages ago, you have to send those messages again.
This means building a multi-turn conversation is really just the act of maintaining a list and appending to it as the conversation progresses.
Let's build a simple conversation loop to make this concrete:
from openai import OpenAI
client = OpenAI()
# Start with just the system message
conversation_history = [
{
"role": "system",
"content": """You are a data quality analyst assistant for a retail company.
Help analysts identify and resolve data quality issues in their datasets.
Be specific and practical in your recommendations."""
}
]
def chat(user_message):
# Add the new user message to history
conversation_history.append({
"role": "user",
"content": user_message
})
# Call the API with the full conversation history
response = client.chat.completions.create(
model="gpt-4o",
messages=conversation_history
)
assistant_reply = response.choices[0].message.content
# Add the assistant's response to history so future turns remember it
conversation_history.append({
"role": "assistant",
"content": assistant_reply
})
return assistant_reply
# Simulate a multi-turn conversation
print(chat("I have a customer table with about 15% null values in the email column. What should I check first?"))
print("---")
print(chat("Good point. I ran that check and it turns out nulls are clustered in records from before 2021. Is that a meaningful pattern?"))
print("---")
print(chat("How would I write a SQL query to flag all those pre-2021 records with null emails?"))
Notice what happens in that third turn. The user says "those pre-2021 records" — a reference that only makes sense given the conversation history. Because we're sending the full conversation_history list on every call, the model has all the context it needs to understand what "those" refers to. Remove the history, and the model would be lost.
Warning: Conversation history grows with every turn, and the model has a finite context window — a maximum amount of text it can process in one call. For GPT-4o, this is 128,000 tokens (roughly 96,000 words), which is enormous. But for long-running applications — think a support chat session that goes on for an hour — you'll eventually need a strategy for trimming or summarizing old history. For now, just be aware the limit exists.
One of the most powerful patterns in LLM application design is context injection: dynamically inserting relevant data into the message thread before you call the API. This is how you connect LLMs to real information.
Suppose you're building a tool that helps sales reps prepare for customer calls. Before the call, you want the model to be aware of the specific account's history. You might pull that data from a CRM and inject it directly into the conversation.
def prepare_call_briefing(account_data: dict, rep_question: str) -> str:
# Dynamically build a context block from real data
context_block = f"""
Account: {account_data['company_name']}
Plan: {account_data['plan_tier']}
Monthly spend: ${account_data['monthly_spend']:,.2f}
Contract renewal: {account_data['renewal_date']}
Open support tickets: {account_data['open_tickets']}
Last interaction: {account_data['last_interaction_summary']}
"""
messages = [
{
"role": "system",
"content": """You are a sales intelligence assistant. Your job is to help
account executives prepare for customer calls. Be concise and strategic.
Focus on renewal risk, expansion opportunities, and relationship health."""
},
{
"role": "user",
"content": f"Here is the account profile for my upcoming call:\n\n{context_block}\n\nMy question: {rep_question}"
}
]
response = client.chat.completions.create(
model="gpt-4o",
messages=messages
)
return response.choices[0].message.content
# Usage
account = {
"company_name": "Brightfield Logistics",
"plan_tier": "Enterprise",
"monthly_spend": 12400,
"renewal_date": "2025-09-01",
"open_tickets": 3,
"last_interaction_summary": "CSM call in April; expressed frustration with API rate limits"
}
print(prepare_call_briefing(
account,
"What should I focus on in this renewal call, and what are the red flags?"
))
This pattern scales to almost anything: retrieved documents, database query results, user profile data, sensor readings. You're not fine-tuning the model or doing anything exotic — you're just putting the right information in the message at the right time.
Tip: When injecting large blocks of context, clearly label them. Use headings like
### Account Dataor### Retrieved Documentswithin the message content. This helps the model distinguish between data and instructions.
There's one more powerful technique worth understanding: you can write assistant messages yourself, without the model having generated them. This is called seeding the conversation or priming, and it lets you shape how the model continues.
For example, if you want the model to always begin its response in a specific format, you can include a partially-written assistant message and let the model complete it. Some APIs support this directly; others require you to structure the prior turn to lead the model there.
More practically, you can use fabricated assistant messages to establish a prior "agreement" with the model:
messages = [
{
"role": "system",
"content": "You are a senior data analyst reviewing ETL pipeline code."
},
{
"role": "user",
"content": "I'd like you to review my Python ETL code and give feedback on structure, error handling, and performance. Can we use a consistent format for your feedback?"
},
{
"role": "assistant",
"content": "Absolutely. For each piece of code you share, I'll structure my feedback in three sections: **Structure** (organization and readability), **Error Handling** (robustness and edge cases), and **Performance** (efficiency and scalability). Ready when you are."
},
{
"role": "user",
"content": "Great. Here's the first function:\n\n```python\ndef load_to_warehouse(df, table_name):\n conn = get_connection()\n df.to_sql(table_name, conn)\n conn.close()\n```"
}
]
By inserting that pre-written assistant message, you've established a format agreement that the model will now follow reliably for the rest of the conversation, without needing to repeat it in every turn.
Warning: Don't use fabricated assistant messages to make the model "agree" to bypass its safety guidelines. That's both ineffective and against usage policies. The technique is for formatting and workflow conventions, not jailbreaking.
Let's put everything together. Build a multi-turn research assistant for a data analyst who is exploring an unfamiliar dataset.
Your task: Write a Python script that:
Sets up a system prompt defining the assistant as a "data exploration guide" that helps analysts understand unfamiliar datasets. It should ask clarifying questions, suggest relevant analyses, and explain statistical concepts in plain language.
Starts the conversation by injecting a realistic dataset description as a user message (make one up — something like a monthly sales table with 15 columns, three years of data, and some known quality issues).
Runs a 3-turn conversation loop where you simulate the analyst asking:
After each assistant response, print the role and content clearly so you can see the full thread growing.
Stretch goal: Modify the script to print the full conversation_history list after the final turn. Count the total number of messages. Notice how the thread includes system, user, and assistant messages interspersed — that's the full context the model sees on every call.
Mistake: Putting everything in the user message instead of the system message
New builders often write long instructions inside the user message because it feels more natural. The problem is that user-message instructions compete with the actual user input and can get confused or overridden in later turns. Persistent behavioral instructions belong in the system message, where they set the frame for the entire conversation.
Mistake: Forgetting to append assistant responses to history
This is the most common bug in multi-turn applications. The conversation feels coherent for the first two turns, then the model starts acting like it has no memory. Check your code — you almost certainly forgot to append {"role": "assistant", "content": assistant_reply} after each call.
Mistake: System prompts that are vague or contradictory
"Be helpful but also be concise but also be thorough" is not a useful instruction. When you have competing directives, the model will do its best to average them out, which usually satisfies neither goal. Prioritize explicitly: "Be concise by default. If the user asks for more detail, expand your response."
Mistake: Not handling the context window
If you're building a persistent chat application and you're not tracking the approximate size of your conversation history, you will eventually hit the context limit and get an API error. Build a simple token counter using the tiktoken library and trim or summarize old messages when you approach the limit.
Mistake: Injecting data without labeling it
When you inject a block of database results or retrieved documents into a user message, the model can struggle to distinguish the injected data from the user's actual question. Always use clear labels and separators.
# ❌ Hard to parse
content = f"{retrieved_docs}\n\nWhat are the main themes in these documents?"
# ✅ Clearly labeled
content = f"### Retrieved Documents\n\n{retrieved_docs}\n\n### Question\n\nWhat are the main themes in these documents?"
You've just covered the architectural core of every LLM-powered application. Let's recap what you now know:
role and content.These patterns aren't just academic. Every serious LLM application — from code assistants to customer support bots to data analysis tools — is built on exactly this foundation.
Where to go next:
The conversation design skills you've built here are the foundation for all of it.
Learning Path: Building with LLMs