Call GPT-4, control temperature, and track token usage.
This guide explains the fundamental concepts behind working with OpenAI's language models, which form the foundation for building AI agents.
The OpenAI API provides programmatic access to powerful language models like GPT-4o and GPT-3.5-turbo. Instead of running models locally, you send requests to OpenAI's servers and receive responses.
Key characteristics:
Comparison with Local LLMs (like node-llama-cpp):
| Aspect | OpenAI API | Local LLMs |
|---|---|---|
| Setup | API key only | Download models, need GPU/RAM |
| Cost | Pay per token | Free after initial setup |
| Performance | Consistent, high-quality | Depends on your hardware |
| Privacy | Data sent to OpenAI | Completely local/private |
| Scalability | Unlimited (with payment) | Limited by your hardware |
You (Client) OpenAI (Server)
| |
| POST /v1/chat/completions |
| { |
| model: "gpt-4o", |
| messages: [...] |
| } |
|------------------------------->|
| |
| [Processing...] |
| [Model inference] |
| [Generate response] |
| |
| Response |
| { |
| choices: [{ |
| message: { |
| content: "..." |
| } |
| }] |
| } |
|<-------------------------------|
| |
Key point: Each request is independent. The API doesn't store conversation history.
Every message has a role that determines its purpose:
{ role: 'system', content: 'You are a helpful Python tutor.' }
Purpose: Define the AI's behavior, personality, and capabilities
Think of it as:
Examples:
// Specialist agent
"You are an expert SQL database administrator."
// Tone and style
"You are a friendly customer support agent. Be warm and empathetic."
// Output format control
"You are a JSON API. Always respond with valid JSON, never plain text."
// Behavioral constraints
"You are a code reviewer. Be constructive and focus on best practices."
Best practices:
{ role: 'user', content: 'How do I use async/await?' }
Purpose: Represent the human's input or questions
Think of it as:
{ role: 'assistant', content: 'Async/await is a way to handle promises...' }
Purpose: Represent the AI's previous responses
Think of it as:
[
{ role: 'system', content: 'You are a math tutor.' },
// First exchange
{ role: 'user', content: 'What is 15 * 24?' },
{ role: 'assistant', content: '15 * 24 = 360' },
// Follow-up (knows context)
{ role: 'user', content: 'What about dividing that by 3?' },
{ role: 'assistant', content: '360 ÷ 3 = 120' },
]
Why this matters: The role structure enables:
Most important principle: OpenAI's API is stateless.
Each API call is independent. The model doesn't remember previous requests.
Request 1: "My name is Alice"
Response 1: "Hello Alice!"
Request 2: "What's my name?"
Response 2: "I don't know your name." ← No memory!
You must send the full conversation history:
const messages = [];
// First turn
messages.push({ role: 'user', content: 'My name is Alice' });
const response1 = await client.chat.completions.create({
model: 'gpt-4o',
messages: messages // ["My name is Alice"]
});
messages.push(response1.choices[0].message);
// Second turn - include full history
messages.push({ role: 'user', content: "What's my name?" });
const response2 = await client.chat.completions.create({
model: 'gpt-4o',
messages: messages // Full conversation!
});
Benefits:
Challenges:
Real-world solutions:
// Trim old messages when too long
if (messages.length > 20) {
messages = [messages[0], ...messages.slice(-10)]; // Keep system + last 10
}
// Summarize old context
if (totalTokens > 10000) {
const summary = await summarizeConversation(messages);
messages = [systemMessage, summary, ...recentMessages];
}
Temperature controls how "creative" or "random" the model's output is.
When generating each token, the model assigns probabilities to possible next tokens:
Input: "The sky is"
Possible next tokens:
- "blue" → 70% probability
- "clear" → 15% probability
- "dark" → 10% probability
- "purple" → 5% probability
Temperature modifies these probabilities:
Temperature = 0.0 (Deterministic)
Always pick the highest probability token
"The sky is blue" ← Same output every time
Temperature = 0.7 (Balanced)
Sample probabilistically with slight randomness
"The sky is blue" or "The sky is clear"
Temperature = 1.5 (Creative)
Flatten probabilities, allow unlikely choices
"The sky is purple" or "The sky is dancing" ← More surprising!
Temperature 0.0 - 0.3: Focused Tasks
Example:
// Extract JSON from text - needs consistency
temperature: 0.1
Temperature 0.5 - 0.9: Balanced Tasks
Example:
// Friendly chatbot
temperature: 0.7
Temperature 1.0 - 2.0: Creative Tasks
Example:
// Generate 10 different marketing taglines
temperature: 1.3
User: "Tell me a story"
[Wait...]
[Wait...]
[Wait...]
Response: "Once upon a time, there was a..." (all at once)
Pros:
Cons:
User: "Tell me a story"
"Once"
"Once upon"
"Once upon a"
"Once upon a time"
"Once upon a time there"
...
Pros:
Cons:
Use Non-Streaming:
Use Streaming:
Tokens are the fundamental units that language models process. They're not exactly words, but pieces of text.
Tokenization examples:
"Hello world" → ["Hello", " world"] = 2 tokens
"coding" → ["coding"] = 1 token
"uncoded" → ["un", "coded"] = 2 tokens
1. Cost You pay per token (input + output):
Request: 100 tokens
Response: 150 tokens
Total billed: 250 tokens
2. Context Limits Each model has a maximum token limit:
gpt-4o: 128,000 tokens (≈96,000 words)
gpt-3.5-turbo: 16,384 tokens (≈12,000 words)
3. Performance More tokens = longer processing time and higher cost
Monitor usage:
console.log(response.usage.total_tokens);
// Track cumulative usage for budgeting
Limit response length:
max_tokens: 150 // Cap the response
Trim conversation history:
// Keep only recent messages
if (messages.length > 20) {
messages = messages.slice(-20);
}
Estimate before sending:
import { encode } from 'gpt-tokenizer';
const text = "Your message here";
const tokens = encode(text).length;
console.log(`Estimated tokens: ${tokens}`);
Best for:
Characteristics:
Example use cases:
Best for:
Characteristics:
Example use cases:
Best for:
Characteristics:
Example use cases:
Is task critical and complex?
├─ YES → GPT-4o
└─ NO
└─ Is speed important and task simple?
├─ YES → GPT-3.5-turbo
└─ NO → GPT-4o-mini
1. Authentication Errors (401)
// Invalid API key
Error: Incorrect API key provided
2. Rate Limiting (429)
// Too many requests
Error: Rate limit exceeded
3. Token Limits (400)
// Context too long
Error: This model's maximum context length is 16385 tokens
4. Service Errors (500)
// OpenAI service issue
Error: The server had an error processing your request
1. Always use try-catch:
try {
const response = await client.chat.completions.create({...});
} catch (error) {
if (error.status === 429) {
// Implement backoff and retry
} else if (error.status === 500) {
// Retry with exponential backoff
} else {
// Log and handle appropriately
}
}
2. Implement retry logic:
async function retryWithBackoff(fn, maxRetries = 3) {
for (let i = 0; i < maxRetries; i++) {
try {
return await fn();
} catch (error) {
if (i === maxRetries - 1) throw error;
await sleep(Math.pow(2, i) * 1000); // Exponential backoff
}
}
}
3. Monitor token usage:
let totalTokens = 0;
totalTokens += response.usage.total_tokens;
if (totalTokens > MONTHLY_BUDGET_TOKENS) {
throw new Error('Monthly token budget exceeded');
}
Use case: One-off queries, simple automation
const response = await client.chat.completions.create({
model: 'gpt-4o',
messages: [{ role: 'user', content: query }]
});
Pros: Simple, easy to understand Cons: No context, no memory
Use case: Chat applications, tutoring, customer support
class Conversation {
constructor() {
this.messages = [
{ role: 'system', content: 'Your behavior' }
];
}
async ask(userMessage) {
this.messages.push({ role: 'user', content: userMessage });
const response = await client.chat.completions.create({
model: 'gpt-4o',
messages: this.messages
});
this.messages.push(response.choices[0].message);
return response.choices[0].message.content;
}
}
Pros: Maintains context, natural conversation Cons: Token costs grow, needs management
Use case: Domain-specific applications
class PythonTutor {
async help(question) {
return await client.chat.completions.create({
model: 'gpt-4o',
messages: [
{
role: 'system',
content: 'You are an expert Python tutor. Explain concepts clearly with code examples.'
},
{ role: 'user', content: question }
],
temperature: 0.3 // Focused responses
});
}
}
Pros: Consistent behavior, optimized for domain Cons: Less flexible
In real-world projects, the best solution often isn't choosing between OpenAI and local LLMs - it's using both strategically.
Cost optimization: Use expensive models only when necessary Privacy compliance: Keep sensitive data local while leveraging cloud for general tasks Performance balance: Fast local models for simple tasks, powerful cloud models for complex ones Reliability: Fallback options when one service is down Flexibility: Match the right tool to each specific task
Simple tasks → Local LLM (fast, free, private)
↓ If complex
Complex tasks → OpenAI API (powerful, accurate)
Example workflow:
async function processQuery(query) {
const complexity = await assessComplexity(query);
if (complexity < 0.5) {
// Use local model for simple queries
return await localLLM.generate(query);
} else {
// Use OpenAI for complex reasoning
return await openai.chat.completions.create({
model: 'gpt-4o',
messages: [{ role: 'user', content: query }]
});
}
}
Use cases:
Public data → OpenAI (best quality)
Sensitive data → Local LLM (private, secure)
Example:
async function handleRequest(data, containsSensitiveInfo) {
if (containsSensitiveInfo) {
// Process locally - data never leaves your infrastructure
return await localLLM.generate(data, {
systemPrompt: "You are a HIPAA-compliant assistant"
});
} else {
// Use cloud for better quality
return await openai.chat.completions.create({
model: 'gpt-4o',
messages: [{ role: 'user', content: data }]
});
}
}
Use cases:
Agent 1 (Local): Fast classifier
↓ Routes to
Agent 2 (OpenAI): Deep analyzer
↓ Routes to
Agent 3 (Local): Action executor
Example:
class MultiModelAgent {
async process(input) {
// Step 1: Local model classifies intent (fast, cheap)
const intent = await localLLM.classify(input);
// Step 2: Route to appropriate handler
if (intent.requiresReasoning) {
// Complex reasoning with GPT-4
const analysis = await openai.chat.completions.create({
model: 'gpt-4o',
messages: [{ role: 'user', content: input }]
});
return analysis.choices[0].message.content;
} else {
// Simple response with local model
return await localLLM.generate(input);
}
}
}
Use cases:
Development → OpenAI (fast iteration, best results)
↓ Optimize
Production → Local LLM (cost-effective, private)
Workflow:
const MODEL_PROVIDER = process.env.NODE_ENV === 'production'
? 'local'
: 'openai';
async function generateResponse(prompt) {
if (MODEL_PROVIDER === 'local') {
return await localLLM.generate(prompt);
} else {
return await openai.chat.completions.create({
model: 'gpt-4o',
messages: [{ role: 'user', content: prompt }]
});
}
}
Strategy:
Query → [Local Model, OpenAI, Another API]
↓ ↓ ↓
Response Response Response
↓ ↓ ↓
Aggregator / Validator
↓
Best Response
Example:
async function ensembleGenerate(prompt) {
// Get responses from multiple sources
const [local, openai, backup] = await Promise.allSettled([
localLLM.generate(prompt),
openaiClient.chat.completions.create({
model: 'gpt-4o',
messages: [{ role: 'user', content: prompt }]
}),
backupAPI.generate(prompt)
]);
// Use validator to pick best or combine
return validator.selectBest([local, openai, backup]);
}
Use cases:
Option A: OpenAI Only
10,000 queries × 500 tokens avg = 5M tokens/day
Cost: ~$25-50/day = ~$750-1500/month
Pros: Highest quality, zero infrastructure
Cons: Expensive at scale, privacy concerns
Option B: Local LLM Only
Infrastructure: $100-500/month (server/GPU)
Cost: $100-500/month
Pros: Predictable costs, private, unlimited usage
Cons: Setup complexity, maintenance, lower quality
Option C: Hybrid (80% local, 20% OpenAI)
8,000 simple queries → Local LLM (free after setup)
2,000 complex queries → OpenAI (~$5-10/day)
Infrastructure: $100-500/month
API costs: $150-300/month
Total: $250-800/month
Pros: Cost-effective, high quality when needed, flexible
Cons: More complex architecture
Winner for most projects: Hybrid approach ✓
START: New query arrives
↓
Is data sensitive/regulated?
├─ YES → Use local model (privacy first)
└─ NO → Continue
↓
Is task simple/repetitive?
├─ YES → Use local model (cost-effective)
└─ NO → Continue
↓
Is high accuracy critical?
├─ YES → Use OpenAI (quality first)
└─ NO → Continue
↓
Is it high volume?
├─ YES → Use local model (cost at scale)
└─ NO → Use OpenAI (simplicity)
Advanced systems will automatically choose models based on real-time factors:
class IntelligentModelSelector {
async selectModel(query, context) {
const factors = {
complexity: await this.analyzeComplexity(query),
latency: context.userTolerance,
budget: context.remainingBudget,
accuracy: context.requiredConfidence,
privacy: context.dataClassification
};
// ML model predicts best provider
const selection = await this.mlSelector.predict(factors);
return {
provider: selection.provider, // 'local' | 'openai-mini' | 'openai-4'
confidence: selection.confidence,
reasoning: selection.reasoning
};
}
}
You don't have to choose. Modern AI applications benefit from using the right model for each task:
The best architecture leverages the strengths of each approach while mitigating their weaknesses.
The concepts covered here are foundational for building AI agents:
Bottom line: You can't build good agents without mastering these fundamentals. Every agent pattern builds on this foundation.