The Problem: Inference Gets Hard at Scale If you’ve shipped an AI feature to production, you already know: the hard part isn’t making a model respond to a prompt. The hard part is making it respond more reliably, at scale, across multiple models, without burning through your budget. The moment real users show up, you’re dealing with GPU resource contention, traffic unpredictability (a single enterprise customer can 10x your request volume overnight), latency-cost tradeoffs that shift constantly, and multi-model orchestration across text, vision, image, video, and audio — each with different API contracts and failure characteristics. Most teams spend months just getting the infrastructure stable. We built DigitalOcean Serverless Inference so you don’t have to. What Serverless Inference Is DigitalOcean Serverless Inference is a fully managed, API-first inference platform — 30+ foundation models across text, code, vision, image generation, video generation, and speech, all through a single API key, a single base URL, and pay-per-token pricing with no minimum commitments. The core idea: Serverless Inference separates model consumption from infrastructure management. It automatically scales to handle incoming requests. Because it does not maintain sessions, each request must include the full context needed by the model. You interact with models through an API surface. We handle GPU allocation, scaling, and model lifecycle underneath. Single Endpoint, Every Mode None https://inference.do-ai.run Authenticate with a Model Access Key (recommended — scoped to specific models, VPC-restrictable) OpenAI and Anthropic Compatible The API is OpenAI-compatible. If you have existing code that calls OpenAI, switch to DigitalOcean by changing two lines — the base URL and the key: Python from openai import OpenAI import os
client = OpenAI(
base_url="https://inference.do-ai.run/v1/",
api_key=os.getenv("MODEL_ACCESS_KEY"),
)
response = client.chat.completions.create(
model="deepseek-v3.2",
messages=[{"role": "user", "content": "Explain the CAP theorem."}],
) We also support Anthropic-compatible patterns through the /v1/messages endpoint, so Claude Code and other agentic workflows work directly through DigitalOcean without vendor lock-in. Intelligent Routing, Built-in Tools, and More Beyond basic inference, the platform includes an Inference Router for automatic multi-model routing, built-in tools for knowledge retrieval, MCP, and web search, prompt caching for cost reduction on repeated contexts, and reasoning for step-by-step thinking traces. We’ll cover each of these in detail later in this post. Colocated with Your Cloud Unlike standalone inference providers, Serverless Inference is part of the DigitalOcean platform. Your inference workloads sit alongside databases, object storage, Kubernetes clusters, and VPCs — all under unified billing and access control. This is a structural advantage where DigitalOcean is unmatched. Architecture: How Requests Flow Here’s what happens under the hood when your application sends a request: None Client Request
→ Cloudflare (edge proxying, DDoS protection, TLS)
→ Load Balancer(auth, validation)
→ Traefik (ingress routing on DOKS)
→ Intelligent Inference API (routing, billing)
→ Model Executor Service (provider translation)
→ Model Backend (Ray + vLLM for open-source,
or provider API for OpenAI/Anthropic)
→ Streaming Response → Client
→ Kafka (billing events, telemetry) Load Balancer Distributes traffic across the inference cluster and serves as the policy enforcement point. Every request is validated against the Model Access Key or DigitalOcean personal access token, with the load balancer resolving tenant identity and confirming the caller is authorized to use the requested model; VPC-bound keys are enforced, rejecting requests originating outside the restricted network. Per-account and per-model rate limits are also enforced via a regional Redis cache to protect platform stability and help prevent single-tenant resource exhaustion. Finally, before reaching any backend, request validation occurs against the model’s contract from the Model Catalog—the centralized source of truth—designed to ensure that requests with unsupported parameters or incorrect endpoint shapes are rejected deterministically with a clear error, preventing the frustrating experience of cryptic provider errors deep in the stack. Intelligent Inference API The customer-facing entry point and policy enforcement layer. Every request passes through this service, where it handles authentication (validating Model Access Keys), request validation against the model’s contract, rate limiting (via Redis), billing metering (token counts dispatched to Kafka), and SSE streaming. It also orchestrates the Inference Router and built-in tool execution. Deployed as a stateless service on DigitalOcean Kubernetes Service (DOKS) with autoscaling. Model Executor Service — The Translation Layer This is one of the most important — and least visible — parts of the platform. Every model provider has quirks. Anthropic structures tool calls are different from OpenAI. Streaming event formats vary. Some providers support parameters that others silently ignore. Request schemas, error shapes, and response normalization all differ in subtle ways. If you’ve ever tried to build a multi-model application yourself, you’ve felt this pain — it’s a constant stream of provider-specific edge cases that break your code in production. The Model Executor Service helps solve this by sitting between the Intelligent Inference API and every model backend. It translates a standardized request envelope into whatever provider-native format the backend expects, executes the call, then normalizes the response back into a consistent shape. Provider-specific quirks — streaming differences, parameter mappings, error translations — are absorbed here to prevent leaks to your application. This is why every model on the platform works identically across the API endpoints (/chat/completions, /responses, and /messages (Anthropic Models Only)), regardless of who built the model or where it runs. You don’t need to know whether the model behind a request is hosted on our GPUs, served by OpenAI, or running on Anthropic’s infrastructure. The translation layer presents a single, consistent API contract for all of them. Model Runtime: Ray + vLLM For DigitalOcean-hosted open-source models, the runtime layer is built on Ray (orchestration and scheduling across NVIDIA H100 GPU nodes) and vLLM (KV cache management, continuous batching, token generation). Ray multiplexes multiple models across shared GPU pools, so you pay for tokens consumed rather than GPU-hours reserved. Commercial Model Routing For commercial models from OpenAI and Anthropic, the Model Executor Service handles the provider translation and forwards to the provider’s API. The response is normalized back through the same pipeline. This means you can call Claude Sonnet and DeepSeek V3.2 from the same application with the same key, the same endpoint, and get back identically structured responses. Billing Pipeline Every request generates a usage event (model ID, token counts, metadata) written to regional Kafka. The billing pipeline consumes these asynchronously — it never sits in the critical request path, so billing latency doesn’t affect inference latency. Getting Started: From Zero to Inference If you’ve used the OpenAI SDK before, you’ll be productive in minutes. Here’s a walkthrough from first API key to streaming responses. Step 1: Get Your Model Access Key In the DigitalOcean Control Panel , click INFERENCE → Manage → Model Access Keys . Create a key and export it: Shell export MODEL_ACCESS_KEY="your-model-access-key-here" Step 2: Chat Completions API The most common endpoint. Send a POST to /v1/chat/completions with a model ID, messages, temperature, and token limit: Shell curl -X POST https://inference.do-ai.run/v1/chat/completions
-H "Authorization: Bearer $MODEL_ACCESS_KEY"
-H "Content-Type: application/json"
-d '{
"model": "llama3.3-70b-instruct",
"messages": [{"role": "user", "content": "What is the capital of France?"}],
"temperature": 0.7,
"max_completion_tokens": 256
}' The same request using the Python OpenAI SDK: Python from openai import OpenAI
from dotenv import load_dotenv
import os
load_dotenv()
client = OpenAI( base_url="https://inference.do-ai.run/v1/", api_key=os.getenv("MODEL_ACCESS_KEY"), )
resp = client.chat.completions.create(
model="llama3.3-70b-instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Tell me a fun fact about octopuses."}
],
)
print(resp.choices[0].message.content) To switch models, change one parameter — “model”: “llama-4-maverick”. No SDK, endpoint, or auth changes needed. Step 3: Responses API For newer integrations and multi-step tool use, use the Responses API at /v1/responses. It takes a single input field instead of a messages array: Shell curl -sS -X POST https://inference.do-ai.run/v1/responses
-H "Authorization: Bearer MODEL_ACCESS_KEY"
-H "Content-Type: application/json"
-d '{
"model": "anthropic-claude-4.6-sonnet",
"messages": [{
"role": "developer",
"content": [{
"type": "text",
"text": "You are a helpful coding assistant with extensive knowledge of Python and cloud infrastructure.",
"cache_control": {"type": "ephemeral", "ttl": "1h"}
}]
},
{"role": "user", "content": "Write a Python function to validate email addresses."}],
"max_completion_tokens": 1024
}' The response shows cache_created_input_tokens on the first call; subsequent calls show cache_read_input_tokens at reduced cost. OpenAI Models For prompts with 1,024+ tokens, use prompt_cache_retention set to in_memory or 24h: JSON {
"model": "gpt-4o-mini",
"prompt_cache_retention": "24h",
"messages": [...],
"temperature": 0.2
} Open-Source Models Prompt caching for DigitalOcean-hosted open-source models is not currently supported . It is available only for Anthropic and OpenAI models at this time. Open-source model caching is on the roadmap as a high-priority investment. Reasoning For models that support it, you can enable step-by-step thinking traces — useful for math, logic, coding, and complex analytical tasks. Anthropic format — use a reasoning object with effort and optional max_tokens: Shell curl -X POST https://inference.do-ai.run/v1/chat/completions
-H "Authorization: Bearer MODEL_ACCESS_KEY"
-H "Content-Type: application/json"
-d '{
"model": "anthropic-claude-4.6-sonnet",
"messages": [
{
"role": "user",
"content": "What is 27 * 453? Think step by step."
}
],
"max_completion_tokens": 8192,
"reasoning_effort": "high"
}' Multimodal Inference Serverless Inference isn’t text-only. We support vision-language models, image generation, video generation, text-to-speech, and vector embeddings — all through the same API key and base URL. Vision-Language Models VLMs accept text + image inputs (PNG, JPG, JPEG, WEBP as base64 or HTTPS URLs) and return text: Shell curl https://inference.do-ai.run/v1/chat/completions
-H "Authorization: Bearer MODEL_ACCESS_KEY"
-H "Content-Type: application/json"
-d '{
"model": "stable-diffusion-3.5-large",
"prompt": "A sunset over mountains",
"n": 1,
"size": "1024x1024",
"quality": "auto",
"response_format": "b64_json",
"background": "auto",
"output_format": "png"
}' Text-to-Video (Asynchronous) Submit a job, poll for status, download MP4 when complete. Output is 480p (9 seconds) or 720p (5 seconds). Videos expire 2 hours after completion: Shell curl -X POST https://inference.do-ai.run/v1/video/generations
-H "Authorization: Bearer $MODEL_ACCESS_KEY"
-H "Content-Type: application/json"
-d '{
"model": "wan2.2-t2v-a14b",
"prompt": "A drone shot flying over a lush green valley at golden hour",
"size": "1280x720",
"fps": 16
}'
The request returns a job ID and job status:
{ "id": "job_abc123", "status": "processing" }
Next, poll the result using the job ID:
curl https://inference.do-ai.run/v1/video/generations/job_abc123
-H "Authorization: Bearer $MODEL_ACCESS_KEY"
You can see the following when the job completes
{
"created_at": 1777003604,
"error": null,
"id": "video_abc",
"model": "wan2.2-t2v-a14b",
"object": "video",
"output": null,
"status": "completed",
"x_request_id": null
} Text-to-Speech Shell curl -sS https://inference.do-ai.run/v1/audio/speech
-H "Authorization: Bearer MODEL_ACCESS_KEY"
-H "Content-Type: application/json"
-d '{
"model": "openai-gpt-4o",
"messages": [{"role": "user", "content": "What features does DigitalOcean Inference offer?"}],
"tools": [{"type": "knowledge_base_retrieval", "knowledge_base_id": ""}],
"max_tokens": 1024
}' Model Context Protocol (MCP) Connect to remote MCP servers — authenticated or unauthenticated — for live data access: Shell curl -X POST https://inference.do-ai.run/v1/chat/completions
-H "Authorization: Bearer MODEL_ACCESS_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openai-gpt-4o",
"messages": [{"role": "user", "content": "Fetch my DigitalOcean account info."}],
"tools": [{
"type": "mcp",
"server_label": "digitalocean",
"server_url": "https://accounts.mcp.digitalocean.com/mcp",
"authorization": "Bearer DIGITALOCEAN_API_TOKEN",
"allowed_tools": ["account-get-information"]
}],
"tool_choice": "required",
"max_tokens": 512
}' Web Search Give models access to real-time web content: Shell curl -X POST https://inference.do-ai.run/v1/responses
-H "Authorization: Bearer MODEL_ACCESS_KEY"
-H "anthropic-version: 2023-06-01"
-H "content-type: application/json"
-d '{
"model": "anthropic-claude-4.6-sonnet",
"max_tokens": 4096,
"tools": [{
"name": "read_file",
"description": "Read a file from the local filesystem.",
"input_schema": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}
}],
"messages": [{"role": "user", "content": "Refactor the authentication logic in src/auth.ts."}]
}' Pricing (current as of May 2026) Knowledge base retrieval and MCP incur no additional charges beyond standard per-token inference costs. Web search is $10 per 1,000 requests. Inference Router We mentioned the Inference Router earlier as a key differentiator. Here’s how it works in practice. The Inference Router classifies each incoming request against your configured tasks, then selects the best model from a pool. Each task has up to 3 models and a selection policy: Cost Efficiency (cheapest by token cost), Speed Optimization (fastest by TTFT), Manual Ranking (your specified order), or Optimal (DigitalOce