Model Providers

A model provider is the LLM configuration agents run on: the provider type, its base URL, the models it exposes (each with its own pricing, lifecycle and data-sensitivity allowlist), a default model, and a max-token cap. A provider record belongs to a tenant, but model resolution uses the caller's tenant's provider of a type when there is one and otherwise any active provider of that type, so the platform's providers serve every tenant that has none of its own.

Supported providers

Agentfleet reaches most providers through an OpenAI-compatible surface, so a new provider is a matter of configuration rather than code. Anthropic is reached through its native Messages API instead, so Claude models keep Anthropic's tool-use format, prompt caching and cache-token accounting. Both clients can stream replies; streaming is what lets a chat show its reply as it is written. The platform ships with these providers configured out of the box:

Provider	Default model	Notes
Groq	llama-3.3-70b-versatile	The original v1 provider.
OpenAI	gpt-4o-mini
Google (Gemini)	gemini-2.5-flash	Via Google's OpenAI-compatible endpoint.
Cerebras	gpt-oss-120b
DeepInfra	meta-llama/Llama-3.3-70B-Instruct-Turbo	Broad open-model catalogue with public pricing.
Anthropic (Claude)	claude-sonnet-4-6	Native Messages API with prompt caching.
Ollama	llama3.2	Local, no API key — for development.

The provider type enum also includes DeepSeek, which has no out-of-the-box configuration.

When an active provider is configured:

• Agent create/update validates the agent's model against the provider's model list, and refuses a sunset model. A sunset model is refused for step and workflow-step model overrides too.
• Runs use the provider's max-token cap (and any per-model override) for model calls.
• Per-provider rate limits (tokens- and requests-per-minute) pace outbound calls so a run stays within the provider account's quota.

When no provider row is configured, the platform falls back to its static defaults — useful for development, but configure a real provider for production tenants.

Configuration

A provider record is edited on Observe → Model providers (or POST/PUT /api/model-providers):

Field	Purpose
Name	Display name.
Provider	The provider type (Anthropic, Groq, OpenAI, Google, Cerebras, DeepInfra, Ollama, DeepSeek).
Base URL	The provider's API endpoint.
API key reference	Where the key lives in secure configuration — never the key itself.
Models / default model	The models agents may select, and the default. Each model has an identifier, a display name and optional pricing.
Max tokens	Output-token cap applied to model calls.
Pricing	Optional per-model input/output price-per-million-tokens and currency, used to estimate run cost, plus optional cache-read and cache-write prices for prompt caching.

Some settings come from the platform's provider registry (ModelProviders:Providers in the API's configuration) rather than the record, and are changed by an operator:

Registry setting	Purpose
Adapter	OpenAI (any OpenAI-compatible API) or Anthropic (the native Messages API).
TokensPerMinute, RequestsPerMinute	Rate limits, enforced by a client-side pacing handler.
Models.<id>.MaxTokens	A per-model max-token override.
Models.<id>.ReasoningEffort / ReasoningFormat	Reasoning settings for models that support them.

Each model also has a lifecycle and a sensitivity allowlist, set separately (below).

API keys

Each provider has its own API key, and that key is never stored in Agentfleet's database. The platform reads it from secure configuration (a key vault in deployed environments), and the provider record holds only a reference. Keys are separated per environment — staging and production use different keys. A provider that is active but whose key is missing fails its model calls at runtime, so configure the key before activating the provider.

Cost estimation

Every recorded model call captures a pricing snapshot: the input, output and cache prices per million tokens (and currency) in effect when the call was made — and computes an estimated cost from its token counts. Because the price is snapshotted per call, later price changes don't rewrite the cost of past runs.

Prompt caching

On Anthropic, every model call marks the tool definitions, the system prompt (the step's instructions and skills) and the newest message as cache breakpoints. Each round of a step's conversation then reads everything the previous round sent from Anthropic's prompt cache and writes only what it added. A call records its cache-read and cache-write tokens alongside its input tokens (they are part of the input total, so token budgets are unchanged). They are billed at the model's cache-read and cache-write prices, or at the input price when those are not set. For Claude, a cache read typically costs 10% of the input price and a cache write 125%.

Estimated cost rolls up onto the step and run summaries, so a run shows what it cost across all its model calls. Calls on a model with no configured pricing are unpriced: they contribute no cost and are counted separately, so a partially-priced run is never silently under-reported. The figure is an estimate from configured prices, not a provider invoice.

Embeddings and summarization

Two built-in tools can use a model of their own, configured in the API's Retrieval section:

• search_result ranks a large result's chunks with an embedding model (Retrieval:EmbeddingProvider and EmbeddingModel, for example OpenAI and text-embedding-3-small) through the provider's OpenAI-compatible embeddings endpoint. With none configured, it uses a free, offline lexical ranking.
• summarize_result uses a cheaper model (Retrieval:SummarizationProvider and SummarizationModel), or the step's own model when none is set.

These calls are recorded as model calls of kind Embedding or Summarization (alongside Chat), billed to the run and counted towards its usage limits. See the semantic retrieval runbook.

Model lifecycle

Each model of a provider has a lifecycle status (Lifecycle → Change on the provider's page, or PUT /api/model-providers/model-lifecycle):

Status	Effect
Active	Supported.
Deprecated	Still runs and can still be chosen, but is flagged "(deprecated)" in pickers and in the dependency inventory. With a sunset date, it becomes sunset automatically on that date.
Sunset	Cannot be chosen for an agent, step override or workflow-step override, and calls to it are refused — including from agents and published versions configured earlier.

A deprecated model can name a replacement model of the same provider, which becomes the default choice when you swap it out.

Model dependencies and bulk swaps

Observe → Dependencies lists every model in use with its lifecycle, and draws a dependency graph for a model, resource, tool or agent: the agents, step overrides, workflows (draft and published versions), schedules and triggers that depend on it, with each node's recent runs, failure rate and cost (permission view-dependencies).

A bulk model swap replaces one model with another across agents, step overrides and workflow-step overrides in one audited operation (permission swap-models, held by identity-admin). Plan it first to see what would change, and optionally measure the swap with an eval experiment before applying it. See the model dependencies runbook.

Sensitivity allowlists

Each model has an allowlist of the data-sensitivity labels it may receive (PUT /api/model-providers/model-sensitivity-allowlist). A model is offered a resource's tools only when its allowlist covers every label on that resource; the default empty allowlist admits only unlabelled data. See Usage & Data Governance.

Lifecycle

Providers are Active, Inactive, Disallowed, or Deleted. Only an active provider participates in model resolution and agent-model validation.

Related

• Agents — what runs on a provider's models.
• Evaluations — comparing models with experiments and model optimisation.
• Usage & Data Governance — budgets, provider back-off and sensitivity labels.
• Operator runbooks: Anthropic client, model dependencies.