Inference
Grant a sandbox access to model providers through provider profiles, attachments, and native endpoints.
OpenShell grants model-provider access through provider profiles and sandbox attachments. The workload calls the provider’s native API. OpenShell evaluates the request against the profile-derived network policy and substitutes the real credential only at an endpoint authorized by that profile.
This keeps the complete access contract in one place:
| Concern | Owner |
|---|---|
| Credential and refresh lifecycle | Provider instance |
| Authorized hosts, ports, paths, and binaries | Provider profile |
| Which workload receives access | Sandbox provider attachment |
| Base URL, model, request shape, and timeout | Native client or workload configuration |
Define and Attach a Hosted Provider
Section titled “Define and Attach a Hosted Provider”Start from the NVIDIA example profile, review its access, and import an edited copy under a new ID. A new gateway has no profiles to export. Download the example so you can add the Python paths used by this workload:
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/providers/nvidia.yaml -o nvidia-native.yamlIn nvidia-native.yaml, change id to nvidia-native, give the profile a
distinct display_name, and set binaries to the executables that may call the
API, as described in Binary
Matching. Keep the
credential and endpoint definitions you intend to grant. For a Python workload,
the edited fields can look like:
id: nvidia-nativedisplay_name: NVIDIA Native APIbinaries: - /usr/bin/python3.* - /usr/local/bin/python3.* - /sandbox/.venv/**Lint and import the complete edited profile, then create a provider from its new ID:
openshell profile lint -f nvidia-native.yamlopenshell profile import -f nvidia-native.yaml
openshell provider create \ --name nvidia-prod \ --type nvidia-native \ --from-existing
openshell sandbox create \ --name inference-demo \ --from registry.example.com/team/python-agent:1.0 \ --provider nvidia-prod \ -- python app.pyThe imported provider profile supplies NVIDIA_API_KEY as an opaque placeholder and
allows the profile’s native endpoint. Configure the client with the real model
identifier:
import osfrom openai import OpenAI
client = OpenAI( base_url="https://integrate.api.nvidia.com/v1", api_key=os.environ["NVIDIA_API_KEY"], timeout=300,)
response = client.responses.create( model="nvidia/nemotron-3-super-120b-a12b", input="Hello!",)print(response.output_text)Use the same export, edit, and import flow for OpenAI, Anthropic, or another hosted service. Set a new profile ID, retain only the endpoints and credentials the workload needs, and name the actual client binaries. Provider attachment does not select or rewrite a model.
The gateway resolves a provider against the profiles you imported, so a provider
whose profile is missing fails at sandbox creation with a message naming the
profile and the import command. Treat the example profiles in
providers/ as
starting templates you copy and adapt, not as defaults.
Attach a Provider to a Running Sandbox
Section titled “Attach a Provider to a Running Sandbox”openshell sandbox provider attach inference-demo nvidia-prod --waitopenshell sandbox provider list inference-demoRunning sandboxes poll for provider and effective-policy changes. --wait
confirms the sandbox has applied the attachment. Launch a new process so it
receives the new credential placeholder:
openshell sandbox exec --name inference-demo -- python app.pyAn already-running process does not gain new environment variables. Credential
rotation and detach take effect once the sandbox applies the provider change.
Use --wait to confirm that change before relying on it.
Detach the provider to revoke its policy and credential access:
openshell sandbox provider detach inference-demo nvidia-prod --waitUse a Custom or Self-hosted Endpoint
Section titled “Use a Custom or Self-hosted Endpoint”Do not reuse the openai or anthropic example profile for an alternate host.
Those profiles bind credentials to the public vendor endpoints, and a provider
that points OPENAI_BASE_URL elsewhere is treated as endpointless rather than
silently carrying the credential to the new host. Import a profile that names the
intended endpoint and the binaries that may call it.
For a credentialless Ollama server on the gateway host, save this profile as
ollama-openai.yaml:
id: ollama-openaidisplay_name: Ollamadescription: Host-local Ollama OpenAI-compatible APIcategory: inferenceinference_capable: truecredentials: []endpoints: - host: host.openshell.internal port: 11434 protocol: rest access: read-write enforcement: enforcebinaries: - /usr/bin/curl - /usr/local/bin/curl - /usr/bin/python3 - /usr/local/bin/python - /sandbox/.uv/python/** - /sandbox/.venv/**Import the profile, create an instance, and attach it:
openshell profile lint -f ollama-openai.yamlopenshell profile import -f ollama-openai.yamlopenshell provider create --name ollama --type ollama-openai
openshell sandbox create \ --name ollama-client \ --from registry.example.com/team/python-agent:1.0 \ --provider ollama \ --env OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 \ -- python app.pyUse any non-empty placeholder value if the client library requires an API key for a server that does not authenticate requests:
import osfrom openai import OpenAI
client = OpenAI( base_url=os.environ["OPENAI_BASE_URL"], api_key="unused",)response = client.chat.completions.create( model="qwen3.5:0.8b", messages=[{"role": "user", "content": "Hello"}],)For an authenticated alternate endpoint, declare a credential in the custom profile, bind it to that endpoint, and create the provider from the original credential source. OpenShell never exports stored credential values.
Verify Access
Section titled “Verify Access”Inspect the effective policy, including provider-derived entries:
openshell policy get inference-demo --fullThen run a native request from a new sandbox process. A successful request confirms endpoint policy, binary attribution, credential substitution, DNS, and upstream service behavior.
If the request is denied:
- Confirm the provider is attached with
openshell sandbox provider list. - Confirm the caller binary and native endpoint appear in
openshell policy get <sandbox> --full. - If the provider was attached after the process started, launch a new process.
- If the sandbox uses a gateway global policy override, add the native endpoint there because a global override suppresses provider-derived policy layers.
- Inspect logs for
credential_endpoint_mismatch. That error means policy admitted the request but the provider profile did not authorize its credential at the requested endpoint.
Migrate from Managed Inference Routes
Section titled “Migrate from Managed Inference Routes”OpenShell removed the workspace-global managed inference route and the
openshell inference commands. Upgrades remove stored route records. Provider
records, provider refresh configuration, and existing sandbox attachments are
preserved. Import the profile each preserved provider refers to before the first
sandbox creation after the upgrade, and give new setups an explicit profile under
their own ID.
The old route cannot be converted automatically. It applied one provider and model to every sandbox in a workspace, while provider attachments intentionally grant access to selected sandboxes. OpenShell cannot infer which sandboxes should receive that authority.
Earlier releases configured one shared route:
openshell provider create --name nvidia-prod --type nvidia --from-existingopenshell inference set \ --provider nvidia-prod \ --model nvidia/nemotron-3-super-120b-a12b \ --timeout 300Clients called a managed virtual endpoint, supplied a placeholder key and model, and relied on OpenShell to rewrite the request.
Export the old provider type’s profile, edit its ID and access contract, import it, and create a replacement provider from the original credential source:
openshell profile export nvidia -o yaml > nvidia-native.yaml# Edit id, display_name, endpoints, and binaries in nvidia-native.yaml.openshell profile lint -f nvidia-native.yamlopenshell profile import -f nvidia-native.yamlopenshell provider create \ --name nvidia-native-prod \ --type nvidia-native \ --from-existingopenshell sandbox provider attach inference-demo nvidia-native-prod --waitOpenShell does not export stored credential values or change a provider’s profile type in place. If the original credential source is unavailable, import the original profile at its own ID so the preserved provider keeps resolving while you arrange credential rotation into the replacement provider.
Update each workload to:
- Call the provider’s native endpoint.
- Read the credential variable declared by its profile.
- Send the real provider model identifier.
- Configure request timeouts in the client.
- Use the provider’s native request format.
Launch a new process after attachment and verify a native request before upgrading production workloads. Code that still calls the retired managed virtual endpoint fails DNS resolution because OpenShell no longer resolves or trusts that host.
Migration Checklist
Section titled “Migration Checklist”Before upgrading, record the old provider, model, and timeout with the previous
release’s openshell inference get. Identify actual consumers, export and edit
the source profile, import it under a new ID, and create the replacement
provider from the original credential source. Attach it only to those
sandboxes, migrate their clients, and test the native path while the old route
is still available. Delete the old route on the previous release to expose
missed consumers. After upgrading the gateway, delete and recreate every
pre-upgrade sandbox so no old supervisor, DNS entry, trust material, or cached
route survives. Reapply only the provider attachments each replacement
sandbox needs.
Special cases require additional work:
- A provider with an alternate
OPENAI_BASE_URLorANTHROPIC_BASE_URLneeds an endpoint-bearing custom profile for that host. - Host-local services need
host.openshell.internalor a reachable LAN/service hostname, not127.0.0.1orlocalhost. - Google Vertex AI clients must use the native Vertex endpoint and authentication behavior. See Google.
- A bridge-fronted AWS Bedrock deployment needs a custom profile that declares the bridge endpoint and allowed client binaries.
Security Differences
Section titled “Security Differences”Provider attachment preserves credential non-disclosure: workloads receive opaque placeholders, and the proxy substitutes a real credential only after network policy and endpoint binding both pass. Native requests are no longer filtered or rewritten by a model-specific router. The provider profile’s L7 rules therefore define the allowed API surface, and the workload controls headers, model selection, request shape, streaming, and timeout behavior.
Next Steps
Section titled “Next Steps”We are a Cloud Native Computing Foundation sandbox project.
Copyright OpenShell a Series of LF Projects, LLC
For website terms of use, trademark policy and other project policies please see lfprojects.org/policies/.