Choosing Inference Placement

Muna’s signature feature is allowing developers to choose where inference runs, per-request.

Running with Adaptive Placement

Muna can adaptively search for the best hardware to run models, depending on your cost, latency, and throughput requirements. Use the muna.predictions.create method, and specify your constraints in natural language:

// 🔥 Run inference with the lowest latency
const prediction = await muna.predictions.create({
  tag: "@openai/gpt-oss-120b",
  inputs: { messages },
  acceleration: "lowest latency"
});

# 🔥 Run inference with the lowest latency
prediction = muna.predictions.create(
  tag="@openai/gpt-oss-120b",
  inputs={ "messages": messages },
  acceleration="lowest latency"
)

// 🔥 Run inference with the lowest latency
const prediction = await muna.predictions.create({
  tag: "@openai/gpt-oss-120b",
  inputs: { messages },
  acceleration: "lowest latency"
});

// 🔥 Run inference with the lowest latency
let prediction = try await muna.predictions.create(
  tag: "@openai/gpt-oss-120b",
  inputs: ["messages": messages],
  acceleration: "lowest latency"
)

// 🔥 Run inference with the lowest latency
val prediction = muna.predictions.create(
  "@openai/gpt-oss-120b",
  mapOf("messages" to messages),
  "lowest latency"
)

// 🔥 Run inference with the lowest latency
var prediction = await muna.Beta.Predictions.Remote.Create(
  tag: "@openai/gpt-oss-120b",
  inputs: new() { ["messages"] = messages },
  acceleration: "lowest latency"
);

This feature is in early alpha, and is only offered to specific teams. Request access on our Slack.

Specifying Placement Constraints

We strongly recommend anchoring your placement constraints around these three canonical intents:

Intent	Examples
Cost	`cheapest`, `lowest cost in the cloud`, `under $0.02`
Latency	`fastest`, `minimize latency`, `lowest latency that runs locally`
Throughput	`highest throughput`, `at least 100 requests per second`

You can also specify constraints with combinations of cost, latency, and throughput intents e.g. lowest cost under 200ms.

Running on Datacenter GPUs

Use the muna.predictions.create method, and specify a remote_* acceleration to run inference on a datacenter GPU:

// 🔥 Run inference with an Nvidia B200 GPU
const prediction = await muna.predictions.create({
  tag: "@bytedance/depth-anything-3",
  inputs: { image },
  acceleration: "remote_b200"
});

# 🔥 Run inference with an Nvidia B200 GPU
prediction = muna.predictions.create(
  tag="@bytedance/depth-anything-3",
  inputs={ "image": image },
  acceleration="remote_b200"
)

// 🔥 Run inference with an Nvidia B200 GPU
const prediction = await muna.beta.predictions.remote.create({
  tag: "@bytedance/depth-anything-3",
  inputs: { image },
  acceleration: "remote_b200"
});

// 🔥 Run inference with an Nvidia B200 GPU
let prediction = try await muna.beta.predictions.remote.create(
  tag: "@fxn/greeting",
  inputs: ["image": image],
  acceleration: .remote_b200
)

// 🔥 Run inference with an Nvidia B200 GPU
val prediction = muna.beta.predictions.remote.create(
  "@bytedance/depth-anything-3",
  mapOf("image" to image),
  "remote_b200"
)

// 🔥 Run inference with an Nvidia B200 GPU
var prediction = await muna.Predictions.Create(
  tag: "@fxn/greeting",
  inputs: new() { ["name"] = "Sosa" },
  acceleration: "remote_b200"
);

Supported Datacenter GPUs

Below are the currently supported cloud GPUs:

Acceleration	Notes
`remote_auto`	Run inference on the ideal datacenter hardware.
`remote_cpu`	Run inference on AMD CPU servers.
`remote_a10`	Run inference on an Nvidia A10 GPU.
`remote_a100`	Run inference on an Nvidia A100 GPU.
`remote_h100`	Run inference on an Nvidia H100 GPU.
`remote_b200`	Run inference on an Nvidia B200 GPU.
`remote_mi350x`	Run inference on an AMD MI350X GPU. Coming soon.
`remote_mi355x`	Run inference on an AMD MI355X GPU. Coming soon.
`remote_qaic100`	Run inference on a Qualcomm Cloud AI 100 accelerator. Coming soon.

If you want to self-host the GPU servers in your VPC or on-prem, reach out to us.

Running Locally

Use the muna.predictions.create method to run inference locally:

// 🔥 Run inference with the local NPU
const prediction = await muna.predictions.create({
  tag: "@bytedance/depth-anything-3",
  inputs: { image },
  acceleration: "local_npu"
});

# 🔥 Run inference with the local NPU
prediction = muna.predictions.create(
  tag="@bytedance/depth-anything-3",
  inputs={ "image": image },
  acceleration="local_npu"
)

// 🔥 Run inference with the local NPU
const prediction = await muna.predictions.create({
  tag: "@bytedance/depth-anything-3",
  inputs: { image },
  acceleration: "local_npu"
});

// 🔥 Run inference with the Apple Neural Engine
let prediction = try await muna.predictions.create(
  tag: "@bytedance/depth-anything-3",
  inputs: ["image": image],
  acceleration: .npu
)

// 🔥 Run inference with the local NPU
val prediction = muna.predictions.create(
  "@bytedance/depth-anything-3",
  mapOf("image" to image),
  Acceleration.NPU
)

// 🔥 Run inference with the local NPU
var prediction = await muna.Predictions.Create(
  tag: "@bytedance/depth-anything-3",
  inputs: new() { ["image"] = image },
  acceleration: @"local_npu"
);

Supported Local Processors

Below are the currently supported local processors:

Acceleration	Notes
`local_cpu`	Use the CPU to accelerate predictions. This is always enabled.
`local_gpu`	Use the GPU to accelerate predictions.
`local_npu`	Use the neural processor to accelerate predictions.

Muna currently does not support multi-GPU local acceleration. This is planned for the future.

Specifying the Local GPU

Some Muna clients allow you to specify the acceleration device used to make predictions. Our clients expose this field as an untyped integer or pointer. The underlying type depends on the current operating system:

OS	Device type	Notes
Android	-	Currently unsupported.
iOS	`id<MTLDevice>`	Metal device.
Linux	`int*`	Pointer to CUDA device ID.
macOS	`id<MTLDevice>`	Metal device.
visionOS	`id<MTLDevice>`	Metal device.
Web	`GPUDevice`	WebGPU device.
Windows	`ID3D12Device*`	DirectX 12 device.

The prediction device is merely a hint. Setting a device does not guarantee that all or any operation in the prediction function will actually use that acceleration device.

You should absolutely (absolutely) never ever do this unless you know what the hell you’re doing.

Get Started

Running Models

Deploying Models

Insiders

Choosing Inference Placement

Running with Adaptive Placement

Specifying Placement Constraints

Running on Datacenter GPUs

Supported Datacenter GPUs

Running Locally

Supported Local Processors

Specifying the Local GPU

​Running with Adaptive Placement

​Specifying Placement Constraints

​Running on Datacenter GPUs

​Supported Datacenter GPUs

​Running Locally

​Supported Local Processors

​Specifying the Local GPU

Running with Adaptive Placement

Specifying Placement Constraints

Running on Datacenter GPUs

Supported Datacenter GPUs

Running Locally

Supported Local Processors

Specifying the Local GPU