# CLI Reference
Source: https://docs.muna.ai/cli/introduction
Compiling and managing predictors from the command line.
The Muna CLI is the primary tool for compiling Python functions and managing your predictors.
## Installation
Install the Muna CLI using pip:
```bash icon="terminal" theme={null}
# Run this in Terminal
$ pip install --upgrade muna
```
The CLI is bundled with the Muna Python SDK, so if you already have the SDK installed, you're ready to go.
## Commands
Transpile Python to C++ source code.
Compile a Python function for deployment.
Invoke a compiled Python function.
# muna transpile
Source: https://docs.muna.ai/cli/transpile
Transpile a Python function to C++ source code.
The `transpile` command converts a Python function into C++ source code. You can then
compile the generated source code into a library or executable:
## Usage
```bash icon="terminal" theme={null}
# Transpile a Python function
$ muna transpile [OPTIONS] PATH
```
The provided `path` can either be a path to a Python module; or a URL pointing to a Python module. There must be a function decorated with [`@compile`](/predictors/create).
### Specifying the Output Directory
Use the `--output` flag to write the generated sources to a given directory:
```bash icon="terminal" theme={null}
# Output to a custom directory
$ muna transpile --output ./generated greeting.py
```
The provided directory must not already exist on the file system.
### Transpiling from GitHub
You can transpile Python files directly from GitHub:
```bash icon="terminal" theme={null}
# Transpile from a GitHub URL
$ muna transpile --trust-remote-code https://github.com/muna-ai/muna-predictors/blob/main/python-coverage/fstring.py
```
When using `--trust-remote-code`, the CLI will download and execute code from the remote URL. Only use this option with code sources you trust.
## Using the Generated Code
The generated code defines a header which can be used in C++ libraries and applications;
along with an example command-line application that runs the compiled function.
The `muna transpile` command will write a self-contained header file (`*.hpp`) along
with a `CMakeLists.txt` file to the output directory.
### Running the Example Code
Build the example application using `cmake`:
```sh icon="terminal" theme={null}
# 🔥 Build the example application
$ cmake -B build && cmake --build build
```
Building the example application requires `cmake` to be installed, along with a
compiler toolchain for the current system (e.g. Visual Studio, Xcode, etc).
Once compiled, you can then run the example app in the command line:
```sh icon="terminal" theme={null}
# 🚀 Run the example app
$ ./example --help
```
### Using the Library
You can also use the transpiled function as a library in a C++ target:
```cmake CMakeLists.txt theme={null}
# Include the transpiled function
include(generated)
# Link against the transpiled function
target_link_libraries(
my_app PRIVATE
greeting::greeting
)
```
For most use cases, you should use [`muna compile`](/cli/compile) instead, which handles
transpilation, compilation, and deployment in a single step.
# Changelog
Source: https://docs.muna.ai/insiders/changelog
Tracking changes.
Muna for Python and the Muna CLI.
Muna for JavaScript, across browser and Node.js.
Muna for iOS and visionOS.
Muna for Android.
Muna for React Native (iOS and Android).
Muna for Unity Engine.
# How it Works
Source: https://docs.muna.ai/insiders/compiler
No magic tricks, no mojo 😇
**Imagine if you could run Python code everywhere**. Python is an incredibly simple and mature language. But
because it requires an interpreter, it either cannot run natively cross-platform; or it incurs a significant
performance cost compared to languages that are "closer to hardware".
We are on a mission to build a world where nobody has to learn Rust to get the performance benefits thereof.
Muna works by lowering your Python code to native code. The benefit is that developers
can think and write code in a high-level language, but still get the raw performance of a low-level language.
## Symbolic Tracing
Our compiler begins by building an **intermediate representation** (IR) of your Python function using a combination
of static analysis and symbolic tracing. We use [PEP 523](https://peps.python.org/pep-0523/) to hook into a sandboxed Python
interpreter before executing your function. We can then build a **trace** of every operation that happened within your
function.
[PEP 523](https://peps.python.org/pep-0523/) also forms the foundation of `torch.compile` in PyTorch 2.0.
In fact, this is what spurred initial development of Muna. [Read the paper](https://pytorch.org/assets/pytorch2-2.pdf).
For example, consider the following function which classifies an image:
```py classifier.py [expandable] icon="python" theme={null}
from muna import compile, Sandbox
from PIL import Image
from torchvision.models import mobilenet_v2, MobileNet_V2_Weights
from typing import Tuple
# Create MobileNet model
weights = MobileNet_V2_Weights.DEFAULT
model = mobilenet_v2(weights=weights).eval()
preprocess = weights.transforms()
labels = weights.meta["categories"]
# Define predictor
@compile(
tag="@vision-co/image-classifier",
description="Classify an image using the MobileNet v2 model.",
sandbox=Sandbox().pip_install("torch", "torchvision")
)
def predict(image: Image.Image) -> Tuple[str, float]:
batch = preprocess(image)[None]
logits = model(batch).squeeze(0).softmax(0)
class_id = logits.argmax().item()
score = logits[class_id].item()
label = labels[class_id]
return label, score
```
The symbolic tracer generates a graph that looks like the following:
```py [expandable] theme={null}
type name target args
------------- ------------------- --------------------------------------------- -------------------------------------------------------------------
input image image ()
call_function resize (image, [232])
call_function center_crop (resize, [224])
call_function pil_to_tensor (center_crop,)
call_function convert_image_dtype (pil_to_tensor, torch.float32)
call_function normalize (convert_image_dtype,)
call_function getitem (normalize, None)
call_module model model (getitem,)
call_method squeeze squeeze (model, 0)
call_method softmax softmax (squeeze, 0)
call_method argmax argmax (softmax,)
call_method item item (argmax,)
call_function getitem_1 (softmax, item)
call_method item_1 item (getitem_1,)
call_function list_index (['tench', 'goldfish', 'plow', ..., 'toilet tissue'], item) {}
output output output ((list_index, item_1),)
```
We inject more metadata with static analysis to build a full IR for your Python code. After this, we then lower
the IR to native code.
## Lowering to Native Code
We lower an IR graph to several different platform-specific implementations in native code. We do so by walking through the
graph's nodes and finding one or more operators that implement the node. Take the `resize` node in the
graph above:
| Type | Name | Target | Args |
| :-------------- | :------- | :--------------------------------- | :--------------- |
| `call_function` | `resize` | `` | `(image, [232])` |
We search through our library of native operators that perform a resize operation on an image.
Here's an example targeting Apple Silicon with [Accelerate.framework](https://developer.apple.com/documentation/accelerate?language=objc):
```cpp resize.mm [expandable] icon="c" theme={null}
@import Accelerate;
muna::image ResizeImageAppleSilicon(
muna::image image,
NSArray size
) {
...
vImageScale_ARGB8888(...);
...
}
```
This approach gives us two main benefits:
Our compiler infrastructure is hardware-aware and allows us to lower Python operations to code as low-level as Assembly and PTX.
This also allows us to work with hardware vendors to hyper-optimize individual operations for their hardware.
Because each IR node can map to several different low-level operations, we simply generate all possible
implementations of a prediction function, ship them out to different users, and gather telemetry data to
discover the implementation with the best performance for each unique device.
As a result of this, we are able to run orders of magnitude more performance experiments than what is possible
with manual performance tuning.
If you are a hardware vendor interested in enabling developers to leverage your custom accelerators, [reach out to us](mailto:hi@muna.ai)!
You'll bring the hardware; we'll bring the software.
## Compiling the Native Binaries
Finally, we compile the lowered native code for each of [our supported targets](/concepts#minumum-requirements).
When an application uses the `muna.predictions.create` method to create a prediction, our client SDK will download
a compiled binary, load it into the process, and invoke it.
You can inspect the native source code generated by Muna using the [Muna CLI](https://github.com/muna-ai/muna-py).
Take an example Python function:
```py area.py icon="python" theme={null}
from muna import compile
@compile(
tag="@yusuf/area",
description="Compute the area of a circle given its radius."
)
def area(radius: float) -> float:
return pi * radius ** 2
```
We can compile this function using the Muna CLI:
```bash icon="terminal" theme={null}
# Compile the function
$ muna compile --overwrite area.py
```
## Inspecting the Generated Code
Once compiled, use the Muna CLI to inspect the generated native source code:
```bash icon="terminal" theme={null}
# Get the generated native code for the current device
$ muna source --predictor @yusuf/area
```
Muna can generate hundreds of implementations for a given compiled function. As such, prefer
the `--prediction ` option instead of `--predictor `.
The result is a reference source file including the relevant native methods:
```cpp icon="c" theme={null}
float area__area(float radius) {
int32_t const_2 = 2;
float temp1 = _operator_pow(radius, const_2);
float temp2 = _operator_mul(area__pi, temp1);
return temp2;
}
...
```
Muna currently generates native code in C++, supported by Rust.
If you would like to compile Python functons to a custom platform, [reach out to us](mailto:hi@muna.ai).
# Introduction
Source: https://docs.muna.ai/introduction
Automated infrastructure for AI inference.
![Migrating from OpenAI to Muna.]()
Run inference with an OpenAI-compatible client, and optimize for cost, latency,
or throughput with one line of code.
## Installing Muna
We provide SDKs for common development frameworks:
```bash JavaScript icon="js" theme={null}
# Run this in Terminal
$ npm install muna
```
```bash Python icon="python" theme={null}
# Run this in Terminal
$ pip install --upgrade muna
```
```bash React Native icon="react" theme={null}
# Run this in Terminal
$ npm install @muna/expo
```
```yaml Flutter icon="flutter" theme={null}
# Add this to your `pubspec.yml`
dependencies:
muna: 0.0.2
```
```kt Android icon="android" theme={null}
// Add this to your `build.gradle` or `build.gradle.kts` file
dependencies {
implementation("ai.muna:muna:0.0.10")
}
```
```swift iOS focus={4-5,8-9} icon="swift" theme={null}
let package = Package(
name: "MySwiftPackage",
dependencies: [
// Add the package to your dependencies
.package(url: "https://github.com/muna-ai/muna-swift.git", from: "0.0.2"),
],
targets: [
// Then add `Muna` as a dependency to your target
.target(name: "MySwitPackage", dependencies: ["Muna"]),
]
)
```
```json Unity focus={3-9,11} icon="unity" theme={null}
// Add this to your `Packages/manifest.json` file
{
"scopedRegistries": [
{
"name": "Muna",
"url": "https://registry.npmjs.com",
"scopes": ["ai.muna"]
}
],
"dependencies": {
"ai.muna.muna": "0.0.48"
}
}
```
Most of our client SDKs are open-source. Star them [on GitHub](https://github.com/muna-ai)!
## Run your First Inference
[Generate an access key](https://muna.ai/settings/developer), then create a chat completion locally with [`@openai/gpt-oss-20b`](https://muna.ai/@openai/gpt-oss-20b).
```ts JavaScript icon="js" theme={null}
import { Muna } from "muna"
// 💥 Create an OpenAI client
const openai = new Muna({ accessKey: "..." }).beta.openai;
// 🔥 Create a chat completion
const completion = await openai.chat.completions.create({
model: "@openai/gpt-oss-20b",
messages: [{ role: "user", content: "What is the capital of France?" }]
});
// 🚀 Print the result
console.log(completion.choices[0]);
```
```py Python icon="python" theme={null}
from muna import Muna
# 💥 Create an OpenAI client
openai = Muna(access_key="...").beta.openai
# 🔥 Create a chat completion
completion = openai.chat.completions.create(
model="@openai/gpt-oss-20b",
messages=[{ "role": "user", "content": "What is the capital of France?" }]
)
# 🚀 Print the result
print(completion.choices[0].message)
```
Our OpenAI-style client in `muna.beta.openai` that has the same interface
as the official OpenAI client. This allows you to **migrate in two lines of code**.
The first time you run the code above might take a few minutes, because we have to download the (rather large) model weights. Subsequent runs should take a few seconds.
## Run on a Datacenter GPU
Muna's central feature is the ability to choose where inference runs, on each request. Let's run
the same model on a datacenter GPU:
```ts JavaScript icon="js" focus={1,5} theme={null}
// 🔥 Create a chat completion with a datacenter GPU
const completion = await openai.chat.completions.create({
model: "@openai/gpt-oss-20b",
messages: [{ role: "user", content: "What is the capital of France?" }],
acceleration: "remote_a100"
});
```
```py Python icon="python" focus={1,5} theme={null}
# 🔥 Create a chat completion with a datacenter GPU
completion = openai.chat.completions.create(
model="@openai/gpt-oss-20b",
messages=[{ "role": "user", "content": "What is the capital of France?" }],
acceleration="remote_a100"
)
```
## Next Steps
With Muna, you control which model to run, and where to run it, with zero infrastructure setup.
Explore some popular models you can use immediately.
Come learn more and ask questions in our community Slack.
# Choosing Inference Placement
Source: https://docs.muna.ai/predictions/accelerate
Choosing where each and every inference runs.
Muna's signature feature is allowing developers to choose where inference runs, per-request.
## Running with Adaptive Placement
Muna can adaptively search for the best hardware to run models, depending on your cost, latency, and throughput requirements. Use the `muna.predictions.create` method, and specify your constraints in natural language:
```ts JavaScript icon="js" focus={1,5} theme={null}
// 🔥 Run inference with the lowest latency
const prediction = await muna.predictions.create({
tag: "@openai/gpt-oss-120b",
inputs: { messages },
acceleration: "lowest latency"
});
```
```py Python icon="python" focus={1,5} theme={null}
# 🔥 Run inference with the lowest latency
prediction = muna.predictions.create(
tag="@openai/gpt-oss-120b",
inputs={ "messages": messages },
acceleration="lowest latency"
)
```
```ts React Native icon="react" focus={1,5} theme={null}
// 🔥 Run inference with the lowest latency
const prediction = await muna.predictions.create({
tag: "@openai/gpt-oss-120b",
inputs: { messages },
acceleration: "lowest latency"
});
```
```swift iOS icon="swift" focus={1,5} theme={null}
// 🔥 Run inference with the lowest latency
let prediction = try await muna.predictions.create(
tag: "@openai/gpt-oss-120b",
inputs: ["messages": messages],
acceleration: "lowest latency"
)
```
```kt Android icon="android" focus={1,5} theme={null}
// 🔥 Run inference with the lowest latency
val prediction = muna.predictions.create(
"@openai/gpt-oss-120b",
mapOf("messages" to messages),
"lowest latency"
)
```
```csharp Unity icon="unity" focus={1,5} theme={null}
// 🔥 Run inference with the lowest latency
var prediction = await muna.Beta.Predictions.Remote.Create(
tag: "@openai/gpt-oss-120b",
inputs: new() { ["messages"] = messages },
acceleration: "lowest latency"
);
```
This feature is in early alpha, and is only offered to specific teams.
Request access [on our Slack](https://muna.ai/slack).
### Specifying Placement Constraints
We strongly recommend anchoring your placement constraints around these three canonical intents:
| Intent | Examples |
| :------------- | :---------------------------------------------------------------- |
| **Cost** | `cheapest`, `lowest cost in the cloud`, `under $0.02` |
| **Latency** | `fastest`, `minimize latency`, `lowest latency that runs locally` |
| **Throughput** | `highest throughput`, `at least 100 requests per second` |
You can also specify constraints with combinations of cost, latency, and
throughput intents e.g. `lowest cost under 200ms`.
## Running on Datacenter GPUs
Use the `muna.predictions.create` method, and specify a `remote_*` acceleration to run inference on a datacenter GPU:
```ts JavaScript icon="js" focus={1,5} theme={null}
// 🔥 Run inference with an Nvidia B200 GPU
const prediction = await muna.predictions.create({
tag: "@bytedance/depth-anything-3",
inputs: { image },
acceleration: "remote_b200"
});
```
```py Python icon="python" focus={1,5} theme={null}
# 🔥 Run inference with an Nvidia B200 GPU
prediction = muna.predictions.create(
tag="@bytedance/depth-anything-3",
inputs={ "image": image },
acceleration="remote_b200"
)
```
```ts React Native icon="react" focus={1,5} theme={null}
// 🔥 Run inference with an Nvidia B200 GPU
const prediction = await muna.beta.predictions.remote.create({
tag: "@bytedance/depth-anything-3",
inputs: { image },
acceleration: "remote_b200"
});
```
```swift iOS icon="swift" focus={1,5} theme={null}
// 🔥 Run inference with an Nvidia B200 GPU
let prediction = try await muna.beta.predictions.remote.create(
tag: "@fxn/greeting",
inputs: ["image": image],
acceleration: .remote_b200
)
```
```kt Android icon="android" focus={1,5} theme={null}
// 🔥 Run inference with an Nvidia B200 GPU
val prediction = muna.beta.predictions.remote.create(
"@bytedance/depth-anything-3",
mapOf("image" to image),
"remote_b200"
)
```
```csharp Unity icon="unity" focus={1,5} theme={null}
// 🔥 Run inference with an Nvidia B200 GPU
var prediction = await muna.Predictions.Create(
tag: "@fxn/greeting",
inputs: new() { ["name"] = "Sosa" },
acceleration: "remote_b200"
);
```
### Supported Datacenter GPUs
Below are the currently supported cloud GPUs:
| Acceleration | Notes |
| :--------------- | :--------------------------------------------------------------------- |
| `remote_auto` | Run inference on the ideal datacenter hardware. |
| `remote_cpu` | Run inference on AMD CPU servers. |
| `remote_a10` | Run inference on an Nvidia A10 GPU. |
| `remote_a100` | Run inference on an Nvidia A100 GPU. |
| `remote_h100` | Run inference on an Nvidia H100 GPU. |
| `remote_b200` | Run inference on an Nvidia B200 GPU. |
| `remote_mi350x` | Run inference on an AMD MI350X GPU. **Coming soon**. |
| `remote_mi355x` | Run inference on an AMD MI355X GPU. **Coming soon**. |
| `remote_qaic100` | Run inference on a Qualcomm Cloud AI 100 accelerator. **Coming soon**. |
If you want to self-host the GPU servers in your VPC or on-prem, [reach out to us](mailto:hi@muna.ai).
## Running Locally
Use the `muna.predictions.create` method to run inference locally:
```ts JavaScript icon="js" focus={1,5} theme={null}
// 🔥 Run inference with the local NPU
const prediction = await muna.predictions.create({
tag: "@bytedance/depth-anything-3",
inputs: { image },
acceleration: "local_npu"
});
```
```py Python icon="python" focus={1,5} theme={null}
# 🔥 Run inference with the local NPU
prediction = muna.predictions.create(
tag="@bytedance/depth-anything-3",
inputs={ "image": image },
acceleration="local_npu"
)
```
```ts React Native icon="react" focus={1,5} theme={null}
// 🔥 Run inference with the local NPU
const prediction = await muna.predictions.create({
tag: "@bytedance/depth-anything-3",
inputs: { image },
acceleration: "local_npu"
});
```
```swift iOS icon="swift" focus={1,5} theme={null}
// 🔥 Run inference with the Apple Neural Engine
let prediction = try await muna.predictions.create(
tag: "@bytedance/depth-anything-3",
inputs: ["image": image],
acceleration: .npu
)
```
```kt Android icon="android" focus={1,5} theme={null}
// 🔥 Run inference with the local NPU
val prediction = muna.predictions.create(
"@bytedance/depth-anything-3",
mapOf("image" to image),
Acceleration.NPU
)
```
```csharp Unity icon="unity" focus={1,5} theme={null}
// 🔥 Run inference with the local NPU
var prediction = await muna.Predictions.Create(
tag: "@bytedance/depth-anything-3",
inputs: new() { ["image"] = image },
acceleration: @"local_npu"
);
```
### Supported Local Processors
Below are the currently supported local processors:
| Acceleration | Notes |
| :----------- | :------------------------------------------------------------- |
| `local_cpu` | Use the CPU to accelerate predictions. This is always enabled. |
| `local_gpu` | Use the GPU to accelerate predictions. |
| `local_npu` | Use the neural processor to accelerate predictions. |
Muna currently does not support multi-GPU local acceleration. This is planned for the future.
### Specifying the Local GPU
Some Muna clients allow you to specify the acceleration device used to make predictions.
Our clients expose this field as an untyped integer or pointer.
The underlying type depends on the current operating system:
| OS | Device type | Notes |
| -------- | --------------- | ------------------------------------------------------------------------------------------------------ |
| Android | - | Currently unsupported. |
| iOS | `id` | [Metal device](https://developer.apple.com/documentation/metal/mtldevice?language=objc). |
| Linux | `int*` | Pointer to [CUDA device ID](https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__DEVICE.html). |
| macOS | `id` | [Metal device](https://developer.apple.com/documentation/metal/mtldevice?language=objc). |
| visionOS | `id` | [Metal device](https://developer.apple.com/documentation/metal/mtldevice?language=objc). |
| Web | `GPUDevice` | [WebGPU device](https://developer.mozilla.org/en-US/docs/Web/API/GPUDevice). |
| Windows | `ID3D12Device*` | [DirectX 12 device](https://learn.microsoft.com/en-us/windows/win32/api/d3d12/nn-d3d12-id3d12device). |
The prediction `device` is merely a hint. Setting a `device` does not guarantee that all
or any operation in the prediction function will actually use that acceleration device.
**You should absolutely (absolutely) never ever do this unless you know what the hell you're doing**.
# Running Inference
Source: https://docs.muna.ai/predictions/create
Running compiled models.
The very first step in making predictions is finding or compiling a model:
Explore public models on Muna.
Compile a model with Muna.
## Making Predictions
Making predictions with Muna can be done in as little as two lines of code.
```ts JavaScript icon="js" theme={null}
import { Muna } from "muna"
// 💥 Create your Muna client
const muna = new Muna({ accessKey: "..." });
// 🔥 Run the prediction locally
const prediction = await muna.predictions.create({
tag: "@fxn/greeting",
inputs: { name: "Yusuf" }
});
// 🚀 Print the result
console.log(prediction.results[0]);
```
```py Python icon="python" theme={null}
from muna import Muna
# 💥 Create your Muna client
muna = Muna(access_key="...")
# 🔥 Run the prediction locally
prediction = muna.predictions.create(
tag="@fxn/greeting",
inputs={ "name": "Muna" }
)
# 🚀 Use the results
print(prediction.results[0])
```
```ts React Native icon="react" theme={null}
import { Muna } from "@muna/expo"
// 💥 Create your Muna client
const muna = new Muna({ accessKey: "..." });
// 🔥 Run the prediction locally
const prediction = await muna.predictions.create({
tag: "@fxn/greeting",
inputs: { name: "Yusuf" }
});
// 🚀 Print the result
console.log(prediction.results[0]);
```
```swift iOS icon="swift" theme={null}
import Muna
// 💥 Create your Muna client
let muna = Muna(accessKey: "...")
// 🔥 Run the prediction locally
let prediction = try await muna.predictions.create(
tag: "@fxn/greeting",
inputs: [ "name": "Terri" ]
)
// 🚀 Use the results
print("\(prediction.results![0]!)")
```
```kt Android icon="android" theme={null}
import ai.muna.muna.Muna
// 💥 Create your Muna client
val muna = Muna("...")
// 🔥 Run the prediction locally
val prediction = muna.predictions.create(
"@fxn/greeting",
mapOf("name" to "Timi")
)
// 🚀 Use the results
println(prediction.results!![0])
```
```csharp Unity icon="unity" theme={null}
using Muna;
// 💥 Create your Muna client
var muna = new Muna(accessKey: "...");
// 🔥 Run the prediction locally
var prediction = await muna.Predictions.Create(
tag: "@fxn/greeting",
inputs: new() { ["name"] = "Peter" }
);
// 🚀 Use the results
Debug.Log(prediction.results[0]);
```
```rust Rust icon="rust" theme={null}
use std::collections::HashMap;
use muna::{Muna, Value};
// 💥 Create your Muna client
let muna = Muna::new(Some("..."), None);
// 🔥 Run the prediction locally
let mut inputs = HashMap::new();
inputs.insert("name".to_string(), "Yusuf".into());
let prediction = muna.predictions.create(
"@fxn/greeting",
Some(inputs),
None,
None,
None,
).await.unwrap();
// 🚀 Print the result
println!("{:?}", prediction.results.unwrap()[0]);
```
### Using Prediction Values
Muna supports a fixed set of value types for prediction input and output values:
Muna supports the following floating-point numbers:
| Muna value type | C/C++ type | Description |
| :-------------- | :-------------------------------------------------------------------- | :------------------------------------- |
| `float16` | [`float16_t`](https://en.cppreference.com/w/cpp/types/floating-point) | IEEE 754 16-bit floating point number. |
| `float32` | `float` | IEEE 754 32-bit floating point number. |
| `float64` | `double` | IEEE 754 64-bit floating point number. |
```ts JavaScript icon="js" theme={null}
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
radius: 4.5
}
});
const radius = prediction.results[0] as number;
```
```py Python icon="python" theme={null}
prediction = muna.predictions.create(
tag="@fxn/identity",
inputs={
"radius": 4.5
}
)
radius: float = prediction.results[0]
```
```ts React Native icon="react" theme={null}
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
radius: 4.5
}
});
const radius = prediction.results[0] as number;
```
```swift iOS icon="swift" theme={null}
let prediction = try await muna.predictions.create(
tag: "@fxn/identity",
inputs: [
"radius": 4.5
]
)
let radius = prediction.results![0] as! Float
```
```kt Android icon="android" theme={null}
val prediction = muna.predictions.create(
"@fxn/identity",
mapOf(
"radius" to 4.5f
)
);
val radius = (Float)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
var prediction = await muna.Predictions.Create(
tag: "@fxn/identity",
inputs: new() {
["radius"] = 4.5f
}
);
var radius = (float)prediction.results[0];
```
```rust Rust icon="rust" theme={null}
let mut inputs = HashMap::new();
inputs.insert("radius".to_string(), 4.5f32.into());
let prediction = muna.predictions.create(
"@fxn/identity",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::Float(radius) = &prediction.results.unwrap()[0] else { unreachable!() };
```
In languages that don't support fixed-size floating point scalars, the data type for floating point values
defaults to `float32`. Use a tensor constructor to explicitly specify the data type.
Support for half-precision floating point scalars `float16` is planned for the future depending on language support.
Muna supports floating point vectors (i.e. one-dimensional floating point tensors):
```ts JavaScript icon="js" theme={null}
import type { Tensor } from "muna"
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
vector: new Float32Array([ 1.2, 2.2, 3.2, 4.5 ])
}
});
const vector = prediction.results[0] as Tensor;
```
```py Python icon="python" theme={null}
import numpy as np
prediction = muna.predictions.create(
tag="@fxn/identity",
inputs={
"vector": np.array([1.2, 2.2, 3.2, 4.5], dtype="float32")
}
)
vector: np.ndarray = prediction.results[0]
```
```ts React Native icon="react" theme={null}
import type { Tensor } from "@muna/expo"
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
vector: new Float32Array([ 1.2, 2.2, 3.2, 4.5 ])
}
});
const vector = prediction.results[0] as Tensor;
```
```kt Android icon="android" theme={null}
import ai.muna.muna.types.Float32Tensor;
val prediction = muna.predictions.create(
"@fxn/identity",
mapOf(
"vector" to floatArrayOf(1.2f, 2.2f, 3.2f, 4.5f)
)
);
val vector = (Float32Tensor)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
var prediction = await muna.Predictions.Create(
tag: "@fxn/identity",
inputs: new() {
["vector"] = new [] { 1.2f, 2.2f, 3.2f, 4.5f }
}
);
var vector = (Tensor)prediction.results[0];
```
```rust Rust icon="rust" theme={null}
use muna::{Value, Tensor, TensorData};
let mut inputs = HashMap::new();
inputs.insert("vector".to_string(), Value::Tensor(Tensor {
data: TensorData::Float32(vec![1.2, 2.2, 3.2, 4.5]),
shape: vec![4],
}));
let prediction = muna.predictions.create(
"@fxn/identity",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::Tensor(vector) = &prediction.results.unwrap()[0] else { unreachable!() };
```
Although Muna supports input vectors, predictors will **always** output either scalars or `Tensor` instances--never plain vectors.
Muna supports floating point tensors:
```ts JavaScript icon="js" theme={null}
import type { Tensor } from "muna"
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
matrix: {
data: new Float64Array([ 1.2, 2.2, 3.2, 4.5 ]),
shape: [2, 2]
} satisfies Tensor
}
});
const matrix = prediction.results[0] as Tensor;
```
```py Python icon="python" theme={null}
import numpy as np
prediction = muna.predictions.create(
tag="@fxn/identity",
inputs={
"matrix": np.array([ [1.2, 2.2], [3.2, 4.5] ], dtype="float64")
}
)
matrix: np.ndarray = prediction.results[0]
```
```ts React Native icon="react" theme={null}
import type { Tensor } from "@muna/expo"
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
matrix: {
data: new Float64Array([ 1.2, 2.2, 3.2, 4.5 ]),
shape: [2, 2]
} satisfies Tensor
}
});
const matrix = prediction.results[0] as Tensor;
```
```kt Android icon="android" theme={null}
import ai.muna.muna.types.Float64Tensor;
val prediction = muna.predictions.create(
"@fxn/identity",
mapOf(
"matrix" to Float64Tensor(
doubleArrayOf(1.2, 2.2, 3.2, 4.5), // data
intArrayOf(2, 2) // shape
)
)
);
val matrix = (Float64Tensor)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
var prediction = await muna.Predictions.Create(
tag: "@fxn/identity",
inputs: new () {
["matrix"] = new Tensor(
data: new [] { 1.2, 2.2, 3.2, 4.5 },
shape: new [] { 2, 2 }
)
}
);
var radius = (Tensor)prediction.results[0];
```
```rust Rust icon="rust" theme={null}
use muna::{Value, Tensor, TensorData};
let mut inputs = HashMap::new();
inputs.insert("matrix".to_string(), Value::Tensor(Tensor {
data: TensorData::Float64(vec![1.2, 2.2, 3.2, 4.5]),
shape: vec![2, 2],
}));
let prediction = muna.predictions.create(
"@fxn/identity",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::Tensor(matrix) = &prediction.results.unwrap()[0] else { unreachable!() };
```
Muna supports several signed and unsigned integer scalars:
| Muna value type | C/C++ type | Description |
| :-------------- | :--------- | :----------------------- |
| `int8` | `int8_t` | Signed 8-bit integer. |
| `int16` | `int16_t` | Signed 16-bit integer. |
| `int32` | `int32_t` | Signed 32-bit integer. |
| `int64` | `int64_t` | Signed 64-bit integer. |
| `uint8` | `uint8_t` | Unsigned 8-bit integer. |
| `uint16` | `uint16_t` | Unsigned 16-bit integer. |
| `uint32` | `uint32_t` | Unsigned 32-bit integer. |
| `uint64` | `uint64_t` | Unsigned 64-bit integer. |
```ts JavaScript icon="js" theme={null}
const prediction = await muna.predictions.create({
tag: "@fxn/squeeze",
inputs: {
oranges: 12
}
});
const cups = prediction.results[0] as number;
```
```py Python icon="python" theme={null}
prediction = muna.predictions.create(
tag="@fxn/squeeze",
inputs={
"oranges": 12
}
)
cups: int = prediction.results[0]
```
```ts React Native icon="react" theme={null}
const prediction = await muna.predictions.create({
tag: "@fxn/squeeze",
inputs: {
oranges: 12
}
});
const cups = prediction.results[0] as number;
```
```kt Android icon="android" theme={null}
val prediction = muna.predictions.create(
"@fxn/squeeze",
mapOf(
"oranges" to 12
)
);
val cups = (Int)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
var prediction = await muna.Predictions.Create(
tag: "@fxn/squeeze",
inputs: new() {
["oranges"] = 12
}
);
var cups = (int)prediction.results[0];
```
```rust Rust icon="rust" theme={null}
let mut inputs = HashMap::new();
inputs.insert("oranges".to_string(), 12i32.into());
let prediction = muna.predictions.create(
"@fxn/squeeze",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::Int(cups) = &prediction.results.unwrap()[0] else { unreachable!() };
```
When integer scalars are passed to predictors, the data type defaults to `int32`. Use
a tensor constructor to explicitly specify the data type.
Muna supports integer vectors (i.e. one-dimensional integer tensors) of the aforementioned integer types:
```ts JavaScript icon="js" theme={null}
import type { Tensor } from "muna"
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
vector: new Int16Array([ 1, 2, 3, 4 ])
}
});
const vector = prediction.results[0] as Tensor;
```
```py Python icon="python" theme={null}
import numpy as np
prediction = muna.predictions.create(
tag="@fxn/identity",
inputs={
"vector": np.array([1, 2, 3, 4], dtype="int16")
}
)
vector: np.ndarray = prediction.results[0]
```
```ts React Native icon="react" theme={null}
import type { Tensor } from "@muna/expo"
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
vector: new Int16Array([ 1, 2, 3, 4 ])
}
});
const vector = prediction.results[0] as Tensor;
```
```kt Android icon="android" theme={null}
import ai.muna.muna.types.Int16Tensor;
val prediction = muna.predictions.create(
"@fxn/identity",
mapOf(
"vector" to shortArrayOf(1, 2, 3, 4)
)
);
val vector = (Int16Tensor)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
var prediction = await muna.Predictions.Create(
tag: "@fxn/identity",
inputs: new () {
["vector"] = new short[] { 1, 2, 3, 4 }
}
);
var vector = (Tensor)prediction.results[0];
```
```rust Rust icon="rust" theme={null}
use muna::{Value, Tensor, TensorData};
let mut inputs = HashMap::new();
inputs.insert("vector".to_string(), Value::Tensor(Tensor {
data: TensorData::Int16(vec![1, 2, 3, 4]),
shape: vec![4],
}));
let prediction = muna.predictions.create(
"@fxn/identity",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::Tensor(vector) = &prediction.results.unwrap()[0] else { unreachable!() };
```
Although Muna supports input vectors, predictors will **always** output either scalars or `Tensor` instances--never plain vectors.
Muna supports integer tensors:
```ts JavaScript icon="js" theme={null}
import type { Tensor } from "muna"
const prediction = await muna.predictions.create({
tag: "@fxn/transpose",
inputs: {
matrix: {
data: new Int16Array([ 1, 2, 3, 4 ]),
shape: [2, 2]
} satisfies Tensor
}
});
const matrix = prediction.results[0] as Tensor;
```
```py Python icon="python" theme={null}
import numpy as np
prediction = muna.predictions.create(
tag="@fxn/transpose",
inputs={
"matrix": np.array([ [1, 2], [3, 4] ], dtype="int16")
}
)
matrix: np.ndarray = prediction.results[0]
```
```ts React Native icon="react" theme={null}
import type { Tensor } from "@muna/expo"
const prediction = await muna.predictions.create({
tag: "@fxn/transpose",
inputs: {
matrix: {
data: new Int16Array([ 1, 2, 3, 4 ]),
shape: [2, 2]
} satisfies Tensor
}
});
const matrix = prediction.results[0] as Tensor;
```
```kt Android icon="android" theme={null}
import ai.muna.muna.types.Int16Tensor;
val prediction = muna.predictions.create(
"@fxn/transpose",
mapOf(
"matrix" to Int16Tensor(
shortArrayOf(1, 2, 3, 4),
intArrayOf(2, 2)
)
)
);
val matrix = (Int16Tensor)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
var prediction = await muna.Predictions.Create(
tag: "@fxn/transpose",
inputs: new() {
["matrix"] = new Tensor(
data: new [] { 1, 2, 3, 4 },
shape: new [] { 2, 2 }
)
}
);
var radius = (Tensor)prediction.results[0];
```
```rust Rust icon="rust" theme={null}
use muna::{Value, Tensor, TensorData};
let mut inputs = HashMap::new();
inputs.insert("matrix".to_string(), Value::Tensor(Tensor {
data: TensorData::Int16(vec![1, 2, 3, 4]),
shape: vec![2, 2],
}));
let prediction = muna.predictions.create(
"@fxn/transpose",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::Tensor(matrix) = &prediction.results.unwrap()[0] else { unreachable!() };
```
Unsigned integer tensors are not supported in our Android client because of missing language support in Java.
Muna supports boolean scalars:
```ts JavaScript icon="js" theme={null}
const prediction = await muna.predictions.create({
tag: "@fxn/negate",
inputs: {
value: true
}
});
const truthy = prediction.results[0] as boolean;
```
```py Python icon="python" theme={null}
prediction = muna.predictions.create(
tag="@fxn/negate",
inputs={
"value": True
}
)
truthy: bool = prediction.results[0]
```
```ts React Native icon="react" theme={null}
const prediction = await muna.predictions.create({
tag: "@fxn/negate",
inputs: {
value: true
}
});
const truthy = prediction.results[0] as boolean;
```
```kt Android icon="android" theme={null}
val prediction = muna.predictions.create(
"@fxn/negate",
mapOf(
"value" to true
)
);
val truthy = (Boolean)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
var prediction = await muna.Predictions.Create(
tag: "@fxn/negate",
inputs: new () {
["value"] = true
}
);
var truthy = (bool)prediction.results[0];
```
```rust Rust icon="rust" theme={null}
let mut inputs = HashMap::new();
inputs.insert("value".to_string(), true.into());
let prediction = muna.predictions.create(
"@fxn/negate",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::Bool(truthy) = &prediction.results.unwrap()[0] else { unreachable!() };
```
Muna supports boolean vectors (i.e. one-dimensional boolean tensors):
```ts JavaScript icon="js" theme={null}
import { BoolArray, type Tensor } from "muna"
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
vector: new BoolArray([ true, true, false, true ])
}
});
const vector = prediction.results[0] as Tensor;
```
```py Python icon="python" theme={null}
import numpy as np
prediction = muna.predictions.create(
tag="@fxn/identity",
inputs={
"vector": np.array([True, True, False, True], dtype="bool")
}
)
vector: np.ndarray = prediction.results[0]
```
```ts React Native icon="react" theme={null}
import { BoolArray, type Tensor } from "@muna/expo"
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
vector: new BoolArray([ true, true, false, true ])
}
});
const vector = prediction.results[0] as Tensor;
```
```kt Android icon="android" theme={null}
import ai.muna.muna.types.BoolTensor;
val prediction = muna.predictions.create(
"@fxn/identity",
mapOf(
"vector" to booleanArrayOf(true, true, false, true)
)
);
val vector = (BoolTensor)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
var prediction = await muna.Predictions.Create(
tag: "@fxn/identity",
inputs: new () {
["vector"] = new[] { true, true, false, true }
}
);
var vector = (Tensor)prediction.results[0];
```
```rust Rust icon="rust" theme={null}
use muna::{Value, Tensor, TensorData};
let mut inputs = HashMap::new();
inputs.insert("vector".to_string(), Value::Tensor(Tensor {
data: TensorData::Bool(vec![true, true, false, true]),
shape: vec![4],
}));
let prediction = muna.predictions.create(
"@fxn/identity",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::Tensor(vector) = &prediction.results.unwrap()[0] else { unreachable!() };
```
Although Muna supports input vectors, predictors will **always** output either scalars or `Tensor` instances--never plain vectors.
Muna supports boolean tensors:
```ts JavaScript icon="js" theme={null}
import { BoolArray, type Tensor } from "muna"
const prediction = await muna.predictions.create({
tag: "@fxn/transpose",
inputs: {
matrix: {
data: new BoolArray([ true, true, false, true ]),
shape: [2, 2]
} satisfies Tensor
}
});
const matrix = prediction.results[0] as Tensor;
```
```py Python icon="python" theme={null}
import numpy as np
prediction = muna.predictions.create(
tag="@fxn/transpose",
inputs={
"matrix": np.array([ [True, True], [False, True] ], dtype="bool")
}
)
matrix: np.ndarray = prediction.results[0]
```
```ts React Native icon="react" theme={null}
import { BoolArray, type Tensor } from "@muna/expo"
const prediction = await muna.predictions.create({
tag: "@fxn/transpose",
inputs: {
matrix: {
data: new BoolArray([ true, true, false, true ]),
shape: [2, 2]
} satisfies Tensor
}
});
const matrix = prediction.results[0] as Tensor;
```
```kt Android icon="android" theme={null}
import ai.muna.muna.types.BoolTensor;
val prediction = muna.predictions.create(
"@fxn/transpose",
mapOf(
"matrix" to BoolTensor(
booleanArrayOf(true, true, false, true),
intArrayOf(2, 2)
)
)
);
val matrix = (BoolTensor)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
var prediction = await muna.Predictions.Create(
tag: "@fxn/transpose",
inputs: new () {
["matrix"] = new Tensor(
data: new [] { true, true, false, true },
shape: new [] { 2, 2 }
)
}
);
var radius = (Tensor)prediction.results[0];
```
```rust Rust icon="rust" theme={null}
use muna::{Value, Tensor, TensorData};
let mut inputs = HashMap::new();
inputs.insert("matrix".to_string(), Value::Tensor(Tensor {
data: TensorData::Bool(vec![true, true, false, true]),
shape: vec![2, 2],
}));
let prediction = muna.predictions.create(
"@fxn/transpose",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::Tensor(matrix) = &prediction.results.unwrap()[0] else { unreachable!() };
```
Muna assumes that boolean values are 1 byte.
Muna supports string values:
```ts JavaScript icon="js" theme={null}
const prediction = await muna.predictions.create({
tag: "@fxn/upper",
inputs: {
text: "hello from function"
}
});
const uppercase = prediction.results[0] as string;
```
```py Python icon="python" theme={null}
prediction = muna.predictions.create(
tag="@fxn/upper",
inputs={
"text": "hello from function"
}
)
uppercase: str = prediction.results[0]
```
```ts React Native icon="react" theme={null}
const prediction = await muna.predictions.create({
tag: "@fxn/upper",
inputs: {
text: "hello from function"
}
});
const uppercase = prediction.results[0] as string;
```
```kt Android icon="android" theme={null}
val prediction = muna.predictions.create(
"@fxn/upper",
mapOf(
"text" to "hello from function"
)
);
val uppercase = (String)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
var prediction = await muna.Predictions.Create(
tag: "@fxn/upper",
inputs: new () {
["text"] = "hello from function"
}
);
var uppercase = prediction.results[0] as string;
```
```rust Rust icon="rust" theme={null}
let mut inputs = HashMap::new();
inputs.insert("text".to_string(), "hello from function".into());
let prediction = muna.predictions.create(
"@fxn/upper",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::String(uppercase) = &prediction.results.unwrap()[0] else { unreachable!() };
```
Muna supports lists of values, each with potentially different types:
```ts JavaScript icon="js" theme={null}
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
elements: ["hello", 10, false]
}
});
const elements = prediction.results[0] as any[];
```
```py Python icon="python" theme={null}
from typing import Any, List
prediction = muna.predictions.create(
tag="@fxn/identity",
inputs={
"elements": ["hello", 10, False]
}
)
elements: List[Any] = prediction.results[0]
```
```ts React Native icon="react" theme={null}
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
elements: ["hello", 10, false]
}
});
const elements = prediction.results[0] as any[];
```
```kt Android icon="android" theme={null}
val prediction = muna.predictions.create(
"@fxn/identity",
mapOf(
"elements" to listOf("hello", 10, false)
)
);
val elements = (List)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
using Newtonsoft.Json.Linq;
var prediction = await muna.Predictions.Create(
tag: "@fxn/identity",
inputs: new() {
["elements"] = new object[] { "hello", 10, false } // can be any `T : IList`
}
);
var elements = prediction.results[0] as JArray;
```
```rust Rust icon="rust" theme={null}
use muna::Value;
let mut inputs = HashMap::new();
inputs.insert("elements".to_string(), Value::List(vec![
serde_json::json!("hello"),
serde_json::json!(10),
serde_json::json!(false),
]));
let prediction = muna.predictions.create(
"@fxn/identity",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::List(elements) = &prediction.results.unwrap()[0] else { unreachable!() };
```
Input list values **must** be JSON-serializable.
Muna supports dictionary values:
```ts JavaScript icon="js" theme={null}
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
person: {
name: "Sara",
age: 27
}
}
});
const person = prediction.results[0] as Record;
```
```py Python icon="python" theme={null}
from typing import Any, Dict
prediction = muna.predictions.create(
tag="@fxn/identity",
inputs={
"person": {
"name": "Sara",
"age": 27
}
}
)
person: Dict[str, Any] = prediction.results[0]
```
```ts React Native icon="react" theme={null}
const prediction = await muna.predictions.create({
tag: "@fxn/identity",
inputs: {
person: {
name: "Sara",
age: 27
}
}
});
const person = prediction.results[0] as Record;
```
```kt Android icon="android" theme={null}
val prediction = muna.predictions.create(
"@fxn/identity",
mapOf(
"person" to mapOf(
"name" to "Sara",
"age" to 27
)
)
);
val person = (Map)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
using System.Collections.Generic;
using Newtonsoft.Json.Linq;
var prediction = await muna.Predictions.Create(
tag: "@fxn/identity",
inputs: new() {
["person"] = new Dictionary { // can be any `T : IDictionary`
["name"] = "Sara",
["age"] = 27
}
}
);
var person = prediction.results[0] as JObject;
```
```rust Rust icon="rust" theme={null}
use muna::Value;
let mut inputs = HashMap::new();
let mut person = serde_json::Map::new();
person.insert("name".to_string(), serde_json::json!("Sara"));
person.insert("age".to_string(), serde_json::json!(27));
inputs.insert("person".to_string(), Value::Dict(person));
let prediction = muna.predictions.create(
"@fxn/identity",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::Dict(person) = &prediction.results.unwrap()[0] else { unreachable!() };
```
Input dictionary values **must** be JSON-serializable.
Muna supports images, represented as raw pixel buffers with 8 bytes per pixel and interleaved by channel.
Muna supports three pixel buffer formats:
| Pixel format | Channels | Description |
| :----------- | :------- | :--------------------------------------- |
| `A8` | 1 | Single channel luminance or alpha image. |
| `RGB888` | 3 | Color image without alpha channel. |
| `RGBA8888` | 4 | Color image with alpha channel. |
Some client SDKs provide `Image` utility types for working with images:
```ts JavaScript icon="js" theme={null}
import type { Image } from "muna"
const prediction = await muna.predictions.create({
tag: "@vision-co/remove-background",
inputs: {
image: {
data: new Uint8ClampedArray(1280 * 720 * 3),
width: 1280,
height: 720,
channels: 3
} satisfies Image
}
});
const image = prediction.results[0] as Image;
```
```py Python icon="python" theme={null}
from PIL import Image
prediction = muna.predictions.create(
tag="@vision-co/remove-background",
inputs={
"image": Image.open("cat.jpg")
}
)
image: Image.Image = prediction.results[0]
```
```ts React Native icon="react" theme={null}
import type { Image } from "@muna/expo"
const prediction = await muna.predictions.create({
tag: "@vision-co/remove-background",
inputs: {
image: {
data: new Uint8ClampedArray(1280 * 720 * 3),
width: 1280,
height: 720,
channels: 3
} satisfies Image
}
});
const image = prediction.results[0] as Image;
```
```kt Android icon="android" theme={null}
import ai.muna.muna.types.Image;
val prediction = muna.predictions.create(
"@vision-co/remove-background",
mapOf(
"image" to Image(
ByteBuffer.allocateDirect(1280 * 720 * 3),
1280, // width
720, // height
3 // channels
)
)
);
val image = (Image)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
var prediction = await muna.Predictions.Create(
tag: "@vision-co/remove-background",
inputs: new () {
["image"] = new Image(
data: new byte[1280 * 720 * 3],
width: 1280,
height: 720,
channels: 3
)
}
);
var image = (Image)prediction.results[0];
```
```rust Rust icon="rust" theme={null}
use muna::{Value, Image};
let mut inputs = HashMap::new();
inputs.insert("image".to_string(), Value::Image(Image {
data: vec![0u8; 1280 * 720 * 3],
width: 1280,
height: 720,
channels: 3,
}));
let prediction = muna.predictions.create(
"@vision-co/remove-background",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::Image(image) = &prediction.results.unwrap()[0] else { unreachable!() };
```
Muna supports binary blobs:
```ts JavaScript icon="js" theme={null}
const prediction = await muna.predictions.create({
tag: "@vision-co/decode-jpeg",
inputs: {
buffer: new ArrayBuffer(1024)
}
});
const buffer = prediction.results[0] as ArrayBuffer;
```
```py Python icon="python" theme={null}
from io import BytesIO
prediction = muna.predictions.create(
tag="@vision-co/decode-jpeg",
inputs={
"buffer": BytesIO(b"\x00\x01") # or `bytes`, `bytearray`, `memoryview`
}
)
buffer: BytesIO = prediction.results[0]
```
```ts React Native icon="react" theme={null}
const prediction = await muna.predictions.create({
tag: "@vision-co/decode-jpeg",
inputs: {
buffer: new ArrayBuffer(1024)
}
});
const buffer = prediction.results[0] as ArrayBuffer;
```
```kt Android icon="android" theme={null}
val prediction = muna.predictions.create(
"@vision-co/decode-png",
mapOf(
"buffer" to ByteArrayInputStream(byteArrayOf(0x1, 0x2)) // must be an `InputStream`
)
);
val buffer = (InputStream)prediction.results[0];
```
```csharp Unity icon="unity" theme={null}
using System.IO;
var prediction = await muna.Predictions.Create(
tag: "@vision-co/decode-jpeg",
inputs: new () {
["buffer"] = new MemoryStream() // must be a `Stream`
}
);
var buffer = (Stream)prediction.results[0];
```
```rust Rust icon="rust" theme={null}
use muna::Value;
let mut inputs = HashMap::new();
inputs.insert("buffer".to_string(), Value::Binary(vec![0x00, 0x01]));
let prediction = muna.predictions.create(
"@vision-co/decode-jpeg",
Some(inputs),
None,
None,
None,
).await.unwrap();
let Value::Binary(buffer) = &prediction.results.unwrap()[0] else { unreachable!() };
```
Because Muna's [security model](/concepts#security-model) prohibits file system access, binary input values
are always fully read into memory before being passed to the predictor.
To make predictions on large files, consider mapping the file into memory
using [`mmap`](https://linux.die.net/man/2/mmap) or your environment's equivalent.
## Streaming Predictions
Muna supports consuming the partial results of a prediction as they are made available by the predictor:
```ts JavaScript icon="js" theme={null}
// 🔥 Create a prediction stream
const stream = await muna.predictions.stream({
tag: "@text-co/split-sentence",
inputs: { text: "Hello world" }
});
// 🚀 Consume the stream
for await (const prediction of stream)
console.log(prediction.results[0]);
```
```py Python icon="python" theme={null}
# 🔥 Create a prediction stream
stream = muna.predictions.stream(
tag="@text-co/split-sentence",
inputs={ "text": "Hello world" }
)
# 🚀 Consume the stream
for prediction in stream:
print(prediction.results[0])
```
```ts React Native icon="react" theme={null}
// 🔥 Create a prediction stream
const stream = await muna.predictions.stream({
tag: "@text-co/split-sentence",
inputs: { text: "Hello world" }
});
// 🚀 Consume the stream
for await (const prediction of stream)
console.log(prediction.results[0]);
```
```swift iOS icon="swift" theme={null}
// 🔥 Create a prediction stream
let stream = try await muna.predictions.stream(
tag: "@text-co/split-sentence",
inputs: ["text": "Hello world"],
)
// 🚀 Consume the stream
for try await prediction in stream {
print("\(prediction.results?[0])")
}
```
```kt Android icon="android" theme={null}
// 🔥 Create a prediction stream
val stream = muna.predictions.stream(
"@text-co/split-sentence",
mapOf("text" to "Hello world")
)
// 🚀 Consume the stream
stream.use {
for (prediction in it.consume())
println(prediction.results!![0])
}
```
```csharp Unity icon="unity" theme={null}
// 🔥 Create a prediction stream
var stream = await muna.Predictions.Stream(
tag: "@text-co/split-sentence",
inputs: new() { ["text"] = "Hello world" }
);
// 🚀 Consume the stream
await foreach (var prediction in stream)
Debug.Log(prediction.results[0]);
```
```rust Rust icon="rust" theme={null}
use std::collections::HashMap;
use futures_util::StreamExt;
// 🔥 Create a prediction stream
let mut inputs = HashMap::new();
inputs.insert("text".to_string(), "Hello world".into());
let mut stream = muna.predictions.stream(
"@text-co/split-sentence",
inputs,
None,
).await.unwrap();
// 🚀 Consume the stream
while let Some(Ok(prediction)) = stream.next().await {
println!("{:?}", prediction.results.unwrap()[0]);
}
```
### Consuming Prediction Streams
Streaming in Muna is designed to fully separate how a prediction function
is implemented from how the function might be consumed. Consider these two predictors:
```py eager.py icon="python" theme={null}
def predict() -> str:
return "hello from Muna"
```
```py generator.py icon="python" theme={null}
def predict() -> Iterator[str]:
yield "hello"
yield "hello from"
yield "hello from Muna"
```
Here are the reuslts of creating vs. streaming each function at runtime:
In this case, the single prediction is returned:
```ts icon="js" theme={null}
// Create a prediction with the eager predictor
const prediction = await muna.predictions.create({
tag: "@muna/eager",
inputs: { }
});
// Display the results
console.log(prediction.results[0]);
// Outputs:
// "hello from Muna"
```
In this case, the Muna client will consume all partial predictions yielded
by the predictor then return **the very last one**:
```ts icon="js" theme={null}
// Create a prediction with the streaming predictor
const prediction = await muna.predictions.create({
tag: "@muna/generator",
inputs: { }
});
// Display the results
console.log(prediction.results[0]);
// Outputs:
// "hello from Muna"
```
In this case, the Muna client will return a prediction stream with the single prediction returned by the predictor:
```ts icon="js" theme={null}
// Create a prediction stream with the eager predictor
const stream = await muna.predictions.stream({
tag: "@muna/eager",
inputs: { }
});
// Display the results
for await (const prediction of stream)
console.log(prediction.results[0]);
// Outputs:
// "hello from Muna"
```
In this case, the Muna client will provide a prediction stream containing all
partial predictions yielded by the predictor:
```ts icon="js" theme={null}
// Create a prediction stream with the generator predictor
const stream = await muna.predictions.stream({
tag: "@muna/generator",
inputs: { }
});
// Display the results
for await (const prediction of stream)
console.log(prediction.results[0]);
// Outputs:
// "hello"
// "hello from"
// "hello from Muna"
```
You can choose how to consume a prediction function depending on what works best for your user experience.
You don't have to care about the underlying function!
# Migrating from OpenAI
Source: https://docs.muna.ai/predictions/openai
Switch to open-source AI models in only 2 lines of code.
OpenAI's client is widely used by developers who consume AI inference in their applications.
Muna's OpenAI client allows developers to switch to open-source AI models in one line of code.
Unlike the official OpenAI client, Muna allows you to specify where the inference runs per-request:
H100s, B200s, or on the local device.
You can easily compile your own custom models to be compatible with Muna's OpenAI client.
[See the guide](/predictors/openai).
## Creating Chat Completions
Muna supports running large language models via our client's
`openai.chat.completions.create` API:
```js JavaScript icon="js" theme={null}
import { Muna } from "muna"
// 💥 Create a Muna client
const openai = new Muna().beta.openai;
// 🔥 Create a chat completion with an Nvidia A10 GPU
const completion = await openai.chat.completions.create({
model: "@google/gemma-3-270m",
messages: [{ role: "user", content: "What is life?" }],
acceleration: "remote_a10"
});
// 🚀 Print the result
console.log(completion.choices[0]);
```
```py Python icon="python" theme={null}
from muna import Muna
# 💥 Create a Muna client
openai = Muna().beta.openai
# 🔥 Create a chat completion with an Nvidia A10 GPU
completion = openai.chat.completions.create(
model="@google/gemma-3-270m",
messages=[{ "role": "user", "content": "What is life?" }],
acceleration="remote_a10"
)
# 🚀 Print the result
print(completion.choices[0].message)
```
```cs Unity icon="unity" theme={null}
using Muna;
using Muna.Beta.OpenAI;
using static Muna.Beta.OpenAI.ChatMessage;
// 💥 Create a Muna client
var openai = MunaUnity.Create().Beta.OpenAI;
// 🔥 Create a chat completion
var completion = await openai.Chat.Completions.Create(
model: "@google/gemma-3-270m",
messages: new[] {
new ChatMessage { Role = "user", Content = "What is life?" }
},
acceleration: "remote_a10"
);
// 🚀 Print the result
Debug.Log(completion.Choices[0].Message);
```
### Streaming Completions
Our OpenAI client also supports creating streaming completions:
```js JavaScript icon="js" theme={null}
import { Muna } from "muna"
// 💥 Create a Muna client
const openai = new Muna().beta.openai;
// 🔥 Stream a chat completion
const stream = await openai.chat.completions.create({
model: "@google/gemma-3-270m",
messages: [{ role: "user", content: "What is life?" }],
stream: true
});
// 🚀 Use completion chunks
for await (const chunk of stream)
...
```
```py Python icon="python" theme={null}
from muna import Muna
# 💥 Create a Muna client
openai = Muna().beta.openai
# 🔥 Stream a chat completion
stream = openai.chat.completions.create(
model="@google/gemma-3-270m",
messages=[{ "role": "user", "content": "What is life?" }],
stream=True
)
# 🚀 Use completion chunks
for chunk in stream:
...
```
```cs Unity icon="unity" theme={null}
using Muna;
using Muna.Beta.OpenAI;
using static Muna.Beta.OpenAI.ChatMessage;
// 💥 Create a Muna client
var openai = MunaUnity.Create().Beta.OpenAI;
// 🔥 Stream a chat completion
var stream = openai.Chat.Completions.Stream(
model: "@google/gemma-3-270m",
messages: new[] {
new ChatMessage { Role = "user", Content = "What is life?" }
},
);
// 🚀 Use completion chunks
await foreach (var chunk in stream)
...
```
## Creating Embeddings
Muna supports running text embedding models via our client's
`openai.embeddings.create` API:
```js JavaScript icon="js" theme={null}
import { Muna } from "muna"
// 💥 Create a Muna client
const openai = new Muna().beta.openai;
// 🔥 Create a text embedding
const embedding = await openai.embeddings.create({
model: "@nomic/nomic-embed-text-v1.5",
input: "What is the capital of France?"
});
// 🚀 Use the embedding
console.log(embedding.data[0].embedding);
```
```py Python icon="python" theme={null}
from muna import Muna
# 💥 Create a Muna client
openai = Muna().beta.openai
# 🔥 Create a text embedding
embedding = openai.embeddings.create(
model="@nomic/nomic-embed-text-v1.5",
input="What is the capital of France?"
)
# 🚀 Use the embedding
print(embedding.data[0].embedding)
```
```csharp Unity icon="unity" theme={null}
using Muna;
// 💥 Create a Muna client
var openai = MunaUnity.Create().Beta.OpenAI;
// 🔥 Create a text embedding
var embedding = await openai.Embeddings.Create(
model: "@nomic/nomic-embed-text-v1.5",
input: "What is the capital of France?"
);
// 🚀 Use the embedding
Debug.Log(embedding.data[0].Floats);
```
## Creating Speech
Muna supports running text-to-speech models via our client's
`openai.audio.speech.create` API:
```js JavaScript icon="js" theme={null}
import { Muna } from "muna"
// 💥 Create a Muna client
const openai = new Muna().beta.openai;
// 🔥 Create speech
const response = await openai.audio.speech.create({
model: "@hexgrad/kokoro-tts",
input: "What a time to be alive",
voice: "af_jessica"
});
// 🚀 Use the speech
console.log(response);
```
```py Python icon="python" theme={null}
from muna import Muna
# 💥 Create a Muna client
openai = Muna().beta.openai
# 🔥 Create speech
response = openai.audio.speech.create(
model="@hexgrad/kokoro-tts",
input="What a time to be alive",
voice="af_jessica"
)
# 🚀 Use the speech
print(response)
```
## Creating Transcriptions
Muna supports using speech-to-text models via our client’s `openai.audio.transcriptions.create` API:
```js JavaScript icon="js" theme={null}
import { Muna } from "muna"
// 💥 Create a Muna client
const openai = new Muna().beta.openai;
// 🔥 Create transcription
const transcription = await openai.audio.transcriptions.create({
model: "@moonshine/moonshine-base",
file: audioFile
});
// 🚀 Use the transcribed text
console.log(transcription.text);
```
```py Python icon="python" theme={null}
from muna import Muna
# 💥 Create a Muna client
openai = Muna().beta.openai
# 🔥 Create transcription
transcription = openai.audio.speech.create(
model="@moonshine/moonshine-base",
file=audio_file
)
# 🚀 Use the transcribed text
print(transcription.text)
```
# Bringing Your Own Compute
Source: https://docs.muna.ai/predictors/byoc
Run compiled models on your own GPUs.
Muna supports deploying compiled models to your GPU fleets, whether on-prem or from third-party GPU platforms. Compiled models can yield up to 45x reductions in cold-start times, along with reduced latency and higher utilization.
![Cumulative time to first token]()
## Deploying to Modal
Use the `muna deploy` CLI command to deploy a compiled model to [Modal](https://modal.com):
```bash theme={null}
# Deploy a compiled LLM to Modal
$ muna deploy @google/gemma-4-26b-a4b-it --provider modal --gpu b200
```
This command will create a lightweight app on Modal that runs an [OpenAI-compatible web server](https://github.com/muna-ai/muna-server). This server then forwards requests to the compiled model.
This command requires the `modal` package to be installed. Run `pip install modal`.
## Deploying to Baseten
Use the `muna deploy` CLI command to deploy a compiled model to [Baseten](https://baseten.co):
```bash theme={null}
# Deploy a compiled LLM to Baseten
$ muna deploy @google/gemma-4-26b-a4b-it --provider baseten --gpu b200
```
This command will create and deploy a lightweight service on Baseten that runs an [OpenAI-compatible web server](https://github.com/muna-ai/muna-server). This server then forwards requests to the compiled model.
This command requires the `truss` package to be installed. Run `pip install truss`.
### Generating a Truss Config
The `muna deploy` CLI command supports emitting a [Truss](https://docs.baseten.co/reference/truss-configuration) config without deploying the app. To generate a Truss config, use the `--dry-run` flag:
```bash theme={null}
# Generate a Truss config for a compiled model
$ muna deploy @google/gemma-4-26b-a4b-it --provider baseten --dry-run
```
# Compiling Models
Source: https://docs.muna.ai/predictors/create
Compiling your models into self-contained binaries.
Muna compiles stateless Python functions into self-contained executable binaries.
Your Python code is lowered directly to native code, and does not rely on
the Python runtime (or any other managed runtime).
## Defining a Function
Write your function, add a docstring, then wrap it with our `@compile` decorator:
```py greeting.py icon="python" theme={null}
from muna import compile
@compile()
def greeting(name: str) -> str:
"""
Say a friendly greeting.
"""
return f"Hey there {name}! We're glad you're using Muna and we hope you like it."
```
The docstring is used as a description, is required, and must be 100 characters or less.
The function **must** specify parameter and return type annotations. [Learn more](/predictors/requirements).
## Compiling the Function
Use the Muna CLI to compile the function. First, make sure you are logged into the Muna CLI:
```bash icon="terminal" theme={null}
# Login to the CLI
$ muna auth login
```
Then run the following command:
```bash icon="terminal" theme={null}
# Compile the function
$ muna compile --overwrite greeting.py
```
Muna will upload your code, perform code generation on your function and its dependencies,
then compile for our supported platforms:
![]()
## Using the Function
Depending on the complexity of your function, it can take anywhere from a few seconds to a
few minutes for the function to be compiled for all platforms. Once the function is compiled, you can
run it everywhere:
```ts JavaScript icon="js" theme={null}
import { Muna } from "muna"
// 💥 Create your Muna client
const muna = new Muna({ accessKey: "..." });
// 🔥 Make a prediction
const prediction = await muna.predictions.create({
tag: "@your-username/greeting",
inputs: { name: "Lina" }
});
// 🚀 Print the result
console.log(prediction.results[0]);
```
```py Python icon="python" theme={null}
from muna import Muna
# 💥 Create your Muna client
muna = Muna(access_key="...")
# 🔥 Make a prediction
prediction = muna.predictions.create(
tag="@your-username/greeting",
inputs={ "name": "Lina" }
)
# 🚀 Use the results
print(prediction.results[0])
```
```swift iOS icon="swift" theme={null}
import Muna
// 💥 Create a Muna client
let muna = Muna(accessKey: "...")
// 🔥 Make a prediction
let prediction = try await muna.predictions.create(
tag: "@your-username/greeting",
inputs: ["name": "Lina"]
)
// 🚀 Use the results
print(prediction.results![0])
```
```kt Android icon="android" theme={null}
import ai.muna.muna.Muna
// 💥 Create a Muna client
val muna = Muna("...")
// 🔥 Make a prediction
val prediction = muna.predictions.create(
"@your-username/greeting",
mapOf("name" to "Lina")
)
// 🚀 Use the results
println(prediction.results!![0])
```
```csharp Unity icon="unity" theme={null}
using Muna;
// 💥 Create your Muna client
var muna = new Muna(accessKey: "...");
// 🔥 Make a prediction
var prediction = await muna.Predictions.Create(
tag: "@your-username/greeting",
inputs: new() { ["name"] = "Lina" }
);
// 🚀 Use the results
Debug.Log(prediction.results[0]);
```
```bash CLI icon="terminal" theme={null}
# 🔥 Make a prediction
$ muna predict @your-username/greeting --name Lina
```
You can check the compilation status of the function at [muna.ai/predictors](https://muna.ai/predictors).
# OpenAI Compatibility
Source: https://docs.muna.ai/predictors/openai
Compiling OpenAI-compatible models.
The OpenAI client is widely used by developers who consume AI inference in their applications. This guide explains how to compile models that can be used via Muna's OpenAI-compatible client by leveraging [parameter annotations](/predictors/requirements#using-parameter-annotations).
Muna's OpenAI-compatible client allows developers to switch to open-source AI models without changing their existing code.
## Compiling Chat Completion Models
You can compile chat completion models compatible with Muna's
`openai.chat.completions.create` interface.
Chat completion functions should accept a list of input messages with type `list[muna.beta.openai.Message]`:
```py llm.py icon="python" focus={2,7-10} theme={null}
from muna import compile, Parameter
from muna.beta.openai import Message
from typing import Annotated
@compile(...)
def create_chat_completion(
messages: Annotated[
list[Message],
Parameter.Generic(description="Messages comprising the conversation so far.")
]
):
...
```
Chat completion functions must return an iterator of completion chunks, with type
`Iterator[muna.beta.openai.ChatCompletionChunk]`:
```py llm.py icon="python" focus={2,11} theme={null}
from muna import compile, Parameter
from muna.beta.openai import ChatCompletionChunk, Message
from typing import Iterator
@compile(...)
def create_chat_completion(
messages: Annotated[
list[Message],
Parameter.Generic(description="Messages comprising the conversation so far.")
]
) -> Iterator[ChatCompletionChunk]:
...
```
We recommend using the [`llama-cpp-python`](https://github.com/abetlen/llama-cpp-python) package to
create chat completions using [`Llama.cpp`](https://github.com/ggml-org/llama.cpp):
```py llm.py icon="python" focus={3,6,15-21} theme={null}
from muna import compile, Parameter
from muna.beta.openai import ChatCompletionChunk, Message
from llama_cpp import Llama
from typing import Iterator
model = Llama(model_path=model_path)
@compile(...)
def create_chat_completion(
messages: Annotated[
list[Message],
Parameter.Generic(description="Messages comprising the conversation so far.")
]
) -> Iterator[ChatCompletionChunk]:
stream = model.create_chat_completion(
messages=messages,
max_tokens=1_000,
stream=True
)
for chunk in stream:
yield chunk
```
## Compiling Embedding Models
You can compile text embedding models compatible with Muna's
`openai.embeddings.create` interface.
Embedding functions should accept a list of input texts to embed, as a `list[str]`:
```py embed_text.py icon="python" focus={7-10} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated
@compile(...)
def embed_text(
texts: Annotated[
list[str],
Parameter.Generic(description="Input texts to embed.")
]
) -> ndarray:
...
```
Embedding functions must return an embedding matrix as a Numpy `ndarray`.
The array must have a [`Parameter.Embedding`](/predictors/requirements#embedding-annotation)
annotation:
```py embed_text.py icon="python" focus={11-14} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated
@compile(...)
def embed_text(
texts: Annotated[
list[str],
Parameter.Generic(description="Input texts to embed.")
]
) -> Annotated[
ndarray,
Parameter.Embedding(description="Embedding matrix.")
]:
...
```
The returned `ndarray` must have a `float32` data type.
The returned `ndarray` must be a 2D array with shape `(N,D)`, where
`N` is the number of input texts and `D` is the embedding dimension.
Some embedding models allow for specifying the number of embedding dimensions, based on Matryoshka representation learning.
To expose this setting, add an `int` parameter with the
[`Parameter.EmbeddingDims`](/predictors/requirements#embedding-dimensions-annotation) annotation:
```py embed_text.py icon="python" focus={11-18} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated
@compile(...)
def embed_text(
texts: Annotated[
list[str],
Parameter.Generic(description="Input texts to embed.")
],
dimensions: Annotated[
int,
Parameter.EmbeddingDims(
description="The number of dimensions the embeddings should have.",
min=256,
max=768
)
] = 768
) -> Annotated[
ndarray,
Parameter.Embedding(description="Embedding matrix.")
]:
...
```
To remain compatible with the OpenAI embeddings interface, the function **must have only one required parameter**. As a result, make sure to specify a default value for all other parameters.
## Compiling Speech Models
You can compile text-to-speech models compatible with Muna's
`openai.audio.speech.create` interface.
Text-to-speech functions should accept an input text `str`:
```py generate_speech.py icon="python" focus={7-10} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated
@compile(...)
def generate_speech(
text: Annotated[
str,
Parameter.Generic(description="Input text.")
]
) -> ndarray:
...
```
Text-to-speech functions must also accept a generation voice argument. We recommend using a
[`Literal`](https://typing.python.org/en/latest/spec/literal.html) or
[`StrEnum`](https://docs.python.org/3/library/enum.html#enum.StrEnum) type.
Regardless of the type you choose, the parameter must have a
[`Parameter.AudioVoice`](/predictors/requirements#audio-voice-annotation) annotation:
```py generate_speech.py icon="python" focus={11-14} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated, Literal
@compile(...)
def generate_speech(
text: Annotated[
str,
Parameter.Generic(description="Input text.")
],
voice: Annotated[
Literal["voice_a", "voice_b"],
Parameter.AudioVoice(description="Voice to use in generating audio.")
]
) -> ndarray:
...
```
The generation voice must be a required parameter, because developers are required to specify
the voice in the OpenAI interface.
Speech generation functions must return the generated audio as a Numpy `ndarray` containing
linear PCM samples. The array must have a [`Parameter.Audio`](/predictors/requirements#audio-annotation)
annotation:
```py generate_speech.py icon="python" focus={15-18} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated, Literal
@compile(...)
def generate_speech(
text: Annotated[
str,
Parameter.Generic(description="Input text.")
],
voice: Annotated[
Literal["voice_a", "voice_b"],
Parameter.AudioVoice(description="Voice to use in generating audio.")
]
) -> Annotated[
ndarray,
Parameter.Audio(description="Generated speech.", sample_rate=24_000)
]:
...
```
The returned `ndarray` must have a `float32` data type.
The returned `ndarray` must either be a 1D array with shape `(F,)` for single channel audio; or a
2D array with shape `(F,C)` where `C` is the channel count (interleaved).
Some text-to-speech functions support configuring the speed of the generated audio. To expose this
setting, add a `float` parameter with a [`Parameter.AudioSpeed`](/predictors/requirements#audio-speed-annotation) annotation:
```py generate_speech.py icon="python" focus={12-19} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated, Literal
@compile(...)
def generate_speech(
text: Annotated[str, Parameter.Generic(description="Input text.")],
voice: Annotated[
Literal["voice_a", "voice_b"],
Parameter.AudioVoice(description="Voice to use in generating audio.")
],
speed: Annotated[
float,
Parameter.AudioSpeed(
description="The speed of the generated audio.",
min=0.25,
max=4.0
)
] = 1.0
) -> Annotated[
ndarray,
Parameter.Audio(description="Generated speech.", sample_rate=24_000)
]:
...
```
The audio speed parameter **must** have a default value, because it is an optional setting in the OpenAI interface.
## Compiling Transcription Models
You can compile speech-to-text models compatible with Muna's
`openai.audio.transcriptions.create` interface.
Transcription functions should accept input audio as a Numpy `ndarray` annotated
with the [`Parameter.Audio`](/predictors/requirements#audio-annotation) annotation:
```py moonshine_base.py icon="python" focus={7-13} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated
@compile(...)
def moonshine_base(
audio: Annotated[
ndarray,
Parameter.Audio(
description="Audio to transcribe with shape (F,C).",
sample_rate=24_000
)
]
) -> str:
...
```
When a user runs your compiled model with an audio file (mp3, wav, etc), the
Muna client will decode it and resample it to your required `sample_rate`.
Transcription functions should return the transcribed text as a string:
```py moonshine_base.py icon="python" focus={14-17} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated
@compile(...)
def moonshine_base(
audio: Annotated[
ndarray,
Parameter.Audio(
description="Audio to transcribe with shape (F,C).",
sample_rate=24_000
)
]
) -> Annotated[
str,
Parameter.Generic(description="Transcribed text.")
]:
...
```
# Defining your Functions
Source: https://docs.muna.ai/predictors/requirements
Ensuring that your functions can be compiled successfully.
Muna supports compiling a tiny-but-growing subset of Python language constructs. Below are requirements
and guidelines for compiling a Python function with Muna:
## Specifying the Function Signature
The prediction function **must** be a module-level function, and **must** have parameter and return
type annotations:
```py icon="python" theme={null}
from muna import compile
@compile(...)
def greeting(name: str) -> str:
return f"Hello {name}"
```
The prediction function **must not** have any variable-length positional or keyword arguments.
### Supported Parameter Types
Muna supports a [fixed set](/predictions/create#using-prediction-values) of predictor
input and output value types. Below are supported type annotations:
Floating-point input and return values should be annotated with the `float` built-in type.
```py icon="python" theme={null}
from muna import compile
@compile(...)
def square(number: float) -> float:
return number ** 2
```
Unlike Python which defaults to 64-bit floats, Muna will always lower a Python `float` to 32 bits.
For control over the binary width of the number, use the `numpy.float[16,32,64]` types:
```py icon="python" theme={null}
from muna import compile
import numpy as np
@compile(...)
def square(number: np.float64) -> float64:
return number ** 2
```
Integer input and return values should be annotated with the `int` built-in type.
```py icon="python" theme={null}
from muna import compile
@compile(...)
def square(number: int) -> int:
return number ** 2
```
Unlike Python which supports arbitrary-precision integers, Muna will always lower a Python `int` to 32 bits.
For control over the binary width of the integer, use the `numpy.int[8,16,32,64]` types:
```py icon="python" theme={null}
from muna import compile
import numpy as np
@compile(...)
def square(number: np.int16) -> np.int16:
return number ** 2
```
Boolean input and return values must be annotated with the `bool` built-in type.
```py icon="python" theme={null}
from muna import compile
@compile(...)
def invert(on: bool) -> bool:
return not on
```
Tensor input and return values must be annotated with the NumPy `numpy.typing.NDArray[T]` type, where `T` is
the tensor element type.
```py icon="python" theme={null}
from muna import compile
import numpy as np
from numpy.typing import NDArray
@compile(...)
def cholesky_decompose(tensor: NDArray[np.float64]) -> np.ndarray:
return np.linalg.cholesky(tensor).astype("float32")
```
You can also annotate with the `np.ndarray` type, but doing so will always assume a `float32` element type (following
[PyTorch semantics](https://pytorch.org/docs/stable/generated/torch.get_default_dtype.html)).
Below are the supported element types:
| Numpy data type | Muna data type |
| :-------------- | :------------- |
| `np.float16` | `float16` |
| `np.float32` | `float32` |
| `np.float64` | `float64` |
| `np.int8` | `int8` |
| `np.int16` | `int16` |
| `np.int32` | `int32` |
| `np.int64` | `int64` |
| `np.uint8` | `uint8` |
| `np.uint16` | `uint16` |
| `np.uint32` | `uint32` |
| `np.uint64` | `uint64` |
| `bool` | `bool` |
Muna does not yet support complex numbers or tensors.
Muna only supports, and will always assume, little-endian ordering for multi-byte element types.
String input and return values must be annotated with the `str` built-in type.
```py icon="python" theme={null}
from muna import compile
@compile(...)
def uppercase(text: str) -> str:
return text.upper()
```
List input and return values must be annotated with the `list[T]` built-in type, where `T` is the element type.
```py icon="python" theme={null}
from muna import compile
@compile(...)
def slice(items: list[str]) -> list[str]:
return items[:3]
```
When the list element type `T` is a Pydantic `BaseModel`, a full JSON schema will be generated.
Providing an element type `T` is optional but strongly recommended because it is used to generate a schema for the parameter or
return value.
Dictionary input and return values can be annotated in one of two ways:
1. Using a Pydantic [`BaseModel`](https://docs.pydantic.dev/latest/concepts/models) subclass.
2. Using the `dict[str, T]` built-in type.
```py icon="python" theme={null}
from muna import compile
from pydantic import BaseModel
from typing import Literal
class Person(BaseModel):
city: str
age: int
class Pet(BaseModel):
sound: Literal["bark", "meow"]
legs: int
@compile(...)
def choose_favorite_pet(person: Person) -> Pet:
return Pet(sound="meow", legs=6)
```
We strongly recommend the Pydantic `BaseModel` annotation, as it allows us to generate a full JSON schema.
When using the `dict` annotation, they key type **must** be `str`. The value type `T` can be any arbitrary type.
Image input and return values must be annotated with the Pillow
[`PIL.Image.Image`](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image) type.
```py icon="python" theme={null}
from muna import compile
from PIL import Image
@compile(...)
def resize(image: Image.Image) -> Image.Image:
return image.resize((512, 512))
```
Binary input and return values can be annotated in one of three ways:
1. Using the `bytes` built-in type.
2. Using the `bytearray` built-in type.
3. Using the `io.BytesIO` type.
```py icon="python" theme={null}
from muna import compile
from PIL import Image
def resize_pixels(pixels: bytes) -> bytes
return Image.frombytes("L", (4,4), pixels).resize((8,8)).tobytes()
```
### Using Parameter Annotations
Muna supports attaching additional annotations to the function's parameter and return types:
```py icon="python" focus={6-13} theme={null}
from muna import compile, Parameter
from typing import Annotated
@compile(...)
def area(
radius: Annotated[
float,
Parameter.Generic(description="Radius of the circle.")
]
) -> Annotated[
float,
Parameter.Generic(description="Area of the circle.")
]:
...
```
These annotations serve multiple important purposes:
* They help users know what input data to provide to the predictor and how to use output data from the predictor, via the parameter `description`.
* They help users search for predictors using highly detailed queries (e.g. MCP clients).
* They help the Muna client automatically provide familiar interfaces around your prediction function, e.g. with the [OpenAI interface](/predictions/openai).
* They help the Muna website automatically create interactive [`visualizers`](https://github.com/muna-ai/visualizers) for
your prediction function.
While not required, we highly recommend using parameter annotations on your compiled functions.
Below are currently supported annotations:
Use the `Parameter.Generic` annotation to provide information about a general input or output parameters:
```py predictor.py icon="python" focus={6-9} theme={null}
from muna import compile, Parameter
from typing import Annotated
@compile(...)
def area(
radius: Annotated[
float,
Parameter.Generic(description="Radius of the circle.")
]
) -> float:
...
```
Below is the full `Parameter.Generic` annotation definition:
```py icon="python" theme={null}
@classmethod
def Generic(
cls,
*,
description: str # Parameter description.
) -> Parameter: ...
```
Use the `Parameter.Numeric` annotation to specify numeric input or output parameters:
```py calculate_area.py icon="python" focus={6-13} theme={null}
from muna import compile, Parameter
from typing import Annotated
@compile(...)
def area(
radius: Annotated[
float,
Parameter.Numeric(
description="Circle radius.",
min=1.,
max=12.
)
]
) -> float:
...
```
Below is the full `Parameter.Numeric` annotation definition:
```py icon="python" theme={null}
@classmethod
def Numeric(
cls,
*,
description: str, # Parameter description.
min: float | None=None, # Minimum value.
max: float | None=None # Maximum value.
) -> Parameter: ...
```
Use the `Parameter.Audio` annotation to specify audio parameters:
```py transcribe_audio.py icon="python" focus={7-13} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated
@compile(...)
def transcribe_audio(
audio: Annotated[
ndarray,
Parameter.Audio(
description="Input audio.",
sample_rate=24_000
)
]
) -> str:
...
```
The `Parameter.Audio` annotation allows the compiled predictor to be used by our
[OpenAI speech client](/predictions/openai#creating-speech).
Below is the full `Parameter.Audio` annotation definition:
```py icon="python" theme={null}
@classmethod
def Audio(
cls,
*,
description: str, # Parameter description.
sample_rate: int # Audio sample rate in Hertz.
) -> Parameter: ...
```
Use the `Parameter.AudioSpeed` annotation to specify audio speed parameters in audio generation predictors:
```py generate_speech.py icon="python" focus={8-15} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated
@compile(...)
def generate_speech(
text: str,
speed: Annotated[
float,
Parameter.AudioSpeed(
description="The speed of the generated audio.",
min=0.25,
max=4.0
)
] = 1.0
) -> ndarray:
...
```
Below is the full `Parameter.AudioSpeed` annotation definition:
```py icon="python" theme={null}
@classmethod
def AudioSpeed(
cls,
*,
description: str, # Parameter description.
min: float | None=None, # Minimum audio speed.
max: float | None=None # Maximum audio speed.
) -> Parameter: ...
```
Use the `Parameter.AudioVoice` annotation to specify audio voice parameters in audio generation predictors:
```py generate_speech.py icon="python" focus={10-13} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated, Literal
Voice = Literal["almas", "parv", "rhea", "sam"]
@compile(...)
def generate_speech(
text: str,
voice: Annotated[
Voice,
Parameter.AudioVoice(description="Voice to use when generating audio.")
],
speed: float=1.0
) -> ndarray:
...
```
Below is the full `Parameter.AudioVoice` annotation definition:
```py icon="python" theme={null}
@classmethod
def AudioVoice(
cls,
*,
description: str # Parameter description.
) -> Parameter: ...
```
Use the `Parameter.BoundingBox` or `Parameter.BoundingBoxes` annotations to specify
bounding box parameters in object detection predictors:
```py Single icon="rectangle-wide" focus={8-11} theme={null}
from muna import compile, Parameter
from PIL import Image
from typing import Annotated, Literal
@compile(...)
def detect_object(
image: Image.Image
) -> Annotated[
Detection,
Parameter.BoundingBox(description="Detected object.")
]:
...
```
```py Multiple icon="rectangles-mixed" focus={8-11} theme={null}
from muna import compile, Parameter
from PIL import Image
from typing import Annotated, Literal
@compile(...)
def detect_objects(
image: Image.Image
) -> Annotated[
list[Detection],
Parameter.BoundingBoxes(description="Detected objects.")
]:
...
```
Below is the full `Parameter.BoundingBox` annotation definition:
```py Single icon="rectangle-wide" theme={null}
@classmethod
def BoundingBox(
cls,
*,
description: str # Parameter description.
) -> Parameter: ...
```
```py Multiple icon="rectangles-mixed" theme={null}
@classmethod
def BoundingBoxes(
cls,
*,
description: str # Parameter description.
) -> Parameter: ...
```
Use the `Parameter.DepthMap` annotation to specify depth map parameters in depth estimation predictors:
```py estimate_depth.py icon="python" focus={9-12} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from PIL import Image
from typing import Annotated
@compile(...)
def estimate_depth(
image: Image.Image
) -> Annotated[
ndarray,
Parameter.DepthMap(description="Metric depth tensor.")
]:
...
```
Below is the full `Parameter.DepthMap` annotation definition:
```py icon="python" theme={null}
@classmethod
def DepthMap(
cls,
*,
description: str # Parameter description.
) -> Parameter: ...
```
Use the `Parameter.Embedding` annotation to specify vector embedding parameters in embedding predictors:
```py embed_text.py icon="python" focus={8-11} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated
@compile(...)
def embed_text(
text: str
) -> Annotated[
ndarray,
Parameter.Embedding(description="Embedding vector.")
]:
...
```
The `Parameter.Embedding` annotation allows the compiled predictor to be used by our\
[OpenAI embedding client](/predictions/openai#creating-embeddings).
Below is the full `Parameter.Embedding` annotation definition:
```py icon="python" theme={null}
@classmethod
def Embedding(
cls,
*,
description: str # Parameter description.
) -> Parameter: ...
```
Use the `Parameter.EmbeddingDims` annotation to specify an embedding
Matryoshka dimension parameter in embedding predictors:
```py embed_text.py icon="python" focus={8-11} theme={null}
from muna import compile, Parameter
from numpy import ndarray
from typing import Annotated
@compile(...)
def embed_text(
text: str,
dims: Annotated[
int,
Parameter.EmbeddingDims(description="Embedding dimensions.")
]
) -> ndarray:
...
```
Below is the full `Parameter.EmbeddingDims` annotation definition:
```py icon="python" theme={null}
@classmethod
def EmbeddingDims(
cls,
*,
description: str, # Parameter description.
min: int | None=None, # Minimum embedding dimensions.
max: int | None=None # Maximum embedding dimensions.
) -> Parameter: ...
```
## Writing the Function Body
The function body can contain arbitrary Python code. Given that the Muna compiler is currently a
proof of concept, it has limited coverage for Python language features. Below is a list of Python
language features that we either partially support, or do not support at all:
| Statement | Status | Notes |
| :------------------ | :----: | :----------------------------------------------------------------------------------------------------------------------------------------------- |
| Recursive functions | 🔨 | Recursive functions **must** have a return type annotation. |
| Lambda expressions | 🚧 | Lambda expressions [can be invoked](https://github.com/muna-ai/compiler/blob/main/predictors/language/lambda.py), but cannot be used as objects. |
| Collection | Status | Notes |
| :------------------ | :----: | :------------------------------------------------------------- |
| List literals | 🚧 | List must contain primitive members (e.g. `int`, `str`). |
| Dictionary literals | 🚧 | Dictionary must contain primitive members (e.g. `int`, `str`). |
| Set literals | 🚧 | Set must contain primitive members (e.g. `int`, `str`). |
| Tuple literals | 🚧 | Tuple must contain primitive members (e.g. `int`, `str`). |
Tracing through classes is not yet supported.
| Statement | Status | Notes |
| :---------------------- | :----: | :---- |
| `raise` statements | 🔨 | |
| `try..except` statement | 🔨 | |
Over time the list of unsupported language features will shrink and eventually, will be empty.
## Using Compiler Sandboxes
Muna supports defining custom sandboxes that can be used to reconstruct your Python environment before compiling your function.
Sandboxes are very much experimental, and will likely see major changes, additions, and revisions in the near future.
Use the `Sandbox.pip_install` method to install Python packages from the PyPi registry:
```py predictor.py icon="python" theme={null}
from muna import compile, Sandbox
# Install numpy and sklearn
sandbox = (Sandbox()
.pip_install("numpy", "scikit-learn")
)
# Compile your function with the sandbox
@compile(..., sandbox=sandbox)
def predict() -> np.ndarray:
...
```
We highly recommend pinning the specific versions of Python packages in use, so as to
prevent incompatibilities when creating the sandbox.
Use the `Sandbox.apt_install` method to install Debian system packages:
```py predictor.py icon="python" theme={null}
from muna import compile, Sandbox
# Install git and wget
sandbox = (Sandbox()
.apt_install("git", "wget")
)
# Compile your function with the sandbox
@compile(..., sandbox=sandbox)
def predict() -> BytesIO:
...
```
Use the `Sandbox.env` method to define plaintext environment variables:
```py predictor.py icon="python" theme={null}
from muna import compile, Sandbox
# Define an environment variable
sandbox = (Sandbox()
.env({ "MUNA_WEBSITE": "https://muna.ai" })
)
# Compile your function with the sandbox
@compile(..., sandbox=sandbox)
def predict(prompt: str) -> str:
...
```
Muna does not yet support defining secrets. **Do not** provide secrets
using sandbox environment variables as they are not designed for storing secrets.
Use the `Sandbox.upload_file` method to upload a file to a path in the sandbox:
```py predictor.py icon="python" theme={null}
from muna import compile, Sandbox
# Upload a model weight to the sandbox
sandbox = (Sandbox()
.upload_file("DeepSeek-R1.gguf", "/Deepseek-R1.gguf")
)
# Compile your function with the sandbox
@compile(..., sandbox=sandbox)
def predict(prompt: str) -> str:
...
```
Use the `Sandbox.upload_directory` method to upload a directory and all its contents to a path in the sandbox:
```py predictor.py icon="python" theme={null}
from muna import compile, Sandbox
# Upload a directory to the sandbox
sandbox = (Sandbox()
.upload_file("resources/", "/resources")
)
# Compile your function with the sandbox
@compile(..., sandbox=sandbox)
def predict(prompt: str) -> str:
...
```
## Using Compiler Metadata
Muna's compiler supports specifying metadata, allowing you to configure the compiler or provide additional information.
Use the `TensorRTInferenceMetadata` metadata type to compile a PyTorch [`nn.Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html) to [TensorRT](https://developer.nvidia.com/tensorrt):
```py ai.py icon="python" focus={1,5-8,12-20} theme={null}
from muna.beta import TensorRTInferenceMetadata
from torch import randn, Tensor
from torch.nn import Module
# Given a PyTorch model...
model: Module = ...
# With some example arguments...
example_args: list[Tensor] = [randn(1, 3, 224, 224)]
@compile(
...,
metadata=[
# Use TensorRT for model inference
TensorRTInferenceMetadata(
model=model,
model_args=example_args,
cuda_arch="sm_100",
precision="int4"
)
]
)
def predict() -> None:
pass
```
The TensorRT inference backend is only available on Linux and Windows devices with compatible Nvidia GPUs.
We are working on adding support for consumer RTX GPUs with [TensorRT for RTX](https://developer.nvidia.com/blog/nvidia-tensorrt-for-rtx-introduces-an-optimized-inference-ai-library-on-windows/).
#### Target CUDA Architectures
TensorRT engines must be compiled for specific target CUDA architectures. Below are CUDA architectures that our compiler supports:
| CUDA Architecture | GPU Family |
| :---------------- | :----------------------- |
| `sm_80` | Ampere (e.g. A100) |
| `sm_86` | Ampere |
| `sm_87` | Ampere |
| `sm_89` | Ada Lovelace (e.g. L40S) |
| `sm_90` | Hopper (e.g. H100) |
| `sm_100` | Blackwell (e.g. B200) |
#### TensorRT Inference Precision
TensorRT allows for specifying the inference engine's precision. Below are supported precision modes:
| Precision | Notes |
| :-------- | :--------------------------------- |
| `fp32` | 32-bit single precision inference. |
| `fp16` | 16-bit half precision inference. |
| `int8` | 8-bit quantized integer inference. |
Use the `OnnxRuntimeInferenceMetadata` metadata type to compile a PyTorch [`nn.Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html) for inference with [ONNXRuntime](https://onnxruntime.ai/):
```py ai.py icon="python" focus={1,5-8,12-18} theme={null}
from muna.beta import OnnxRuntimeInferenceMetadata
from torch import randn, Tensor
from torch.nn import Module
# Given a PyTorch model...
model: Module = ...
# With some example arguments...
example_args: list[Tensor] = [randn(1, 3, 224, 224)]
@compile(
...,
metadata=[
# Use ONNXRuntime for model inference
OnnxRuntimeInferenceMetadata(
model=model,
model_args=example_args
)
]
)
def predict() -> None:
pass
```
Use the `OnnxRuntimeInferenceSessionMetadata` metadata type to compile an OnnxRuntime [`InferenceSession`](https://onnxruntime.ai/docs/api/python/api_summary.html#inferencesession):
```py ai.py icon="python" focus={1,4-6,10-16} theme={null}
from muna.beta import OnnxRuntimeInferenceSessionMetadata
from onnxruntime import InferenceSession
# Given an ONNXRuntime inference session...
model_path = "/path/to/model.onnx"
session = InferenceSession(model_path)
@compile(
...,
metadata=[
# Use ONNXRuntime for model inference
OnnxRuntimeInferenceSessionMetadata(
session=session,
model_path=model_path
)
]
)
def predict(...) -> None:
pass
```
The ONNX model file must exist at the provided `model_path` **within the compiler sandbox**.
Use the `CoreMLInferenceMetadata` metadata type to compile a PyTorch
[`nn.Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html) to
[CoreML](https://developer.apple.com/documentation/coreml):
```py ai.py icon="python" focus={1,5-8,12-18} theme={null}
from muna.beta import CoreMLInferenceMetadata
from torch import randn, Tensor
from torch.nn import Module
# Given a PyTorch model...
model: Module = ...
# With some example arguments...
example_args: list[Tensor] = [randn(1, 3, 224, 224)]
@compile(
...,
metadata=[
# Use CoreML for model inference
CoreMLInferenceMetadata(
model=model,
model_args=example_args
)
]
)
def predict() -> None:
pass
```
The CoreML inference backend is only available on iOS, macOS, and visionOS devices.
Use the `LlamaCppInferenceMetadata` metadata type to compile a [`Llama`](https://github.com/abetlen/llama-cpp-python)
instance:
```py llm.py icon="python" focus={9-15} theme={null}
from muna.beta import LlamaCppInferenceMetadata
from llama_cpp import Llama
# Given an LLM
llm = Llama(...)
@compile(
...,
metadata=[
# Specify Llama.cpp inference metadata
LlamaCppInferenceMetadata(
model=llm,
backends=["cuda"]
)
]
)
def predict() -> None:
pass
```
## Llama.cpp Hardware Backends
Llama.cpp supports several hardware backends to accelerate model inference.
Below are targets that are currently supported by Muna:
| Backend | Notes |
| :------ | :------------------------------------------------------------------------------------------------------- |
| `cuda` | [Nvidia CUDA backend](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#cuda). Linux only. |
Use the `ExecuTorchInferenceMetadata` metadata type to compile a PyTorch [`nn.Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html) for inference with [ExecuTorch](https://docs.pytorch.org/executorch/stable/index.html):
```py ai.py icon="python" focus={1,5-8,12-19} theme={null}
from muna.beta import ExecuTorchInferenceMetadata
from torch import randn, Tensor
from torch.nn import Module
# Given a PyTorch model...
model: Module = ...
# With some example arguments...
example_args: list[Tensor] = [randn(1, 3, 224, 224)]
@compile(
...,
metadata=[
# Use ExecuTorch for model inference
ExecuTorchInferenceMetadata(
model=model,
model_args=example_args,
backend="xnnpack"
)
]
)
def predict() -> None:
pass
```
The ExecuTorch inference backend is only available on Android.
#### ExecuTorch Hardware Backends
ExecuTorch supports several [hardware backends](https://docs.pytorch.org/executorch/stable/backends-overview.html) to
accelerate model inference. Below are targets that are currently supported by Muna:
| Backend | Notes |
| :-------- | :---------------------------------------------------------------------------------------------------------------- |
| `xnnpack` | [XNNPACK CPU backend](https://docs.pytorch.org/executorch/stable/backends-xnnpack.html). Always enabled. |
| `vulkan` | [Vulkan GPU backend](https://docs.pytorch.org/executorch/stable/backends-vulkan.html). Only supported on Android. |
Use the `LiteRTInferenceMetadata` metadata type to compile a PyTorch [`nn.Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html) for inference with [LiteRT](https://ai.google.dev/edge/litert):
```py ai.py icon="python" focus={1,5-8,12-18} theme={null}
from muna.beta import LiteRTInferenceMetadata
from torch import randn, Tensor
from torch.nn import Module
# Given a PyTorch model...
model: Module = ...
# With some example arguments...
example_args: list[Tensor] = [randn(1, 3, 224, 224)]
@compile(
...,
metadata=[
# Use LiteRT for model inference
LiteRTInferenceMetadata(
model=model,
model_args=example_args
)
]
)
def predict() -> None:
pass
```
Use the `TFLiteInterpreterMetadata` metadata type to compile a TensorFlow Lite
[`Interpreter`](https://ai.google.dev/edge/api/tflite/python/tf/lite/Interpreter):
```py ai.py icon="python" focus={1,4-6,10-16} theme={null}
from muna.beta import TFLiteInterpreterMetadata
from tensorflow import lite
# Given a TFLite interpreter...
model_path = "/path/to/model.tflite"
interpreter = lite.Interpreter(model_path)
@compile(
...,
metadata=[
# Use TensorFlow Lite for model inference
TFLiteInterpreterMetadata(
interpreter=interpreter,
model_path=model_path
)
]
)
def predict(...) -> None:
pass
```
The TensorFlow Lite model file must exist at the provided `model_path` **within the compiler sandbox**.
Use the `QnnInferenceMetadata` metadata type to compile a PyTorch [`nn.Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html) to a [Qualcomm QNN](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/introduction.html?product=1601111740009302) context binary:
```py ai.py icon="python" focus={1,5-8,12-20} theme={null}
from muna.beta import QnnInferenceMetadata
from torch import randn, Tensor
from torch.nn import Module
# Given a PyTorch model...
model: Module = ...
# With some example arguments...
example_args: list[Tensor] = [randn(1, 3, 224, 224)]
@compile(
...,
metadata=[
# Use QNN for model inference
QnnInferenceMetadata(
model=model,
model_args=example_args,
backend="gpu",
quantization=None
)
]
)
def predict() -> None:
pass
```
The QNN inference backend is only available on Android and Windows devices with Qualcomm processors.
#### QNN Hardware Backends
QNN requires that a hardware device `backend` is specified ahead of time. Below are supported backends:
| Backend | Notes |
| :------ | :----------------------------------------- |
| `cpu` | Reference `aarch64` CPU backend. |
| `gpu` | Adreno GPU backend, accelerated by OpenCL. |
| `htp` | Hexagon NPU backend. |
Learn more about [QNN hardware backends](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/backend.html?product=1601111740009302).
#### QNN Model Quantization
When using the `htp` backend, you **must** specify a model `quantization` mode as the Hexagon NPU only supports
running integer-quantized models. Below are supported quantization modes:
| Quantization | Notes |
| :----------- | :---------------------------------------------------------------------------- |
| `w8a8` | Weights and activations are quantized to `uint8`. |
| `w8a16` | Weights are quantized to `uint8` while activations are quantized to `uint16`. |
| `w4a8` | Weights are quantized to `uint4` while activations are quantized to `uint8`. |
| `w4a16` | Weights are quantized to `uint4` while activations are quantized to `uint16`. |
Use the `OpenVINOInferenceMetadata` metadata type to compile a PyTorch [`nn.Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html) to [OpenVINO](https://docs.openvino.ai/2025/index.html) IR:
```py ai.py icon="python" focus={1,5-8,12-18} theme={null}
from muna.beta import OpenVINOInferenceMetadata
from torch import randn, Tensor
from torch.nn import Module
# Given a PyTorch model...
model: Module = ...
# With some example arguments...
example_args: list[Tensor] = [randn(1, 3, 224, 224)]
@compile(
...,
metadata=[
# Use OpenVINO for model inference
OpenVINOInferenceMetadata(
model=model,
model_args=example_args
)
]
)
def predict() -> None:
pass
```
At runtime, the OpenVINO IR will be used for inference with the [OpenVINO toolkit](https://github.com/openvinotoolkit/openvino).
The OpenVINO inference backend is only available on Linux and Windows `x86_64` devices with Intel processors.
Use the `muna.beta.IREEInferenceMetadata` metadata type to compile a PyTorch [`nn.Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html) for inference with [IREE](https://iree.dev/):
```py ai.py icon="python" focus={1,5-8,12-19} theme={null}
from muna.beta import IREEInferenceMetadata
from torch import randn, Tensor
from torch.nn import Module
# Given a PyTorch model...
model: Module = ...
# With some example arguments...
example_args: list[Tensor] = [randn(1, 3, 224, 224)]
@compile(
...,
metadata=[
# Use IREE for model inference
IREEInferenceMetadata(
model=model,
model_args=example_args,
backend="vulkan"
)
]
)
def predict() -> None:
pass
```
The IREE inference backend is only available on Android devices.
### IREE HAL Target Backends
IREE supports several HAL target backends that
the `model` can be compiled against. Below are targets that are currently supported by Muna:
| Target | Notes |
| :------- | :-------------------------------------------------------------------------------------------------------------- |
| `vulkan` | [Vulkan GPU backend](https://iree.dev/guides/deployment-configurations/gpu-vulkan/). Only supported on Android. |
*Coming soon* 🤫.
## Library Coverage
We are adding support for popular libraries, across tensor frameworks, scientific computing, and more:
Below are libraries currently supported by our compiler:
If you need a specific library to be supported by the Muna compiler, [reach out to us](mailto:hi@muna.ai).
# Introduction
Source: https://docs.muna.ai/ref/introduction
Using the Muna REST API.
First, you should probably not be using the Muna REST API direclty unless you are building a client SDK for your language,
or you have a very advanced use-case not supported by one of our provided client SDKs.
## Authentication
Most API endpoints are authenticated using Bearer tokens authentication. [Generate](https://muna.ai/settings/developer) and provide an access key
to use in requests:
# Create a Prediction
Source: https://docs.muna.ai/ref/predictions/create
POST /v1/predictions
### Body
Predictor tag.
Prediction client identifier.
Prediction configuration identifier.
Device identifier, used for choosing optimal implementation to respond with.
For making predictions with embedded predictors, providing the original prediction identifier ensures that
the same prediction implementation is provided to the device.
### Response
Prediction identifier.
Predictor tag.
Prediction configuration token.
Prediction resources.
Prediction resource type.
Prediction resource URL.
Prediction resource name.
Prediction creation date.
# Retrieve a Predictor
Source: https://docs.muna.ai/ref/predictors/retrieve
GET /v1/predictors/{tag}
### Parameters
Predictor tag.
### Response
Predictor tag.
Predictor owner.
Username.
User creation date.
Predictor name.
Predictor description.
Predictor card.
Predictor status.
Predictor access.
Predictor signature.
Predictor inputs.
Parameter name.
Parameter data type.
Parameter description.
Whether the parameter is optional.
Parameter range for numeric parameters.
Parameter value choices for enumeration parameters.
Enumeration member name.
Enumeration member value.
Parameter default value.
Parameter JSON schema.
This is only populated for `list` and `dict` parameters.
Predictor outputs.
Parameter name.
Parameter data type.
Parameter value choices for enumeration parameters.
Enumeration member name.
Enumeration member value.
Parameter JSON schema.
This is only populated for `list` and `dict` parameters.
# Security Model
Source: https://docs.muna.ai/security
How Muna guarantees code safety and security.
Muna works by sending compiled binaries to end users' devices. As such, Muna is
carefully designed to minimize any attack surface that exists in downloading and executing software binaries.
## Minimum Requirements
Muna compiles Python functions against recent versions of the operating systems on which it will run.
Below are the minimum requirements across each platform:
Android API level 24+ (Android Nougat or newer) across the following ABIs:
* `armeabi-v7a`
* `arm64-v8a`
iOS 14+.
Linux distributions with GLIBC 2.35+ across the following architectures:
* `aarch64`
* `x86_64` with `AVX2` or newer
macOS 14+ with Apple Silicon.
visionOS 1.3+.
Browsers with [WebAssembly + fixed-width SIMD](https://webassembly.org/features/):
* Chrome 91+
* Firefox 90+
* Safari 16.4+
Windows 10+ across the following architectures:
* `amd64` with `AVX2` or newer
* `arm64`
## Code Provenance
Muna works by [lowering Python code](/insiders/compiler) to native code that is then compiled. This
process involves reimplementing Python operations natively. These native implementations are written and
maintained by us, and are rigorously tested to ensure correctness and memory safety.
This means that regardless of what the original Python code does, the resulting compiled binary will only
ever contain code, written, reviewed, and tested by us.
Python code that uses the following sensitive or dangerous APIs will fail to compile:
* File system access.
* Hardware access (e.g. camera, microphone).
This list is not exhaustive.
## Code Signing
When we compile native binaries, we perform code signing for platforms that support it:
Android does not support code-signing on individual native binaries. Instead, apps which contain these
binaries are code-signed for Play Store distribution.
Binaries are code signed.
Code signing is not supported by Linux.
Binaries are code signed. At runtime, code-signing is verified with [`SecStaticCodeCheckValidity`](https://developer.apple.com/documentation/security/secstaticcodecheckvalidity\(_:_:_:\))
before the predictor is loaded.
Current Muna SDKs do not perform signature verification at runtime.
We will add signature verification in upcoming updates.
Binaries are code signed.
Code signing is not supported by WebAssembly.
While not yet supported, binaries will be code signed. At runtime, code-signing is verified with [`WinVerifyTrust`](https://learn.microsoft.com/en-us/windows/win32/api/wintrust/nf-wintrust-winverifytrust).
We are working on code-signing all Windows binaries.
## Code Sandboxing
On Android, iOS, macOS (App Store), and visionOS, there are strict sandboxing restrictions that prohibit
downloading and executing code at runtime. As a result, Muna client SDKs for Android, Swift,
React Native, and Unity Engine allow you to **embed predictors** into the app bundle at build time:
Use the [`ai.muna.muna-gradle`](https://central.sonatype.com/artifact/ai.muna/muna-gradle) Gradle
plugin in your `build.gradle` or `build.gradle.kts` file like so:
```kt build.gradle.kts icon="android" focus={1,6,13-17} theme={null}
import ai.muna.muna.gradle.MunaEmbed
plugins {
id("com.android.application")
id("org.jetbrains.kotlin.android")
id("ai.muna.muna-gradle") version "0.0.5"
}
android {
...
}
muna {
embeds.addAll(
MunaEmbed(tag = "@supertone/supertonic-2")
)
}
```
Make sure to add a `MunaEmbed` entry for every predictor your app uses.
*Coming soon*.
Embedding works in two stages. First, you must specify the predictor tags that you
want embedded within your app by creating a `muna.predictors` list in your `pubspec.yaml` file:
```yaml pubspec.yaml icon="flutter" theme={null}
# Embed `@supertone/supertonic-2`
muna:
predictors:
- tag: "@supertone/supertonic-2"
```
Next, run the `muna:embed` tool to download and embed the predictors you listed earlier:
```sh icon="terminal" theme={null}
# Run this in Terminal
$ dart run muna:embed --access-key
```
You can also define your `MUNA_ACCESS_KEY` in a `.env` file or as an environment variable.
The `muna:embed` tool will automatically detect and use it.
*Coming soon*.
Embed predictors by adding the `Muna.EmbedAttribute` attribute to any `class` or `struct` in your project code:
```csharp AppBehaviour.cs icon="unity" focus={4} theme={null}
using UnityEngine;
using Muna;
[Muna.Embed("@fxn/greeting")]
public class AppBehaviour : MonoBehaviour {
...
}
```
When building your Unity app, the Muna SDK will fetch and embed predictors using the Muna access key in your
project settings.
The `Muna.EmbedAttribute` attribute can accept multiple predictor tags.
If you use a custom proxy URL with custom authentication, you can instead apply
the attribute to a static property that returns your authenticated `Muna` client:
```csharp AppBehaviour.cs theme={null}
using UnityEngine;
using Muna;
public class AppBehaviour : MonoBehaviour {
[Muna.Embed("@apple/openelm")]
private static Muna muna => new Muna(
url: "https://apple.com/api",
accessKey: "tim apple"
);
}
```
With predictor embedding, all prediction code will be present for code review and signing when the application
is archived for distribution (e.g. on the App Store or Play Store).
## Data Collection
At runtime, end user devices will make web requests to the Muna API to retrieve a predictor; and to report
telemetry data. Below is the data that the Muna SDK transmits from user devices to the Muna API:
Because Muna is designed to hyper-target hardware, the Muna client SDKs report metadata including:
* Operating system (e.g. `ios`).
* Processor architecture (e.g. `arm64`).
* CPU instruction sets (e.g. `avx512-vnni`).
* GPU compute capability (e.g. Nvidia `sm_90`).
The Muna SDK **never reports** any user-identifying information.
This is a unique, random string identifying the Muna client SDK on the current device.
The Muna client SDKs report performance statistics for predictions run on the current device. This is used to
search for optimal predictor implementations for a given device.