Inkling-Small: The Definitive Guide

Frontier - HowiPrompt

---

What it is & why it matters

Inkling-Small is the latest Mixture-of-Experts (MoE) language model released by Thinking Machines. Built on the same research foundation as the larger Inkling family, Inkling-Small is positioned as a "small" model only in name - the community is already testing it as a 276-billion-parameter system that can rival much larger proprietary offerings.

Why does this matter?

In short, Inkling-Small is the first open-source MoE model that ships with a production-ready stack for reasoning, tool use, and multimodal AI--a combination that has previously been limited to closed-source offerings from the big cloud providers.

---

What's new / key features (detailed breakdown)

The Inkling-Small release bundles a set of capabilities that were previously scattered across separate projects. Below is a feature-by-feature look, grounded in the official SGLang documentation page "Deployment Playground -> Advanced Usage".

> Note: The exact token limits, memory footprints, and performance numbers can vary by hardware and configuration. Always verify the latest values in the official Inkling-Small docs or the llms.txt index file.

---

Installation -- every OS

Inkling-Small runs on top of SGLang, so the first step is to install the SGLang runtime. Below are the supported methods for Windows, macOS, and Linux. The commands are taken directly from the official "Deploy Inkling-Small with SGLang" page; where a step is ambiguous, we recommend checking the latest SGLang installation guide.

Windows

> Prerequisites > - Python 3.10+ (official installer from python.org) > - Docker Desktop (for GPU acceleration, ensure WSL 2 backend is enabled) > - CUDA Toolkit matching your GPU driver (CUDA 12 is the most widely supported right now).

1. Install Python packages

   # Upgrade pip
   python -m pip install --upgrade pip

   # Install SGLang from source (the Inkling-Small branch isn't in a PyPI release yet)
   pip install "git+https://github.com/sgl-project/sglang.git#subdirectory=python"

1. Pull the appropriate Docker image

Choose the CUDA version that matches your driver. For CUDA 12:

   docker pull lmsysorg/sglang:dev-cu12-inkling-dspark

For CUDA 13 (if you have the newer toolkit):

   docker pull lmsysorg/sglang:dev-inkling-dspark

1. Run the container

The command generator on the official page produces a sglang serve ... line. A minimal example:

   docker run --gpus all -p 3000:3000 \
       lmsysorg/sglang:dev-cu12-inkling-dspark \
       sglang serve --model inkling-small --port 3000

Adjust --port or add --max-context 65536 for long-context use.

macOS

> Prerequisites > - Homebrew (for dependencies) > - Python 3.10+ (brew install python) > - Docker Desktop (Apple Silicon builds are supported via the arm64 image).

1. Install SGLang from source

   # Ensure pip is up-to-date
   python3 -m pip install --upgrade pip

   # Install the latest SGLang code
   pip install "git+https://github.com/sgl-project/sglang.git#subdirectory=python"

1. Pull the arm64 CUDA-compatible image (the Docker image uses NVIDIA's CUDA on macOS via the external GPU bridge, but for Apple Silicon you'll typically run the CPU-only fallback; the image still works for API compatibility).

   docker pull lmsysorg/sglang:dev-inkling-small-dgx-spark   # arm64 tag

1. Start the server

   docker run --platform linux/arm64 -p 3000:3000 \
       lmsysorg/sglang:dev-inkling-small-dgx-spark \
       sglang serve --model inkling-small --port 3000

If you lack an NVIDIA GPU, you can run the pure-Python server instead (see "First run / quick start" below).

Linux

> Prerequisites > - Python 3.10+ (sudo apt-get install python3 python3-pip or your distro's equivalent) > - Docker Engine (>= 20.10) with NVIDIA Container Toolkit if you plan to use GPU acceleration. > - CUDA Toolkit matching your driver (CUDA 12 or CUDA 13).

Option A - Install via pip (source)

# Upgrade pip
python3 -m pip install --upgrade pip

# Install SGLang from the GitHub source
pip install "git+https://github.com/sgl-project/sglang.git#subdirectory=python"

Option B - Docker (recommended for production)

1. Select the right image

• CUDA 13 (newest): lmsysorg/sglang:dev-inkling-dspark
• CUDA 12: lmsysorg/sglang:dev-cu12-inkling-dspark
• ROCm (AMD GPUs): lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark

1. Pull the image (example for CUDA 12):

   docker pull lmsysorg/sglang:dev-cu12-inkling-dspark

1. Run the container

   docker run --gpus all -p 3000:3000 \
       lmsysorg/sglang:dev-cu12-inkling-dspark \
       sglang serve --model inkling-small --port 3000 \
       --max-context 65536 --enable-hicache

Flags explained:

• --max-context enables the MXFP8 KV long-context mode.
• --enable-hicache turns on hierarchical KV caching.
• Add --enable-dspark to activate speculative decoding (usually on by default for the -dspark tags).

Verifying the installation

After the server starts, test the health endpoint:

curl http://localhost:3000/v1/health
# Expected JSON: {"status":"ok"}

If you see ok, Inkling-Small is ready to accept API calls.

---

First run / quick start (a few clicks)

The quickest way to get a feel for Inkling-Small is to use the Python client that ships with SGLang. The steps below assume you have completed the pip-install path (Windows/macOS/Linux) and are running on a machine with a decent GPU.

1. Create a virtual environment (optional but recommended)

   python3 -m venv inkling-env
   source inkling-env/bin/activate   # Windows: .\inkling-env\Scripts\activate

1. Launch the server in the background

   sglang serve --model inkling-small --port 3000 &

The & runs the process in the background; on Windows use start /B.

1. Run a one-off request

   import json, requests

   url = "http://localhost:3000/v1/completions"
   payload = {
       "model": "inkling-small",
       "messages": [{"role": "user", "content": "Explain the significance of the Model Context Protocol (MCP) in one paragraph."}],
       "max_tokens": 150,
       "temperature": 0.7
   }

   response = requests.post(url, json=payload)
   print(json.dumps(response.json(), indent=2, ensure_ascii=False))

You should see a concise explanation generated by the model. From here you can experiment with the advanced flags (tool_calls, multimodal, lora_id, etc.) that are documented under "Deployment Playground -> Advanced Usage".

---

Examples (several varied, concrete, with snippets)

Below are three practical scenarios that showcase Inkling-Small's flagship capabilities. All snippets assume the server is reachable at http://localhost:3000.

1. Reasoning + Tool-Calling (MCP) - Real-time weather lookup

import requests, json

url = "http://localhost:3000/v1/chat/completions"
payload = {
    "model": "inkling-small",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant with access to the 'weather' tool via MCP."},
        {"role": "user", "content": "What will the temperature be in Tokyo tomorrow at 9 am?"}
    ],
    "max_tokens": 200,
    "temperature": 0.0,
    "tool_calls": [
        {
            "name": "weather",
            "arguments": {"city": "Tokyo", "date": "tomorrow", "time": "09:00"}
        }
    ]
}
resp = requests.post(url, json=payload)
print(json.dumps(resp.json(), indent=2, ensure_ascii=False))

What happens:

1. Inkling-Small decides it needs external data, emits a tool_calls request.
2. Your MCP-enabled backend fetches the forecast from an API (e.g., OpenWeather).
3. The model receives the result and returns a natural-language answer: "Tomorrow at 9 am, Tokyo is expected to be 22 °C with light rain..."

2. Multimodal Input - Image-based question answering

import base64, requests, json
from pathlib import Path

# Load an image and encode as base64
img_path = Path("sphinx.jpg")
img_b64 = base64.b64encode(img_path.read_bytes()).decode()

url = "http://localhost:3000/v1/chat/completions"
payload = {
    "model": "inkling-small",
    "messages": [
        {"role": "user", "content": "What is the animal in this picture?"},
        {"role": "assistant", "content": None, "image": img_b64}
    ],
    "max_tokens": 50,
    "temperature": 0.2,
    "multimodal": True
}
resp = requests.post(url, json=payload)
print(resp.json()["choices"][0]["message"]["content"])

Result: "The animal is a common Egyptian sphinx cat, recognizable by its short hair and large ears."

> Tip: The multimodal flag is required; without it the server will reject the request. Verify the image size is ≤ 2 MiB for the default endpoint.

3. LoRA Adapter - Domain-specific legal drafting

Assume you have a LoRA adapter trained on contract clauses, stored at /models/lora/legal_v1.

# Launch the server with LoRA enabled
sglang serve --model inkling-small --port 3000 \
    --lora-path /models/lora/legal_v1 \
    --max-context 32768

Now a request:

payload = {
    "model": "inkling-small",
    "messages": [
        {"role": "user", "content": "Draft a non-disclosure agreement for a software startup."}
    ],
    "max_tokens": 512,
    "temperature": 0.3,
    "lora_id": "legal_v1"
}
resp = requests.post("http://localhost:3000/v1/completions", json=payload)
print(resp.json()["choices"][0]["text"])

The output reflects the legal phrasing learned by the LoRA, saving you from manually editing generic boilerplate.

---

Benefits & best use-cases

In practice, teams that need both high reasoning capability and the ability to call external services (e.g., a "research-assistant" that fetches PDFs, parses them, and answers questions) will find Inkling-Small uniquely positioned.

---

Alternatives & how it compares

Key takeaways

• Scale vs. sparsity - Inkling-Small's MoE architecture gives it a higher parameter count without the same memory footprint as a dense 276 B model.
• Tool-calling integration - While OpenAI's function calling is powerful, Inkling-Small's MCP is open-standard, letting you plug any backend without vendor lock-in.
• Multimodal parity - Only Inkling-Small (in the open-source space) ships with native image + audio handling; other models need separate vision encoders.

If you need purely text-only generation on a single GPU, a dense 7 B model may be more lightweight. But for agentic or multimodal workloads, Inkling-Small is currently the most feature-complete open option.

---

Tips, performance & troubleshooting (FAQ)

Performance tricks

1. Batch requests - SGLang can process a batch of up to 8 prompts per GPU step; use the batch field in the request body.
2. FP8 quantization - Not yet exposed in the public release, but the dev-rocm image includes experimental FP8 kernels. Keep an eye on the tag list.
3. CPU fallback for tool calls - Offload heavy API calls to a separate thread pool to avoid blocking the GPU inference loop.

If you encounter an error that isn't covered here, the official SGLang GitHub Issues page and the Inkling-Small Discord channel are the fastest places to get community help.

---

What the community says

The buzz around Inkling-Small is palpable across YouTube, podcasts, and developer forums. The main themes are:

Overall, the community consensus is that Inkling-Small is the most feature-rich open-source LLM released to date, but it requires careful hardware matching to unleash its full potential.

---

Verdict (honest pros/cons, who it's for)

Pros

• Full-stack open ecosystem - SGLang runtime, Docker images, and source install are all publicly available.
• MCP native tool calling - No extra wrappers; ideal for autonomous agents.
• Multimodal (image + audio) input - Ready for vision-language tasks out of the box.
• Scalable MoE architecture - High parameter count with manageable GPU memory.
• Advanced performance features - HiCache, MXFP8 KV, DSpark give real-world latency gains.

Cons

• Steep hardware requirements for optimal speed (CUDA 13, high-end GPUs).
• Documentation still maturing - Some flags and limits are only hinted at in the index file; users may need to experiment or consult the community.
• Docker images are large (several GB) and may take time to pull on slower connections.
• Limited CPU performance - While a CPU fallback exists, it is not suitable for production workloads.

Who should adopt Inkling-Small?

• AI startups building agentic products that need tool calling and multimodal reasoning without paying for proprietary APIs.
• Research labs interested in exploring MoE routing, hierarchical caching, or speculative decoding on open models.
• **Enter