How to Run Inkling AI

Inkling's full weights need 1.9TB of disk. Quantized versions start at 270GB. Here is exactly what hardware you need — and the cloud options when you don't have it.

Inkling AI Model Sizes & Requirements

Inkling AI ships as BF16 full weights plus an official NVFP4 variant, and Unsloth publishes aggressive dynamic GGUF quantizations with day-zero access. Pick by the hardware you have:

Variant Disk Size Quality Hardware Needed
BF16 (full weights) ~1.9 TB 100% (reference) Multi-node GPU cluster
NVFP4 (official) ~500 GB Well-calibrated official variant 8× B200 class
Unsloth Dynamic 2-bit 317 GB ~81% top-1 retained ~340 GB RAM+VRAM
Unsloth Dynamic 1-bit (UD-IQ1_S) 270–285 GB ~74.2% top-1 retained ~290 GB RAM+VRAM (Mac Studio Ultra)

Source: Unsloth's Inkling documentation. The 1-bit dynamic quant retains ~74.2% of top-1 accuracy while being 86% smaller — shrinking size far faster than it loses quality.

Four Ways to Run Inkling AI

Be honest about your hardware before downloading 270GB. For most people the cloud path wins on cost and time.

Cloud GPU (most practical)

Renting GPU time is the realistic path for most people: an 8×H100 or B200-class node runs quantized Inkling AI well. Spot pricing on GPU marketplaces starts around $2–4/hour per H100 — a weekend of experiments costs less than the RAM upgrade a local build would need.

Best for: everyone without a 290GB+ RAM workstation

Mac Studio Ultra

The Unsloth Dynamic 1-bit GGUF (270GB) fits on a maxed-out Mac Studio with unified memory around 290GB+. Expect usable but not fast generation — this is the cheapest fully-local path to run Inkling AI today.

Best for: Mac owners who want fully local inference

Multi-GPU workstation

The 2-bit quant (317GB) needs roughly 340GB of combined RAM+VRAM; 6/8-bit variants need up to 900GB. That means multiple RTX 6000 Ada / A100-class cards plus deep system RAM — realistic for labs, not hobbyists.

Best for: teams with existing GPU infrastructure

Hosted APIs (zero setup)

Inkling AI is available on Thinking Machines' Tinker platform (with a chat Playground), and through the Databricks Unity AI Gateway. No hardware needed — and Tinker adds fine-tuning on top.

Best for: trying Inkling before committing to hardware

Running Inkling AI Locally with llama.cpp

If your machine clears the 290GB RAM+VRAM bar, the Unsloth GGUF route is the proven path:

# 1. Download the dynamic 1-bit GGUF (270GB — use hf_transfer)
pip install huggingface_hub hf_transfer
hf download unsloth/Inkling-GGUF --include "*UD-IQ1_S*"
# 2. Run with llama.cpp (offload what fits to GPU)
./llama-cli -m Inkling-UD-IQ1_S.gguf --ctx-size 32768 \
--n-gpu-layers 99 --temp 0.7

Full flags and tuning tips are in Unsloth's Inkling guide. Start with a small context size — 1M context at this model scale multiplies memory needs dramatically.

Running Inkling FAQ

What hardware do I need to run Inkling AI locally?
At minimum, about 290GB of combined RAM+VRAM for the Unsloth Dynamic 1-bit quant (270GB on disk) — a maxed-out Mac Studio Ultra qualifies. The 2-bit needs ~340GB, and 6/8-bit variants need up to 900GB of RAM.
Can I run Inkling AI on a single RTX 4090 or 5090?
No. Even the most aggressive 1-bit quantization is 270GB — over ten times a 5090's VRAM. For single-GPU budgets, wait for Inkling-Small (12B active) or use a hosted API.
How much accuracy do the quantized versions lose?
Unsloth's Dynamic 1-bit retains about 74.2% of top-1 accuracy while being 86% smaller than full weights; Dynamic 2-bit retains ~81% at 82% smaller. Degradation is real but far less than the size reduction suggests.
What does it cost to run Inkling AI on cloud GPUs?
H100s rent for roughly $2–4/hour on GPU marketplaces; a quantized Inkling deployment on an 8×H100 node lands around $20–30/hour. For occasional use, hosted APIs on Tinker or Databricks are far cheaper than dedicated rentals.
Is there an official quantized version?
Yes — Thinking Machines ships a well-calibrated NVFP4 variant alongside the BF16 full weights, cutting the footprint to roughly a quarter with minimal quality loss on NVIDIA hardware that supports FP4.
Can I fine-tune Inkling AI without owning GPUs?
Yes. That is Thinking Machines' core pitch: Inkling is available for fine-tuning on the Tinker platform from day one — the launch demo even had Inkling write and run its own fine-tuning job.

Keep Exploring

Inkling vs Other Models Inkling-Small Preview