Inkling 2 Mini on 12, 16, and 24 GB GPUs: Which Quant Fits, and Doubling Context with -ub 128
We tested Unsloth's Mainline_Support GGUFs (UD-Q2_K_XL, Q3_K_XL, Q4_K_XL) of Inkling 2 Mini on llama.cpp b12611 on an A100 against 8, 12, 16, and 24 GB card budgets. KV allocates for 48 loop visits rather than 24 physical layers (1,264 MiB at 32K), while the CUDA compute buffer takes 3,356.2 MiB at default -ub 512. Lowering -ub to 128 cuts compute memory nearly in half twice, keeping 43K on 12 GB (Q2), 41K on 16 GB (Q3), and 121K on 24 GB (Q4) within budget after a 30K-token prompt. On 4 Python coding tasks at 16K, Q2 scored 0.975 within 12 GB (11,586 MiB peak, 174 MiB margin).

Per Thinking Machines Lab's model card (Parameters and Required hardware sections), Inkling 2 Mini has 26.4B total parameters (3.4B active per pass, 5.4B effective active across three loop passes), ships as an official ~20 GB checkpoint, and recommends 32 GB+ VRAM or a 32 GB+ Apple Silicon Mac. For local users running 12, 16, or 24 GB GPUs, the immediate questions are which Unsloth GGUF quant fits on the card, whether 32K context fits with it, and how much quality is lost down at Q2. We ran Unsloth's Mainline_Support UD-Q2_K_XL, UD-Q3_K_XL, and UD-Q4_K_XL files on llama.cpp b12611 to find out.
Three findings drive the setup:
- KV cache allocates for 48 layer visits, not 24 physical layers. Inkling 2 Mini loops its middle 12 layers three times. Counting 48 KV slots (8 global visits plus 40 local sliding-window visits) yielded
1,264 MiBat 32K, andllama-serverallocated exactly1,264 MiB(zero error, double the 24-layer count). - The compute buffer, not KV, eats most of the context VRAM — and
-ubnearly halves it. At 32K with default-ub 512, the CUDA0 compute buffer takes3,356.2 MiB. Lowering-ubfrom512to256and128cuts that compute buffer nearly in half at each step, keeping up to 43K on 12 GB (Q2), 41K on 16 GB (Q3), and 121K on 24 GB (Q4) within budget after launching at that-cand running a 30K-token prompt. - UD-Q2_K_XL scored
0.975on our 4 Python coding tasks inside 12 GB. At-c 16384and-ub 512, Q2 peaked at11,586 MiB(leaving174 MiBinside our11,760 MiB12 GB budget) and beat the previous 12 GB leader from our earlier comparison, Qwen3.5-9B (0.781).
All figures below were measured on an A100 80GB PCIe against card VRAM budgets (7,664 / 11,760 / 15,856 / 24,048 MiB for 8/12/16/24 GB, computed as GiB × 1,024 − 512 − 16 MiB), and every tok/s figure is A100 speed. No quant fits fully on an 8 GB card (Q2_K_XL CUDA model buffer alone is 8,469.3 MiB), and upstream llama.cpp requires the Mainline_Support/ GGUF file plus build b12611 or newer.
KV allocates for 48 layer visits, matching our count to 0 MiB
Per config.json and inkling2-loop.cpp, Inkling 2 Mini has 24 physical layers (0–23), and layers 6–17 run three times for 48 layer applications per token. Upstream llama.cpp assigns a separate KV slot per pass (each cache slot keeps its own KV). Global attention layers (5, 11, 17, 23) run once at 5 and 23 and three times at 11 and 17 — 8 global visits. The remaining 20 physical layers use a 1,024-token sliding window and total 40 local visits.
With f16 KV (2 × 8 KV heads × 128 head_dim × 2 B = 4,096 B per visit-token), the local sliding-window cache allocates 1,536 cells (240 MiB across 40 slots) at -ub 512, or 1,280 cells (200 MiB) at -ub 256 and 128. Measured llama_kv_cache buffers matched that 48-visit formula identically across Q2, Q3, and Q4:
| Context | 48-visit formula (global + local) | Measured KV buffer | Naive 24-layer count |
|---|---|---|---|
| 8,192 (8K) | 496 MiB (256 + 240) | 496 MiB | 248 MiB |
| 32,768 (32K) | 1,264 MiB (1,024 + 240) | 1,264 MiB | 632 MiB |
| 65,536 (64K) | 2,288 MiB (2,048 + 240) | 2,288 MiB | 1,144 MiB |
| 131,072 (128K) | 4,336 MiB (4,096 + 240) | 4,336 MiB | 2,168 MiB |
At 8K, the server log prints 8192 cells, 8 layers (256 MiB) and 1536 cells, 40 layers (240 MiB), confirming all 48 KV slots directly.

Compute buffer is the main VRAM sink — and `-ub` nearly halves it
Even though KV doubles relative to 24 layers, the right panel above shows that the CUDA0 compute buffer dominates long-context VRAM at default -ub 512 (using -ngl 99 -fa on --parallel 1 --jinja --no-webui, f16 KV, read from nvidia-smi right after server health ok):
| Context | KV (MiB) | CUDA0 compute (-ub 512) | Q2 process | Q3 process | Q4 process |
|---|---|---|---|---|---|
| 8K | 496 | 980.2 | 10,434 | 14,650 | 18,298 |
| 32K | 1,264 | 3,356.2 | 13,578 | 17,794 | 21,442 |
| 64K | 2,288 | 6,524.2 | 17,770 | 21,986 | 25,634 |
| 128K | 4,336 | 12,860.2 | 26,154 | 30,370 | 34,018 |
CUDA model buffers account for 8,469.3 MiB (Q2), 12,685.2 MiB (Q3), and 16,332.4 MiB (Q4), with CPU_Mapped buffers of 234.7, 303.7, and 469.3 MiB.
By reading inkling2.cpp in llama.cpp, the cause is that Inkling 2 Mini uses relative position biases instead of RoPE: the f32 bias tensor (kq_b) is sized heads × context × ubatch, plus a contiguous copy. What we directly verified by measurement is the linear scaling itself: across a sweep at 32K and 128K using an exact 30,000-token prompt (sent as token IDs), halving -ub cuts the CUDA0 compute buffer nearly in half every time (for example, 1,677.8 to 837.3 MiB at 32K, against an exact half of 838.9 MiB).
Quant · -ub | 32K (loaded / after prompt / compute / A100 prompt tok/s) | 128K (loaded / after prompt / compute / A100 prompt tok/s) |
|---|---|---|
Q2 · ub 512 | 13,578 / 13,594 / 3,356.2 / 662.8 | 26,154 / 26,170 / 12,860.2 / 665.3 |
Q2 · ub 256 | 11,858 / 11,870 / 1,677.8 / 634.5 | 19,682 / 19,694 / 6,429.8 / 636.3 |
Q2 · ub 128 | 11,018 / 11,126 / 837.3 / 559.6 | 16,466 / 16,574 / 3,213.3 / 560.4 |
Q3 · ub 512 | 17,794 / 17,810 / 3,356.2 / 645.5 | 30,370 / 30,386 / 12,860.2 / 643.6 |
Q3 · ub 256 | 16,074 / 16,086 / 1,677.8 / 602.2 | 23,898 / 23,910 / 6,429.8 / 602.4 |
Q3 · ub 128 | 15,234 / 15,342 / 837.3 / 558.5 | 20,682 / 20,790 / 3,213.3 / 559.8 |
Q4 · ub 512 | 21,442 / 21,456 / 3,356.2 / 642.5 | 34,018 / 34,032 / 12,860.2 / 642.7 |
Q4 · ub 256 | 19,722 / 19,732 / 1,677.8 / 600.3 | 27,546 / 27,556 / 6,429.8 / 600.3 |
Q4 · ub 128 | 18,882 / 18,982 / 837.3 / 563.7 | 24,330 / 24,430 / 3,213.3 / 563.9 |
Across all three quants, total VRAM grows by 131.0 MiB per 1K context at -ub 512, 81.5 MiB/1K at -ub 256, and 56.8 MiB/1K at -ub 128 (two-point slope between 32K and 128K). The trade-off on our A100 is prompt ingestion speed: -ub 256 runs 4-7% slower and -ub 128 runs 12-16% slower than -ub 512. At 32K and above, lowering -ub decides whether a quant fits on the card at all.
What fits 8, 12, 16, and 24 GB cards (and vision `mmproj`)
Using the same card budget rule as our gpu-fit and sotaaz-8gb posts (card GiB × 1,024 − 512 MiB display/apps − 16 MiB margin), we calculated the maximum -c (floored to a multiple of 1,024) from the 32K–128K line for every quant and -ub. We then launched all 18 fitting cells at that -c, ran a 30,000-token prompt (-c − 512 on the three cells under 30K), and verified process VRAM stayed within budget. All 18 stayed under budget, though the 15 cells of 30K or larger did not fill the full window — the 259K cell was tested at about 12% full.

Text-only, f16 KV, full GPU (each cell launched at the -c shown and verified under budget after a 30K, or -c − 512, prompt):
| Quant | -ub | 8 GB (7,664 MiB) | 12 GB (11,760 MiB) | 16 GB (15,856 MiB) | 24 GB (24,048 MiB) |
|---|---|---|---|---|---|
UD-Q2_K_XL | 512 (default) | Does not fit | 18K | 49K | 111K |
UD-Q2_K_XL | 256 | Does not fit | 30K | 80K | 181K |
UD-Q2_K_XL | 128 | Does not fit | 43K | 115K | 259K |
UD-Q3_K_XL | 512 (default) | Does not fit | Does not fit | 17K | 79K |
UD-Q3_K_XL | 256 | Does not fit | Does not fit | 29K | 129K |
UD-Q3_K_XL | 128 | Does not fit | Does not fit | 41K | 185K |
UD-Q4_K_XL | 512 (default) | Does not fit | Does not fit | Does not fit | 51K |
UD-Q4_K_XL | 256 | Does not fit | Does not fit | Does not fit | 84K |
UD-Q4_K_XL | 128 | Does not fit | Does not fit | Does not fit | 121K |
Four practical notes apply to these table values:
- Headroom after the 30K prompt was
0–106 MiB. The straight-line estimate is tight enough that 12 GBQ2 -ub 512(18K, 17,920-token prompt) left0 MiBof margin and 16 GBQ3 -ub 128(41K, 30K prompt) left2 MiB. Because your desktop may reserve more than512 MiB, pick slightly below the table ceiling in daily use. - 32K on 12 GB requires lowering
-ub. Default-ub 512needs13,594 MiBat 32K;-ub 256reaches30K, and-ub 128reaches43K. - 8 GB cannot fit the model fully on GPU. Even
Q2_K_XLneeds8,469.3 MiBfor its CUDA model buffer alone against the7,664 MiBbudget. To run Inkling 2 Mini on 8 GB, use a CPU MoE offload option such as--n-cpu-moe. - Vision (
mmproj-F16.gguf) and >128K quality: Addingmmprojconsumed an extra+710 MiBidentically at 8K and 32K across all three quants (including+98.8 MiBcompute buffer).Q4 + mmprojat 32K uses22,152 MiBright after load (before any image input), inside the 24 GB budget (24,048 MiB). Dividing 710 MiB by the slopes above gives an estimated-cdrop of about5Kat-ub 512and12Kat-ub 128, so subtract that headroom when enabling vision. Becausen_ctx_trainis 1M (1,048,576), cells above 128K stay inside the model's trained window; check output quality on your own long prompts before relying on them.
Coding quality and perplexity from Q4 down to Q2
To measure how much agent coding ability Q2 gives up, we ran the exact harness from our 8/12/16/24 GB local coding model comparison: four small Python tasks (T1–T4) × 3 runs each (12 runs per quant, -c 16384, -ub 512, agent.py 11568cd). A run's score is the pass rate on hidden tests against the final workspace, and we track both hidden-test passes and whether the agent declared done.

| Quant | Mean score | T1 (6 tests) | T2 (8 tests) | T3 (8 tests) | T4 (10 tests) | Peak VRAM | A100 decode tok/s | wikitext-2 PPL (b12611) |
|---|---|---|---|---|---|---|---|---|
UD-Q2_K_XL | 0.975 | 6/6 ×3 | 8/8 ×3 | 8/8 ×3 | 10, 7, 10 | 11,586 MiB (fits 12 GB) | 35.1-43.0 (med 42.6) | 6.27 ± 0.10 |
UD-Q3_K_XL | 0.958 | 6/6 ×3 | 8/8 ×3 | 8/8 ×3 | 9, 10, 6 | 15,802 MiB (fits 16 GB) | 41.8-42.3 (med 42.0, 10 clean runs) | 6.08 ± 0.10 |
UD-Q4_K_XL | 1.000 | 6/6 ×3 | 8/8 ×3 | 8/8 ×3 | 10, 10, 10 | 19,442 MiB (fits 24 GB) | 33.3-41.6 (med 41.3) | 6.04 ± 0.10 |
Four details matter when reading this table:
- Every missed test came from T4 (semver range matching), with
donedeclared. T1, T2, and T3 were 100% across all 27 runs. In the three imperfect T4 runs (Q2 r27/10</<=bugs;Q3 r19/10max_satisfyingreturning"1.10.1"instead of"v1.10.1";Q3 r36/10ValueErroron=1.2.3/||plus prerelease handling), visible tests passed and the agent declareddone. - Q4 declared
donein 11 of 12 runs, not 12 of 12. On T4 run 2, Q4 hit the 1,800-second timeout without declaringdone; its workspace at timeout already passed 10/10 hidden tests, which scores1.0under the harness rule (35 of 36 runs total declareddone). - Q3 speed exclusion and sample variance: Two of Q3's 12 runs (T3 r2 and T4 r2) shared GPU 1 with another external job (
11,898 MiB) and are excluded from the A100 decode range and median (42.0) above; neither timed out, so their scores remain. With only 3 runs per task, one T4 run (Q3 r3, 6/10) dropped Q3 (0.958) slightly below Q2 (0.975). Onwikitext-2test perplexity (20 chunks × 2,048 tokens), Q4 scored6.04 ± 0.10, Q36.08 ± 0.10, and Q26.27 ± 0.10: Q2 sits roughly0.2higher, while the0.04difference between Q3 and Q4 is smaller than the±0.10error bar and cannot separate the two. - Against our earlier
sotaaz-8gbbaselines (and the 174 MiB margin): At-c 16384 -ub 512,Inkling Q2_K_XL(0.975) peaked at11,586 MiB, stayed inside the 12 GB budget (11,760 MiB), and moved ahead of the previous 12 GB leader Qwen3.5-9B (0.781). However, Qwen3.5-9B left5,708 MiBof headroom on 12 GB, whereas Inkling Q2 leaves only174 MiB. As warned in the card-fit notes above, desktop usage can exceed512 MiB, so-c 16384at default-ub 512sits tight against the 12 GB limit unless you lower-ub. For reference on larger cards, Flint UD-Q2_K_XL (the model used to build the harness and T1) scored1.000on 16 GB and 24 GB, while Qwen3.5-27B scored0.953on 24 GB.
Before downloading: grab `Mainline_Support/` and build `b12611`+
Upstream llama.cpp users hit two sharp edges if they pull the default GGUF or build a day-old tag. As Unsloth's README notes, the root-level GGUFs (Inkling-2-Mini-UD-Q2_K_XL.gguf, etc.) use an unrolled 48-block format (general.architecture = inkling2, block_count = 48) meant for Unsloth Studio and custom builds; loading one in upstream llama-server fails immediately with check_tensor_dims: tensor 'blk.24.attn_q.weight' not found.
Instead, download from Mainline_Support/ (inkling2_loop, 24 physical layers plus loop metadata) and rename .gguf_file to .gguf. You also need llama.cpp b12611 (690b87d, Nov 21, 2026 — first tag with PR #32704) or newer. Running the exact same Mainline_Support Q2 file on b12603 vs b12611 shows why:
| Build | Q2_K_XL wikitext-2 PPL | Prompt: Fibonacci function, code only (temp 0, 120 tokens) |
|---|---|---|
b12603 (before PR #32704) | 117.34 ± 3.01 | "The user is a Python function/ ... F. F." followed by repeated invisible zero-width spaces |
b12611 (with PR #32704) | 6.27 ± 0.10 | "Need decide indexing convention. Common: F(0)=0, F(1)=1." clean reasoning into def fibonacci(n): |
Recommended setups per card
Based on our measurements on A100 against card VRAM budgets, these are the setups we would pick:
- 12 GB (e.g., RTX 3060 12 GB):
UD-Q2_K_XL. At 16K, default-ub 512leaves only174 MiB(11,586vs11,760 MiBbudget), which is tight if your desktop takes more than512 MiB. Use-ub 256for 16K (30Kstays within budget after a 30K-token prompt) or-ub 128for 32K (43Kstays within budget after a 30K-token prompt). - 16 GB (e.g., RTX 4070 Ti Super):
UD-Q3_K_XLwith-ub 128(41Kstays within budget after a 30K-token prompt) passes 32K context with Q3 (6.08 ± 0.10perplexity vs6.27 ± 0.10on Q2); pickUD-Q2_K_XLwith-ub 256/128if you need80K–115Kwithin budget after a 30K prompt. - 24 GB (e.g., RTX 4090):
UD-Q4_K_XLstays within budget up to51Kafter a 30K-token prompt even at default-ub 512, and holds32Kwith visionmmproj(22,152 MiBright after load, before any image input, vs24,048 MiBbudget). For ~100K+ context, use-ub 128on Q4 (121Kwithin budget after a 30K prompt) or Q3 (-ub 256129K/-ub 128185K). - 8 GB: Does not fit fully on GPU; keep Qwen3.5-9B from our 8 GB comparison or test partial CPU MoE offloading (
--n-cpu-moe) if you want to try Inkling 2 Mini on 8 GB.
Example download and 32K launch for a 12 GB GPU on llama.cpp b12611+:
hf download unsloth/Inkling-2-Mini-GGUF Mainline_Support/Inkling-2-Mini-UD-Q2_K_XL.gguf_file --local-dir models
mv models/Mainline_Support/Inkling-2-Mini-UD-Q2_K_XL.gguf_file models/Inkling-2-Mini-UD-Q2_K_XL.gguf
llama-server \
-m models/Inkling-2-Mini-UD-Q2_K_XL.gguf \
-c 32768 -ub 128 -ngl 99 -fa on --parallel 1 --jinja --no-webui --port 8080When calling /v1/chat/completions against this server, note that Inkling 2 Mini is a reasoning model and emits its thought trace in the reasoning_content field.
Before pushing -c close to a row limit, check your actual free VRAM with nvidia-smi --query-gpu=memory.free --format=csv and verify headroom on your own card using the two short runs and a straight line check.
The newsletter below sends the next episode of this series when it is out.
