Models & Algorithms•SOTAAZ Lab••KR

Inkling 2 Mini on 12, 16, and 24 GB GPUs: Which Quant Fits, and Doubling Context with -ub 128

We tested Unsloth's Mainline_Support GGUFs (UD-Q2_K_XL, Q3_K_XL, Q4_K_XL) of Inkling 2 Mini on llama.cpp b12611 on an A100 against 8, 12, 16, and 24 GB card budgets. KV allocates for 48 loop visits rather than 24 physical layers (1,264 MiB at 32K), while the CUDA compute buffer takes 3,356.2 MiB at default -ub 512. Lowering -ub to 128 cuts compute memory nearly in half twice, keeping 43K on 12 GB (Q2), 41K on 16 GB (Q3), and 121K on 24 GB (Q4) within budget after a 30K-token prompt. On 4 Python coding tasks at 16K, Q2 scored 0.975 within 12 GB (11,586 MiB peak, 174 MiB margin).

Inkling 2 Mini on 12, 16, and 24 GB GPUs: Which Quant Fits, and Doubling Context with -ub 128

Per Thinking Machines Lab's model card (Parameters and Required hardware sections), Inkling 2 Mini has 26.4B total parameters (3.4B active per pass, 5.4B effective active across three loop passes), ships as an official ~20 GB checkpoint, and recommends 32 GB+ VRAM or a 32 GB+ Apple Silicon Mac. For local users running 12, 16, or 24 GB GPUs, the immediate questions are which Unsloth GGUF quant fits on the card, whether 32K context fits with it, and how much quality is lost down at Q2. We ran Unsloth's Mainline_Support UD-Q2_K_XL, UD-Q3_K_XL, and UD-Q4_K_XL files on llama.cpp b12611 to find out.

Three findings drive the setup:

  1. KV cache allocates for 48 layer visits, not 24 physical layers. Inkling 2 Mini loops its middle 12 layers three times. Counting 48 KV slots (8 global visits plus 40 local sliding-window visits) yielded 1,264 MiB at 32K, and llama-server allocated exactly 1,264 MiB (zero error, double the 24-layer count).
  2. The compute buffer, not KV, eats most of the context VRAM — and -ub nearly halves it. At 32K with default -ub 512, the CUDA0 compute buffer takes 3,356.2 MiB. Lowering -ub from 512 to 256 and 128 cuts that compute buffer nearly in half at each step, keeping up to 43K on 12 GB (Q2), 41K on 16 GB (Q3), and 121K on 24 GB (Q4) within budget after launching at that -c and running a 30K-token prompt.
  3. UD-Q2_K_XL scored 0.975 on our 4 Python coding tasks inside 12 GB. At -c 16384 and -ub 512, Q2 peaked at 11,586 MiB (leaving 174 MiB inside our 11,760 MiB 12 GB budget) and beat the previous 12 GB leader from our earlier comparison, Qwen3.5-9B (0.781).

All figures below were measured on an A100 80GB PCIe against card VRAM budgets (7,664 / 11,760 / 15,856 / 24,048 MiB for 8/12/16/24 GB, computed as GiB × 1,024 − 512 − 16 MiB), and every tok/s figure is A100 speed. No quant fits fully on an 8 GB card (Q2_K_XL CUDA model buffer alone is 8,469.3 MiB), and upstream llama.cpp requires the Mainline_Support/ GGUF file plus build b12611 or newer.

KV allocates for 48 layer visits, matching our count to 0 MiB

Per config.json and inkling2-loop.cpp, Inkling 2 Mini has 24 physical layers (0–23), and layers 6–17 run three times for 48 layer applications per token. Upstream llama.cpp assigns a separate KV slot per pass (each cache slot keeps its own KV). Global attention layers (5, 11, 17, 23) run once at 5 and 23 and three times at 11 and 17 — 8 global visits. The remaining 20 physical layers use a 1,024-token sliding window and total 40 local visits.

With f16 KV (2 × 8 KV heads × 128 head_dim × 2 B = 4,096 B per visit-token), the local sliding-window cache allocates 1,536 cells (240 MiB across 40 slots) at -ub 512, or 1,280 cells (200 MiB) at -ub 256 and 128. Measured llama_kv_cache buffers matched that 48-visit formula identically across Q2, Q3, and Q4:

Context48-visit formula (global + local)Measured KV bufferNaive 24-layer count
8,192 (8K)496 MiB (256 + 240)496 MiB248 MiB
32,768 (32K)1,264 MiB (1,024 + 240)1,264 MiB632 MiB
65,536 (64K)2,288 MiB (2,048 + 240)2,288 MiB1,144 MiB
131,072 (128K)4,336 MiB (4,096 + 240)4,336 MiB2,168 MiB

At 8K, the server log prints 8192 cells, 8 layers (256 MiB) and 1536 cells, 40 layers (240 MiB), confirming all 48 KV slots directly.

Two-panel VRAM chart for Inkling 2 Mini. Left: KV buffer sizes at 8K, 32K, and 128K comparing naive 24-layer numbers (248, 632, 2,168 MiB) against 48-visit formula and measured values (496, 1,264, 4,336 MiB, 0 MiB error). Right: stacked Q2_K_XL process VRAM at -ub 512 from 8K (10,434 MiB) to 128K (26,154 MiB), showing the CUDA0 compute buffer growing from 980 to 12,860 MiB against 12, 16, and 24 GB budgets.

Compute buffer is the main VRAM sink — and `-ub` nearly halves it

Even though KV doubles relative to 24 layers, the right panel above shows that the CUDA0 compute buffer dominates long-context VRAM at default -ub 512 (using -ngl 99 -fa on --parallel 1 --jinja --no-webui, f16 KV, read from nvidia-smi right after server health ok):

ContextKV (MiB)CUDA0 compute (-ub 512)Q2 processQ3 processQ4 process
8K496980.210,43414,65018,298
32K1,2643,356.213,57817,79421,442
64K2,2886,524.217,77021,98625,634
128K4,33612,860.226,15430,37034,018

CUDA model buffers account for 8,469.3 MiB (Q2), 12,685.2 MiB (Q3), and 16,332.4 MiB (Q4), with CPU_Mapped buffers of 234.7, 303.7, and 469.3 MiB.

By reading inkling2.cpp in llama.cpp, the cause is that Inkling 2 Mini uses relative position biases instead of RoPE: the f32 bias tensor (kq_b) is sized heads × context × ubatch, plus a contiguous copy. What we directly verified by measurement is the linear scaling itself: across a sweep at 32K and 128K using an exact 30,000-token prompt (sent as token IDs), halving -ub cuts the CUDA0 compute buffer nearly in half every time (for example, 1,677.8 to 837.3 MiB at 32K, against an exact half of 838.9 MiB).

Quant · -ub32K (loaded / after prompt / compute / A100 prompt tok/s)128K (loaded / after prompt / compute / A100 prompt tok/s)
Q2 · ub 51213,578 / 13,594 / 3,356.2 / 662.826,154 / 26,170 / 12,860.2 / 665.3
Q2 · ub 25611,858 / 11,870 / 1,677.8 / 634.519,682 / 19,694 / 6,429.8 / 636.3
Q2 · ub 12811,018 / 11,126 / 837.3 / 559.616,466 / 16,574 / 3,213.3 / 560.4
Q3 · ub 51217,794 / 17,810 / 3,356.2 / 645.530,370 / 30,386 / 12,860.2 / 643.6
Q3 · ub 25616,074 / 16,086 / 1,677.8 / 602.223,898 / 23,910 / 6,429.8 / 602.4
Q3 · ub 12815,234 / 15,342 / 837.3 / 558.520,682 / 20,790 / 3,213.3 / 559.8
Q4 · ub 51221,442 / 21,456 / 3,356.2 / 642.534,018 / 34,032 / 12,860.2 / 642.7
Q4 · ub 25619,722 / 19,732 / 1,677.8 / 600.327,546 / 27,556 / 6,429.8 / 600.3
Q4 · ub 12818,882 / 18,982 / 837.3 / 563.724,330 / 24,430 / 3,213.3 / 563.9

Across all three quants, total VRAM grows by 131.0 MiB per 1K context at -ub 512, 81.5 MiB/1K at -ub 256, and 56.8 MiB/1K at -ub 128 (two-point slope between 32K and 128K). The trade-off on our A100 is prompt ingestion speed: -ub 256 runs 4-7% slower and -ub 128 runs 12-16% slower than -ub 512. At 32K and above, lowering -ub decides whether a quant fits on the card at all.

What fits 8, 12, 16, and 24 GB cards (and vision `mmproj`)

Using the same card budget rule as our gpu-fit and sotaaz-8gb posts (card GiB × 1,024 − 512 MiB display/apps − 16 MiB margin), we calculated the maximum -c (floored to a multiple of 1,024) from the 32K–128K line for every quant and -ub. We then launched all 18 fitting cells at that -c, ran a 30,000-token prompt (-c − 512 on the three cells under 30K), and verified process VRAM stayed within budget. All 18 stayed under budget, though the 15 cells of 30K or larger did not fill the full window — the 259K cell was tested at about 12% full.

Two-panel -ub and card fit chart. Left: Q2_K_XL after-prompt VRAM at 32K and 128K across -ub 512, 256, and 128, showing how -ub 128 (11,126 MiB) drops below the 12 GB budget line (11,760 MiB). Right: max -c that stays within budget after launching at that -c and running a 30K (or -c - 512) prompt across all 18 card, quant, and -ub combinations.

Text-only, f16 KV, full GPU (each cell launched at the -c shown and verified under budget after a 30K, or -c − 512, prompt):

Quant-ub8 GB (7,664 MiB)12 GB (11,760 MiB)16 GB (15,856 MiB)24 GB (24,048 MiB)
UD-Q2_K_XL512 (default)Does not fit18K49K111K
UD-Q2_K_XL256Does not fit30K80K181K
UD-Q2_K_XL128Does not fit43K115K259K
UD-Q3_K_XL512 (default)Does not fitDoes not fit17K79K
UD-Q3_K_XL256Does not fitDoes not fit29K129K
UD-Q3_K_XL128Does not fitDoes not fit41K185K
UD-Q4_K_XL512 (default)Does not fitDoes not fitDoes not fit51K
UD-Q4_K_XL256Does not fitDoes not fitDoes not fit84K
UD-Q4_K_XL128Does not fitDoes not fitDoes not fit121K

Four practical notes apply to these table values:

  • Headroom after the 30K prompt was 0–106 MiB. The straight-line estimate is tight enough that 12 GB Q2 -ub 512 (18K, 17,920-token prompt) left 0 MiB of margin and 16 GB Q3 -ub 128 (41K, 30K prompt) left 2 MiB. Because your desktop may reserve more than 512 MiB, pick slightly below the table ceiling in daily use.
  • 32K on 12 GB requires lowering -ub. Default -ub 512 needs 13,594 MiB at 32K; -ub 256 reaches 30K, and -ub 128 reaches 43K.
  • 8 GB cannot fit the model fully on GPU. Even Q2_K_XL needs 8,469.3 MiB for its CUDA model buffer alone against the 7,664 MiB budget. To run Inkling 2 Mini on 8 GB, use a CPU MoE offload option such as --n-cpu-moe.
  • Vision (mmproj-F16.gguf) and >128K quality: Adding mmproj consumed an extra +710 MiB identically at 8K and 32K across all three quants (including +98.8 MiB compute buffer). Q4 + mmproj at 32K uses 22,152 MiB right after load (before any image input), inside the 24 GB budget (24,048 MiB). Dividing 710 MiB by the slopes above gives an estimated -c drop of about 5K at -ub 512 and 12K at -ub 128, so subtract that headroom when enabling vision. Because n_ctx_train is 1M (1,048,576), cells above 128K stay inside the model's trained window; check output quality on your own long prompts before relying on them.

Coding quality and perplexity from Q4 down to Q2

To measure how much agent coding ability Q2 gives up, we ran the exact harness from our 8/12/16/24 GB local coding model comparison: four small Python tasks (T1–T4) × 3 runs each (12 runs per quant, -c 16384, -ub 512, agent.py 11568cd). A run's score is the pass rate on hidden tests against the final workspace, and we track both hidden-test passes and whether the agent declared done.

Two-panel quality chart. Left: hidden-test scores across four Python coding tasks at 16K and -ub 512, comparing Inkling Q4 (1.000, 19,442 MiB peak), Q2 (0.975, 11,586 MiB), and Q3 (0.958, 15,802 MiB) against sotaaz-8gb leaders Flint (1.000), Qwen3.5-27B (0.953), and Qwen3.5-9B (0.781). Right: wikitext-2 perplexity (20 chunks of 2,048 tokens) dropping from 117.34 on b12603 to 6.27, 6.08, and 6.04 on b12611.
QuantMean scoreT1 (6 tests)T2 (8 tests)T3 (8 tests)T4 (10 tests)Peak VRAMA100 decode tok/swikitext-2 PPL (b12611)
UD-Q2_K_XL0.9756/6 ×38/8 ×38/8 ×310, 7, 1011,586 MiB (fits 12 GB)35.1-43.0 (med 42.6)6.27 ± 0.10
UD-Q3_K_XL0.9586/6 ×38/8 ×38/8 ×39, 10, 615,802 MiB (fits 16 GB)41.8-42.3 (med 42.0, 10 clean runs)6.08 ± 0.10
UD-Q4_K_XL1.0006/6 ×38/8 ×38/8 ×310, 10, 1019,442 MiB (fits 24 GB)33.3-41.6 (med 41.3)6.04 ± 0.10

Four details matter when reading this table:

  • Every missed test came from T4 (semver range matching), with done declared. T1, T2, and T3 were 100% across all 27 runs. In the three imperfect T4 runs (Q2 r2 7/10 </<= bugs; Q3 r1 9/10 max_satisfying returning "1.10.1" instead of "v1.10.1"; Q3 r3 6/10 ValueError on =1.2.3/|| plus prerelease handling), visible tests passed and the agent declared done.
  • Q4 declared done in 11 of 12 runs, not 12 of 12. On T4 run 2, Q4 hit the 1,800-second timeout without declaring done; its workspace at timeout already passed 10/10 hidden tests, which scores 1.0 under the harness rule (35 of 36 runs total declared done).
  • Q3 speed exclusion and sample variance: Two of Q3's 12 runs (T3 r2 and T4 r2) shared GPU 1 with another external job (11,898 MiB) and are excluded from the A100 decode range and median (42.0) above; neither timed out, so their scores remain. With only 3 runs per task, one T4 run (Q3 r3, 6/10) dropped Q3 (0.958) slightly below Q2 (0.975). On wikitext-2 test perplexity (20 chunks × 2,048 tokens), Q4 scored 6.04 ± 0.10, Q3 6.08 ± 0.10, and Q2 6.27 ± 0.10: Q2 sits roughly 0.2 higher, while the 0.04 difference between Q3 and Q4 is smaller than the ±0.10 error bar and cannot separate the two.
  • Against our earlier sotaaz-8gb baselines (and the 174 MiB margin): At -c 16384 -ub 512, Inkling Q2_K_XL (0.975) peaked at 11,586 MiB, stayed inside the 12 GB budget (11,760 MiB), and moved ahead of the previous 12 GB leader Qwen3.5-9B (0.781). However, Qwen3.5-9B left 5,708 MiB of headroom on 12 GB, whereas Inkling Q2 leaves only 174 MiB. As warned in the card-fit notes above, desktop usage can exceed 512 MiB, so -c 16384 at default -ub 512 sits tight against the 12 GB limit unless you lower -ub. For reference on larger cards, Flint UD-Q2_K_XL (the model used to build the harness and T1) scored 1.000 on 16 GB and 24 GB, while Qwen3.5-27B scored 0.953 on 24 GB.

Before downloading: grab `Mainline_Support/` and build `b12611`+

Upstream llama.cpp users hit two sharp edges if they pull the default GGUF or build a day-old tag. As Unsloth's README notes, the root-level GGUFs (Inkling-2-Mini-UD-Q2_K_XL.gguf, etc.) use an unrolled 48-block format (general.architecture = inkling2, block_count = 48) meant for Unsloth Studio and custom builds; loading one in upstream llama-server fails immediately with check_tensor_dims: tensor 'blk.24.attn_q.weight' not found.

Instead, download from Mainline_Support/ (inkling2_loop, 24 physical layers plus loop metadata) and rename .gguf_file to .gguf. You also need llama.cpp b12611 (690b87d, Nov 21, 2026 — first tag with PR #32704) or newer. Running the exact same Mainline_Support Q2 file on b12603 vs b12611 shows why:

BuildQ2_K_XL wikitext-2 PPLPrompt: Fibonacci function, code only (temp 0, 120 tokens)
b12603 (before PR #32704)117.34 ± 3.01"The user is a Python function/ ... F. F." followed by repeated invisible zero-width spaces
b12611 (with PR #32704)6.27 ± 0.10"Need decide indexing convention. Common: F(0)=0, F(1)=1." clean reasoning into def fibonacci(n):

Based on our measurements on A100 against card VRAM budgets, these are the setups we would pick:

  • 12 GB (e.g., RTX 3060 12 GB): UD-Q2_K_XL. At 16K, default -ub 512 leaves only 174 MiB (11,586 vs 11,760 MiB budget), which is tight if your desktop takes more than 512 MiB. Use -ub 256 for 16K (30K stays within budget after a 30K-token prompt) or -ub 128 for 32K (43K stays within budget after a 30K-token prompt).
  • 16 GB (e.g., RTX 4070 Ti Super): UD-Q3_K_XL with -ub 128 (41K stays within budget after a 30K-token prompt) passes 32K context with Q3 (6.08 ± 0.10 perplexity vs 6.27 ± 0.10 on Q2); pick UD-Q2_K_XL with -ub 256/128 if you need 80K–115K within budget after a 30K prompt.
  • 24 GB (e.g., RTX 4090): UD-Q4_K_XL stays within budget up to 51K after a 30K-token prompt even at default -ub 512, and holds 32K with vision mmproj (22,152 MiB right after load, before any image input, vs 24,048 MiB budget). For ~100K+ context, use -ub 128 on Q4 (121K within budget after a 30K prompt) or Q3 (-ub 256 129K / -ub 128 185K).
  • 8 GB: Does not fit fully on GPU; keep Qwen3.5-9B from our 8 GB comparison or test partial CPU MoE offloading (--n-cpu-moe) if you want to try Inkling 2 Mini on 8 GB.

Example download and 32K launch for a 12 GB GPU on llama.cpp b12611+:

bash
hf download unsloth/Inkling-2-Mini-GGUF Mainline_Support/Inkling-2-Mini-UD-Q2_K_XL.gguf_file --local-dir models
mv models/Mainline_Support/Inkling-2-Mini-UD-Q2_K_XL.gguf_file models/Inkling-2-Mini-UD-Q2_K_XL.gguf

llama-server \
  -m models/Inkling-2-Mini-UD-Q2_K_XL.gguf \
  -c 32768 -ub 128 -ngl 99 -fa on --parallel 1 --jinja --no-webui --port 8080

When calling /v1/chat/completions against this server, note that Inkling 2 Mini is a reasoning model and emits its thought trace in the reasoning_content field.

Before pushing -c close to a row limit, check your actual free VRAM with nvidia-smi --query-gpu=memory.free --format=csv and verify headroom on your own card using the two short runs and a straight line check.

The newsletter below sends the next episode of this series when it is out.