Can a 32 GB Mac Run a Model Three Times Its Memory? Qwen3.8-Flash-Next on an M2 Max

Can a 32 GB Mac Run a Model Three Times Its Memory? Qwen3.8-Flash-Next on an M2 Max

Qwen3.8-Flash-Next is a mixture-of-experts model whose 4-bit weights take about 105 GB (98 GiB). A 32 GB Mac can't hold that, so the usual answer to "can I run it on my Mac?" is no.

slotstream is a recent Mac app built to change that answer. It keeps the small, always-used parts of the model in memory. It leaves the experts, 512 per layer, on the SSD, reads only the ones each token needs, and keeps the most used ones in a memory cache. The bet is that a fast internal SSD can stand in for memory the model doesn't touch on every token.

Its author has measured speed carefully: 20.91-23.69 tokens per second on a 48 GB M5 Pro, plus a table of community reports from other Macs. For quality, the project runs a 15-question damage check (facts, arithmetic, sorting, instructions, translation, code) that passes 15/15. The author is clear that this only catches large damage and is not a comparison with the original model. Public benchmark scores such as GSM8K or HumanEval are not in the project's documents.

Our earlier post scored a different file of the same model (a 2-bit GGUF on an A100) and noted that it didn't test an SSD. So we put both questions to one ordinary machine: a 2023 MacBook Pro 14-inch, M2 Max, 32 GB, internal 1 TB SSD. How fast is it, how good are the answers, and are they repeatable?

How we measured

• Software: slotstream 0.2.28 as installed and its model qwen3.8-flash-next:4bit (an MLX 4-bit file). macOS 15.7.3, on AC power, other apps closed, sleep disabled. We wrote the plan and its rules before the first measurement.
• Memory plans: with other apps closed, slotstream doctor chose a 20.9 GB process budget. We used that automatic plan (A) for speed and scores; its server started with a cache of about 75 of 512 experts per layer (10 GB). For comparison we also ran a fixed 10 GB budget (B), whose server started with about 22 experts per layer (3 GB). The plans differ in more than the cache. Going from A to B, slotstream also turned off its expert lookahead (prefetching), moved the draft head's experts from memory to the SSD, read prompts 256 tokens at a time instead of 1,024, and lowered its prompt-reuse limit. Both kept speculative decoding (drafting tokens ahead) on.
• Speed: three short prompts, each sent three times, 256 output tokens. Decode speed is the server's own count (eval_count / eval_duration). Following the author's definition, "warm" is the third identical request. For long prompts we used slotstream's context-check with an 8,192-token prompt, the tool behind the "long prompt" column in its hardware table.
• Scores: the harness from our Strata post, with the same questions, prompts and grading: 300 GSM8K questions (seed 0) and all 164 HumanEval problems, thinking off, up to 1,024 output tokens, one request at a time. Two things changed for slotstream. It rejects unknown request fields, so thinking is turned off with its own think: false. And its default sampling is temperature 0.7 with a presence penalty of 1.5, so we set greedy decoding explicitly (temperature 0, top_k 1, presence penalty 0).
• Repeatability: the first 30 GSM8K questions asked three times on plan A, then once each with prompt reuse off on plan A (whose server then started with about 85 experts per layer) and on plan B.

Speed

On the automatic plan, a short answer streams at about 9.5 tokens per second, close to doctor's estimate. The 10 GB plan decoded at half that speed and took a third longer to read an 8K prompt. That gap comes from everything plan B changed together (cache size, lookahead, where the draft head's experts live, prompt pass size), not from the cache alone.

For context, the community reports in slotstream's hardware table list 6.22 tokens/s for a 32 GB M5 Air and 8.41 for a 36 GB M4 Max. Those were on older releases and different plans, so we don't compute ratios.

Short-reply speed is not what a long job feels like. Over the 300 GSM8K questions, counting prompt reading, each request took a median of 37.5 seconds, and the whole set took 3.6 hours, or 6.9 output tokens per second. HumanEval took 1.8 hours.

Answer quality

For reference, our Strata post scored a different file of the same model, ISTA-DASLab's 2-bit IQ2_XS GGUF, on an A100: GSM8K 279 (llama.cpp) and 280 (Strata), HumanEval 157 for both. The files, engines and hardware differ, so we don't rank them. The two files' scores were one or two questions apart, but with the engine and hardware also different, that can't be read as a 2-bit versus 4-bit comparison. The original bf16 model's score on this harness is unknown, so these numbers also say nothing about how much the 4-bit file loses. For reference, on a decision model, cutting bits moved confidence before accuracy (A 2-Bit Model Remembers How to Be Right and Forgets How to Be Sure); this post did not measure probabilities.

Repeatability

Every output matched to the byte, including across the two plans, where the cache differed about fourfold and lookahead, prompt pass size and draft-expert placement differed too. slotstream's docs say the expert cache changes speed, not output, and its own tests check that. Here it held on a real task.

This isn't automatic for engines that move experts around. In our Strata post, with the GPU held to 12 GiB and default settings, Strata gave the same answer on all three repeats for only 12 of 30 questions. Strata's documentation describes this and offers switches that make it repeatable at some cost in speed.

What it costs

• Download and disk: 88.3 GB compressed, 19 minutes on our line, 98 GiB (about 105 GB) on disk. That much free space has to be on a fast internal SSD. The author's table shows the same 64 GB M4 Max dropping from 15.93 to 2.98 tokens/s on a 10 Gb/s external drive. We had no external drive to test.
• SSD wear: reading experts from the SSD is reading, not writing. slotstream writes conversation state to disk only when started with --prefix-cache-dir, which we didn't use. We didn't measure the bytes written, so treat this as the design, not a measurement.
• Time and heat: long jobs run for hours, and the Mac does nothing else meanwhile. Over the 3.6-hour GSM8K run the laptop didn't slow down: the first 50 requests ran at a median 6.73 output tokens per second and the last 50 at 6.83.
• One dropped request. On HumanEval problem 32, a reply that ran to the full 1,024 tokens was stopped after 106 seconds with insufficient_memory, and the client got an HTTP 503. slotstream then shrank its cache from about 75 to about 60 experts per layer, and grew it back a few minutes later. Sending the same request again succeeded in 125 seconds. slotstream's command reference says the automatic cache "resizes between requests", and in our log the cache shrank right after this request ended. So a memory squeeze in the middle of a long reply ends that reply. If you script long jobs against it, retry on 503. We don't know what took the memory at that moment.

SOTAAZ course

This course alone $79 · 24 lectures · first 3 freeWatch free →

Compared with a 9B model that fits in memory

A 32 GB Mac owner isn't choosing between this model and nothing. The real choice is this model or one that fits in memory. So on the same Mac we ran Qwen3.5-9B Q4_K_M (5.7 GB) in llama.cpp (0.6.0, Metal) with the same harness. The comparison rule was set before measuring: a McNemar exact test on the same questions, p < 0.05 means the accuracies differ, otherwise this many questions can't tell them apart. The engines differ, so the speeds below compare each model the way you would actually run it.

On GSM8K the two can't be told apart with 300 questions: Flash-Next alone solved 14, the 9B alone 7 (p = 0.19). On HumanEval they differ on the same problems: Flash-Next alone solved 20, the 9B alone 2 (p = 0.0001). The 9B also had 5 answers cut at the token limit on each benchmark.

So, can a 32 GB Mac run it?

Yes, for answers you can wait for. On this M2 Max a short reply arrives at about 9.5 tokens per second and the same question gets the same answer. Against a 9B model that fits in memory, the trade is clear: about 18 times the disk and about five times the wait for the same job. In return, no detectable difference on grade-school math (GSM8K), and 18 more of 164 coding problems (HumanEval) solved. We didn't test agent work; at 6.9 tokens per second over a long job, anything that needs many long replies in a row takes hours.

Limits

• One Mac (M2 Max, 32 GB, internal SSD), one release (0.2.28), one model file. The comparison is with one 9B model; a larger model that still fits in 32 GB (a 27B, say) was not measured. Speeds on other Macs, releases or drives can differ a lot, as the author's own table shows.
• Thinking off and replies of up to 1,024 tokens. Long reasoning and agent work were not tested.
• Three hundred and 164 questions can only show large quality gaps.
• Plan B is the same Mac with a smaller budget, not a 16 GB Mac.
• The repeatability checks used 30 questions.
• Two runs were interrupted and resumed on the same server: HumanEval after the dropped request at problem 32, and the plan B run after our client was stopped at question 22 by its own time limit (resumed at 23). The times and points are in the raw notes.

The plan, the client script, and every raw file from the Mac (requests, server logs, doctor output, download record) are in the reproduction package below.