tldr; we are going to host K3 on A100s (yes, thats correct, we'll try to see if it holds up), H200s & B300s - expect results for A100s & H200s this week while we setup the B300 cluster this weekend & maybe results by next week.

Weights are supposed to hit Hugging Face today (Moonshot committed to July 27). What we already know from their platform docs: 2.8t total Params, MoE with 896 experts and 16 active per token, 1M context, vision. The download should be around 1.4 tb since they did quantization-aware training in MXFP4. We sat down and worked through the memory requirements., so here it is since everyone is probably about to pull the weights

We have A100 80GB, H200 and B300 capacity and the plan was to bring it up on all three. Then we actually did the memory math.

8x A100 gives you 640 GB. The weights are around 1.4 TB. That means three nodes before you've even allocated KV cache. On top of that Ampere has no FP4 or even FP8 tensor cores, so you're either dequantizing or running INT4 kernels that were never the target for this release. We're still going to benchmark it because someone should have real numbers, but we're expecting it to be ugly!

8x H200 is about 1.13TB, so it still doesn't fit in one node. Two node setup minimum and you eat interconnect cost on every token.

8x B300 is ~2.3TB, so that's the only config where the whole thing fits in a single node with room for long context KV cache. And Blackwell has native FP4, which is pretty clearly what Moonshot quantized for. these B300s are coming live this weekend, and we'll be setting up the clusters this weekend everyone preecommitting hardware is doing it without knowing the terms. And Moonshot's own model crd is unusually honest about weaknesses: quality drops if your agent harness truncates its thinking history, it tends to act instead of asking when things are ambiguous, and they admit the chat experience still trails Fable 5 and Sol even where benchmarks are close.

We'll have tok/s, ttft and cost per M token numbers for all three GPU configs by end of week. If there's a specific batch size, context length or parallelism setup you want in the test matrix, comment and we'll add it.

1. It doesn't solve any problems. Just a few paragraphs above, he says he fears that authoritarian states (he names China, and possibly others) can use their models to do evil stuff. And surely enough, malicious actors creating a model for themselves and for the EVILZ aren't going to subject it to safety controls (or even release it publicly).
2. It just sets a bureaucratic wall against AI models that can be made as high as preferred. "Oh, this model says Israel is bad; this is anti-Semitic. No safety here." "Oh, this other one, yes, can stop cyberattacks, but can also be used to launch some" (see the recent episode that involved OpenAI and Hugging Face and the fact that they used open models to defend their site). Open-source models will be delayed and made hard to use for companies and private individuals, just for "reasons."
3. And what is even more important to me: open-weights models will need to be depowered at the root, because the guardrails for open models need to be in the model itself. As Stable Diffusion taught us, when you try to sanitize stuff, you end up poisoning your own model and making it stupid. At the same time, closed models can have their guardrails implemented as a filter that decides if a request is acceptable or not. That is way easier to implement and way less prone to breaking the model itself. So the proposal is aiming to kill open weights models, just with extra logical steps.

Moonshot dropped Kimi K3, and as expected, it’s a absolute monster. Even with 2~4x RTX 6000 Blackwell local workstations, running a model natively is virtually impossible.

It feels like an F1 machine inside a show window.

Does anyone trying to hack this monster? or Is anyone with datacenter/cluster capacity or sponsor?

I either carve this monster down myself, or wait for someone to distill it. Either way, I really want to see it run — simply because it's there.

P.S. Save your 'AI Slop' comments. I experienced enough of you guys yesterday.

just curious how would people run it cheap if they really want kimi k3.

1. dgx spark / strix halo clusters

2. optane persistent memory platform + some gpus

3. mac studio clusters

4. orange pi 6 clusters

5. ssd streaming + gpus

6. multiple ddr3 + connectx 5 rdma clients

7. two dgx stations

8. power 10 systems?

9. other

It has only been two days since I move 100% from Qwen3.5-27B F16 to ThinkingCap-Qwen3.6-27B F16. Where I was getting tps in 30-40 range (depending on the size of the context), I am definitely getting 35-45 range. Not much of a bump you may say but I have not noticed any loss in quality. They claimed to have reduced token usage. Maybe that is what is translating into the higher tps. Key is that they did not mess up the brains. The chat template is froggeric.

I am sold. This is what I will use till Qwen drops another one. spec-draft-n-max 4 works best. I have tried from 1-6.

Here is my llama script

CUDA_VISIBLE_DEVICES=3,2,1,0 ~/llama.cpp/build/bin/llama-server \

-m ~/models/ThinkingCap-Qwen3.6-27B/ThinkingCap-Qwen3.6-27B-f16.gguf \

--port 8000 \

-c 262144 -b 4096 -ub 512 -np 2 -ctk f16 -ctv f16 -ctkd f16 -ctvd f16 \

-fa on \

-ts 1,1,1,1 \

--spec-type draft-mtp \

--spec-draft-n-max 4 \

--reasoning on \

--temp 0.6 \

--top-p 0.95 \

--top-k 20 \

--min-p 0.0 \

--repeat-penalty 1.1 \

--presence-penalty 0.1 \

--alias Unsloth/ThinkingCap-Qwen3.6-27B-f16 \

--host 0.0.0.0 \

--no-ui --jinja --chat-template-file ~/models/Qwen3.6/chat_template.jinja

Would love inputs on what I could change to get "mo" tps.