How We Optimized the Qwen 3.6 Model for Our Junie Agent

A while ago, we launched a long-term project to enable users to run Junie entirely locally, with local inference, across a wide variety of hardware setups. After much anticipation, we recently released an initial version of Junie Local that works on a MacBook M5 with Qwen3.6-27B.

In this blog post, I will share what it took to make it work and why we chose to release it on Qwen3.6-27B rather than Qwen3.8-27B. We made optimizations across the whole stack, from the Junie agent itself to our chosen inference engine.

Let’s start with Junie – after all, this is where you will begin interacting with the local model.

Junie optimizations

Extending the agent’s rolling context

Like any other coding agent, Junie has a main execution loop, where all its work is performed:

First, the user specifies a task.
Then, Junie sends it to the LLM.
Then, the LLM responds with some tool calls (like Bash commands, instructions to read/write files, and more).
Finally, Junie sends the result of this command back.

Here is a very simplified flow chart of what is happening under the hood:

As you can see from the chart, the LLM continues to receive context-expanding requests, allowing it to partially reuse information from those it has already handled. To be more specific, this means we can reuse the prefill data from the previous request for the next, and this data is referred to as the KV-cache.

But when we ask Junie to do the second task in the same session, it takes only relevant pieces from the context and puts them in the window:

With cloud models, this usually works perfectly because even if the model needs the content of some files again, it will request access and process them again. Prefill is really fast, too.

For local models, that is not the case – prefill isn’t that fast, and it actually takes significant time to “read” files.

To solve this problem, we changed the logic for local inference. Now, we add every new request directly to the rolling context:

This way, we can re-use KV caches from the previous task, i.e. if the model has already read a file, it will stay in the context window, and we don’t need to “read” it again.

Maximizing initial reusable prefix

Another similar optimization we have made relates to the system prompt and initial context that Junie sends when starting a new coding session.

In the previous section, the flow was somewhat simplified: Upon the first LLM request, much more data is actually sent to the LLM than was shown in the chart:

As you can see, a whole different set of information is sent to the LLM. And all this information is sent and processed at the start of every new session. Naturally, we would prefer to be able to cache it somehow 🙂

With this in mind, we changed the order in which we send this data:

We also added special logic to the inference engine to cache the prefix all the way up to the user’s request, so for subsequent tasks (in the same project), it is simply reused. We left out any project context after the user’s request because it is quite small and mostly consists of top-level project files, so it can change frequently.

Getting progress updates to work

With more powerful cloud models, Junie asks the LLM to add a special block in an XML-like format with updates that will be shown to the user. Unfortunately, Qwen 3.6 mostly ignores such requests. At the same time, the model writes its actions as plain text as part of the result of an LLM request – i.e. the LLM sends some tool calls with some accompanying text explaining them.

So, the fix is simple – just use this text generated by Qwen 3.6 as an update for the user. Such adaptations are model-specific, that is to say that we were lucky that Qwen 3.6 behaves in this way – some models don’t print anything there, and others generate far too much text.

Removing unnecessary calls to the LLM

This next optimization, disabling all optional LLM requests, might seem trivial, but it really helped make the agent more efficient. In practical terms, this meant that we disabled any logic that produced a short description of the task. While it’s true that we sacrificed some UX by doing so, we didn’t consider this loss excessive. In addition, we completely disabled multi-agent mode, as the most efficient way to process LLM requests on an M5 is via sequential processing, so there is no point in enabling multiple agents – they will be bottlenecked by the inference anyway.

Model parameter optimizations

reasonning_effort: None

When we were testing the cloud version of the Qwen3.6-27B internally, we noticed that enabling reasoning does not add a significant quality boost. So, for the local version, we decided to completely disable reasoning. This is a big deal, as reasoning tokens used by the inference engine perspective are the same as tokens used to generate the main response. Therefore, with reasoning disabled, we need to generate 2–3x fewer tokens, and that translates to a 2x speed-up on task execution, with an insignificant effect on quality.

Quantization

We decided to use the 4-bit version because it performs only slightly worse on benchmarks than its 8-bit counterpart, and because generation is memory-bottlenecked, using the 4-bit version is ~2x faster than the 8-bit version. However, when we compared the prefill speed between the 8-bit and 4-bit versions, we noticed they were the same… We thought that this was odd, so we dug deeper.

Inference engine optimization

Prefill hack

Some might wonder why we’re even concerned about prefill anyway. After all, the whole internet is full of generation speed benchmarks and optimization options.

Well, on a discrete GPU (like the RTX 5090), they might be right to question its importance. On that kind of hardware, it is indeed extremely fast because prefill is compute-bound, and discrete GPUs are usually quite powerful. This means you can get something like 3,700 t/s prefill speed on default configurations. On the M5 out of the box, it was somewhere in the neighborhood of 650 t/s. So when the model was requesting the content of the file, i.e. when investigating, the majority of time was spent on prefill, not generation!

What is worse is that there was no difference between 4-bit, 8-bit, or 16-bit quantization in terms of prefill speed. But why? Well, prefill is compute-bound, which means we are not limited by the memory speed at all. And it turns out that the majority of matrix operations during prefill were performed in full 16-bit mode. i.e. all 4-bit weights were converted to 16-bit numbers before operations were performed. But the M5 processor has special operations for 8-bit numbers that are significantly faster than 16-bit operations. So, when we applied a patch to the MLX-VLM package that switched some^* matrix operations during prefill to 8-bit, we got a ~40% prefill speed gain!

By the way, this is the main reason why we decided to focus on M5 chips. M4 chips don’t have these 8-bit arithmetic instructions, and the M4’s 16-bit arithmetic delivers 20–30% slower prefill.

^*Qwen3.6-27B uses both full-attention layers and self-attention layers. We found that even under 4-bit quantization, the full-attention weights remain stored in full 16-bit precision. Since these layers stay at full precision regardless, we didn’t apply this optimization to them – it’s applied only to the self-attention layers, where it actually reduces memory/compute. Here is the link to the patch in MLX-VLM. What’s more, the same optimization can also be applied in vLLM, just by editing the model’s config file. Config example

Speculative decoding via MTP and n-gram matching

As standard optimizations, we applied:

MTP (Multi-Token Prediction) with a separate draft model: A speculative decoding approach where a smaller draft model proposes several tokens ahead, which the main model then verifies.
N-gram speculative decoding: Instead of a draft model, this method looks for previously repeated token sequences in the context and “predicts” upcoming tokens by matching against them.

We enabled both methods simultaneously. In practice, this means that during generation, we sometimes accept not just the ~3 tokens proposed by MTP, but also up to 8 additional tokens accepted from the n-gram method. The illustration below shows raw-generated tokens color-coded by the method that produced each accepted token (draft model vs. n-gram).

Combined, this gives up to a 2x speedup in generation.

Qwen3.8-27b

Given all this, why didn’t we use 3.8 instead of 3.6?

Unfortunately, Qwen 3.8 requires reasoning mode to be enabled in order to function well. Without it, output quality degrades significantly – on typical tasks, it can even fail entirely, getting stuck in a loop where it repeats the same tool call indefinitely. But enabling reasoning mode substantially increases the number of generated tokens: At medium reasoning effort, roughly 5x more tokens are produced. Since prefill time stays roughly constant, the net slowdown is closer to 4x rather than a full 5x. That’s still a significant cost – which is why, for now, on Mac hardware, Qwen3.6-27b remains the better choice.

Closing thoughts

I hope that after reading about our journey, you can see that focusing solely on generation t/s is the wrong approach when it comes to typical agent programming tasks.

You need to optimize all parts of the stack, including:

Generation and prefill: The token-by-token decoding process and the initial context-processing pass.
Model parameters and quantization: The model’s weight precision and configuration.
Agent harness: The surrounding orchestration layer (tool calls, control flow, prompting logic) that drives the model.

So, this is what we are planning to do in the future. M5 support is just the first step – we already have prototypes for DGX Spark and RTX 5090 (and we’re even looking at 24 GB cards, too), so stay tuned!

Discover more

Fuente: Artículo original