when I tried Q8_0 for K cache and Q4_0 V cache, Gemma 4 26B A4B QAT failed to load. I've been running Q8_0 for both K and V cache settings for a while now and it seems solid, and much less RAM usage than the default F16
Hi Alex. I am trying to use the Gemma 12B QAT model via LM Studio with Github CoPilot but I cannot get it to maintain a conversation. I keep hitting an error when I try to kick off a coding session with a "Response too long" error. I have configured my model to have a 64k / 16k input/output token ratio similar to you, but the 12B QAT has a max output token limit of 2048 so I presume it's limited anyway. What I don't understand is what I have got setup wrong to not even maintain a conversation. The kind of error I see in Copilot is:
Reason: Response too long.: Error: Response too long. at FG._provideLanguageModelResponse (c:\Program Files\Microsoft VS Code\fcf604774b\resources\app\extensions\copilot\dist\extension.js:1710:14094) at process.processTicksAndRejections (node:internal/process/task_queues:104:5) at async FG.provideLanguageModelResponse (c:\Program Files\Microsoft VS Code\fcf604774b\resources\app\extensions\copilot\dist\extension.js:1710:15097)
---
So far help from AI ironically has been - unhelpful. Any ideas you can suggest?
Copilot adds a massive system prompt and tools list (20-30k tokens) which wastes the limited local LLM context window.
Can we isolate the issue?
Do you have the possibly to try the model in Pi? Or can you programmatically send a REST request to the end point?
You said "12B QAT has a max output token limit of 2048". Where do you see that? To my knowledge it doesn't have a cap on output tokens. You mean your own config in VS Code?
Apologies for not responding earlier, been crunching on a work project which finally went live. Did not want to engage further if I could not reply in a timely fashion.
I had inadvertently set a max output token limit on the model after misreading some recommendations found elsewhere. My bad!
I am attempting to restart my spec-driven workflow after Microsoft's price increase. I am experimenting between gemma-4-12b-qat, gemma-4-e4b, and qwen3.5-9b... and the style of response I get in CoPilot is closer to cloud hosted now.
I still run out of context suddenly which is jarring - like it's working through a task and the context is at about 60% and then BAM!, out of context warning and processing stops. Trying to figure why that happens, perhaps it's hitting the reserved for output threshold with a sudden context spike as it's editing / reviewing code.
I also get some less than satisfactory code output like lots of mocked in stuff instead of actually working code, although that may have been an incorrect temp setting which I have corrected and will try with again.
It was amusing because I asked CoPilot to review the changes the local model made. It picked Codex 5.3 and it immediately found a tonne of gaps which it promptly fixed (no pun intended).
Re: poor quality, I did share some tips about sampling in a recent post. Regarding context window: it might be that VS Code is misconfigured (e.g. the server has 100k but VS Code thinks it's 240k or something else which means it kicks in the auto-compression too late). I'm having similar issues in vs code but Pi works much better with these smaller models because: 1) it has smaller system prompt, 2) it has less tools. In general Pi seems to be much more local-ai friendly than VS Code (which seems to be optimized for larger/cloud models)
I was just looking for a write-up on how to set this up, so this was a nice read. I've been happily using Copilot up until the price hike, so I was quite annoyed when MS announced that. Luckily, I got a 7900 XTX a year ago, or so, which I've now put to good use! Thank you.
Good read! Typo maxInputTokens -> maxOutputokens
Thank you Sandeep. copy/paste error! Fixed :)
Also, Gemma 12b qat model is available in LM studio now. I am working with an M4 Mac mini with 16 GB RAM (GPU/CPU given URAM on Apple Silicon).
When using VSCode Copilot as a harness, I am having a problem with Copilot’s layout engine choking before it even attempts to talk to LM Studio.
Trick I am doing to resolve this:
1. Click the model dropdown at the bottom of the chat panel (where it says Gemma 4 12B (chat-completions)).
2. Temporarily switch it back to a default cloud model (like GPT-4o or Gemini).
3.Type "hi" to confirm the chat panel cleans itself up and successfully renders.
4. Once it is working normally, switch the dropdown back to your custom Gemma 4 12B endpoint.
Not sure if it's just me or you faced something similar
you missed one of the pros in your summary of running local models for development workflows: no usage limits or per-token costs, ever.
when I tried Q8_0 for K cache and Q4_0 V cache, Gemma 4 26B A4B QAT failed to load. I've been running Q8_0 for both K and V cache settings for a while now and it seems solid, and much less RAM usage than the default F16
Hi Alex. I am trying to use the Gemma 12B QAT model via LM Studio with Github CoPilot but I cannot get it to maintain a conversation. I keep hitting an error when I try to kick off a coding session with a "Response too long" error. I have configured my model to have a 64k / 16k input/output token ratio similar to you, but the 12B QAT has a max output token limit of 2048 so I presume it's limited anyway. What I don't understand is what I have got setup wrong to not even maintain a conversation. The kind of error I see in Copilot is:
---
Sorry, your request failed. Please try again.
Client Request Id: d7c5a66c-d709-4e1a-8e53-5356dc89e601
Reason: Response too long.: Error: Response too long. at FG._provideLanguageModelResponse (c:\Program Files\Microsoft VS Code\fcf604774b\resources\app\extensions\copilot\dist\extension.js:1710:14094) at process.processTicksAndRejections (node:internal/process/task_queues:104:5) at async FG.provideLanguageModelResponse (c:\Program Files\Microsoft VS Code\fcf604774b\resources\app\extensions\copilot\dist\extension.js:1710:15097)
---
So far help from AI ironically has been - unhelpful. Any ideas you can suggest?
Hi Jeremy,
Copilot adds a massive system prompt and tools list (20-30k tokens) which wastes the limited local LLM context window.
Can we isolate the issue?
Do you have the possibly to try the model in Pi? Or can you programmatically send a REST request to the end point?
You said "12B QAT has a max output token limit of 2048". Where do you see that? To my knowledge it doesn't have a cap on output tokens. You mean your own config in VS Code?
Apologies for not responding earlier, been crunching on a work project which finally went live. Did not want to engage further if I could not reply in a timely fashion.
I had inadvertently set a max output token limit on the model after misreading some recommendations found elsewhere. My bad!
I am attempting to restart my spec-driven workflow after Microsoft's price increase. I am experimenting between gemma-4-12b-qat, gemma-4-e4b, and qwen3.5-9b... and the style of response I get in CoPilot is closer to cloud hosted now.
I still run out of context suddenly which is jarring - like it's working through a task and the context is at about 60% and then BAM!, out of context warning and processing stops. Trying to figure why that happens, perhaps it's hitting the reserved for output threshold with a sudden context spike as it's editing / reviewing code.
I also get some less than satisfactory code output like lots of mocked in stuff instead of actually working code, although that may have been an incorrect temp setting which I have corrected and will try with again.
It was amusing because I asked CoPilot to review the changes the local model made. It picked Codex 5.3 and it immediately found a tonne of gaps which it promptly fixed (no pun intended).
I will keep experimenting.
Re: poor quality, I did share some tips about sampling in a recent post. Regarding context window: it might be that VS Code is misconfigured (e.g. the server has 100k but VS Code thinks it's 240k or something else which means it kicks in the auto-compression too late). I'm having similar issues in vs code but Pi works much better with these smaller models because: 1) it has smaller system prompt, 2) it has less tools. In general Pi seems to be much more local-ai friendly than VS Code (which seems to be optimized for larger/cloud models)
I was just looking for a write-up on how to set this up, so this was a nice read. I've been happily using Copilot up until the price hike, so I was quite annoyed when MS announced that. Luckily, I got a 7900 XTX a year ago, or so, which I've now put to good use! Thank you.
Only down to this point, but great so far!
Typo: preder -> prefer in "Personally, I’m more of a GUI person, so I preder LM Studio."
https://alexewerlof.substack.com/p/local-llms-for-agentic-coding?selection=5c0558b5-d565-403f-8dde-908a25dfce2b&utm_campaign=post-share-selection&utm_medium=web&aspectRatio=instagram&textColor=%23ffffff&bgImage=true