Instructions to use EschaLabs/Qwen3.6-35B-A3B-Escha-W2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use EschaLabs/Qwen3.6-35B-A3B-Escha-W2 with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("EschaLabs/Qwen3.6-35B-A3B-Escha-W2")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use EschaLabs/Qwen3.6-35B-A3B-Escha-W2 with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "EschaLabs/Qwen3.6-35B-A3B-Escha-W2"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "EschaLabs/Qwen3.6-35B-A3B-Escha-W2"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

MLX LM

How to use EschaLabs/Qwen3.6-35B-A3B-Escha-W2 with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "EschaLabs/Qwen3.6-35B-A3B-Escha-W2"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "EschaLabs/Qwen3.6-35B-A3B-Escha-W2"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "EschaLabs/Qwen3.6-35B-A3B-Escha-W2",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Hermes Agent

How to use EschaLabs/Qwen3.6-35B-A3B-Escha-W2 with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "EschaLabs/Qwen3.6-35B-A3B-Escha-W2"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default EschaLabs/Qwen3.6-35B-A3B-Escha-W2

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use EschaLabs/Qwen3.6-35B-A3B-Escha-W2 with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "EschaLabs/Qwen3.6-35B-A3B-Escha-W2"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "EschaLabs/Qwen3.6-35B-A3B-Escha-W2" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Fix: TP+EP produces silently corrupted outputs (2 bugs in qwen3_5.py load_weights)

by juanginer - opened Aug 16

Discussion

juanginer

Aug 16

Hi Escha team,

Running Qwen3.6-35B-A3B-Escha-W2 on 2×RTX 3090 with --tp-size 2 --ep-size 2 produced silently corrupted outputs (arithmetic drift, degenerate loops with non-English fragments, <think> blocks that never close). --dp-size 2 was fine, and DP=1 baseline was fine.

My motivation for chasing this: I care more about total KV cache pool than raw throughput. DP=N replicates the model per rank and carves KV pool out of the remaining VRAM per replica — great for concurrency, but the max context I can serve per request is bound by one GPU's leftover. TP+EP distributes the model across ranks so ~half the VRAM per GPU is freed for KV, and the pool from all ranks is effectively shared for long-context requests. On 2×3090 this went from ~340 K tokens per DP replica to 1.36 M tokens aggregated under TP=2+EP=2 — decisive for agentic / long-context workloads.

Bisecting the runtime narrowed the silent corruption to two bugs in sglang/srt/models/qwen3_5.py (the sglang fork inside escha-1.0.2+qwen3moe.whl):

eschamoe checkpoint loader ignores the EP rank offset. The manual eschamoe branch in load_weights falls into a shape-mismatch else that was written for mixed-bit K3-vs-K2 last-dim cases and does _qp[_pn].data = _w.to(...) without slicing on dim 0. Under ep_size>1 both ranks end up with experts [0:num_local); experts [num_local:num_experts) are never loaded on any rank. My mask (topk_w=0 for topk_ids<0) turned the missing contributions into silent zeros, so 4/8 top-k slots per token contributed nothing — visible as consistent ~5-15 % numerical drift.
SGLANG_INT8_EMBED install ignores TP sharding. The embed_tokens int8 branch registers the full-vocab checkpoint tensor as a buffer directly into a VocabParallelEmbedding whose .weight was preallocated as [vocab/tp, hidden]. Every rank then indexes a full-vocab table with local row indices → garbage embeddings from step 0 → degenerate output.

Both are drop-in fixable in Python without touching the compiled kernels. I published the patch, a bisect log, and full-runtime reproduction steps at:

https://github.com/ginerJuanUdesa/escha-tp-fix

Verified on 2×3090: post-fix TP=2+EP=2 matches DP=1 greedy tokens; ~5× larger total KV pool at ~10 % single-request latency cost (all-reduce over PCIe, no NVLink here). Full stack works: attn/GDN/shared-expert/embed/lm_head TP-sharded, routed MoE EP-sharded.

Happy to open a PR against a public source tree if one exists — I could only find the precompiled wheel.

Thanks for the runtime — this is by far the fastest way to serve the 35B on consumer 3090s.

yzhwang

Escha Labs org Aug 17

Thanks for this! Confirmed both bugs in the shipped source.

One warning for anyone applying the patch: it isn't sufficient on a stock install. Under ep_size>1 sglang rewrites topk_ids to -1 for experts on the other rank, and our MoE kernel corrupts its routing table on those. Your topk_w=0 for topk_ids<0 mask is what keeps your build out of that, and it isn't in the diff. Worth adding to the repo.

We also found pure TP (ep_size=1) is silently wrong the same way, so we'll add a guard rather than let it run.

On the PR offer: the sglang runtime isn't open sourced, so there's no tree to send it to, sorry. We're folding your fix into the next wheel along with the -1 remap and the guard.

We now only have single GPU setups, so if you'd run a release candidate on your 2x3090 against your DP=1 baseline, that's the gate for shipping. We will credit your findings in our release note either way. Thanks again!

juanginer

Aug 17

Amazing!! Thanks for the quick and friendly response! I am currently using this model as my local daily driver so I am happy to help to make it better in any way I can! Let me know if u will make versions of 3.6-27 and 3.8-27, would love to test and add support for both if u need someone with 2 GPUs to do it.

yzhwang

Escha Labs org Aug 18

We are working hard on 3.8-27B release. Also we have decided to opensource the whole runtime stack soon (because to make it the best, we really need the whole community). So hopefully by next week your stuff will be just merged in as PR. Stay tuned!

juanginer

Aug 18

Amazing sounds like a plan! Good luck!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment