litert-community

/

gemma-4-E2B-it-litert-lm

Like 486

Follow

LiteRT Community (FKA TFLite) 9.78k

Title: Compiling & running Gemma on MediaTek NPU 780 – where do I start?

Hey everyone,

I'm trying to get Gemma running on-device with hardware acceleration on the MediaTek NPU 780 (Dimensity 8300/8350), but I'm fairly new to the MediaTek toolchain and looking for some direction.

I'm aiming to offload as much of the model to the NPU as possible, but I’m a bit lost on the actual compilation and deployment pipeline.

A few questions for anyone who has worked with this NPU:

1. Which workflow should I use?

   • Is it better to go through LiteRT (TFLite) with the NeuroPilot delegate, use ExecuTorch, or compile directly to .dla using the standalone NeuroPilot SDK?

2. Quantization & Support:

   • Does NPU 780 handle INT4 weights smoothly for LLMs, or is INT8/FP16 the safer target?
   • If some operations aren't supported on the NPU, does it automatically fall back to CPU/GPU, or will the compile step just fail?

3. KV Cache & Shapes:

   • Do I need to compile with fixed, bucketed sequence lengths (like 512, 1024), or does the runtime support dynamic context lengths well?

If anyone has a working example repo, export script, or general pointers to get started, I’d really appreciate the help!

Thanks!