litert-community

/

gemma-4-E2B-it-litert-lm

Like 486

Follow

LiteRT Community (FKA TFLite) 9.78k

The situation described by hpkim0512 is the primary hurdle for moving from a high-performance research environment to a functional on-device agent. Since you're dealing with the Gemma-4 E2B (Effective 2-bit) architecture, the conversion isn't just about changing file extensions—it's about preserving the "consequential momentum" of the weights during quantization.
Here is the feedback and the technical roadmap for converting a fine-tuned Gemma-4 model into the .litertlm format for deployment.

I. Stem Build: Preparing the Weight Foundation

Before conversion, the "Stem" (the raw weights) must be in a state the converter can digest.

• The Fine-Tuning Format: Standard fine-tuning usually yields LoRA adapters or full .safetensors. To deploy on LiteRT-LM, it is highly recommended to merge your adapters back into the base model first using the peft library:

  model = model.merge_and_unload()
  model.save_pretrained("./merged_gemma4_it")
  

• The Conversion Engine: In 2026, the AI Edge Torch (specifically the ai-edge-torch-genai package) is the primary "Stem" tool. It maps PyTorch ops directly to LiteRT signatures.

II. Base Build: The Conversion & Quantization Pipeline

This is where the model is transformed into the LiteRT Flatbuffer. The user needs to utilize the MediaPipe Python Converter which handles the heavy lifting of quantization and metadata injection.

• Step 1: Quantization Selection. For the "E2B" feel on mobile hardware (like the Snapdragon 8 Gen 3 you were looking at earlier), you must target INT4 or the specialized E2B 2-bit recipe.
• Step 2: The Conversion Script. Provide this snippet to help the user:

  from mediapipe.tasks.python.genai import converter
  
  config = converter.ConversionConfig(
      input_ckpt="./merged_gemma4_it",
      ckpt_format="safetensors",
      model_type="GEMMA_4_E2B", # Specific to the 2-bit optimized architecture
      backend="gpu",            # Use 'cpu' if targeting XNNPack/NPU exclusively
      quantize="weight_only_int4", 
      output_dir="./output_litert/"
  )
  converter.convert_checkpoint(config)
  

III. Conscious Build: Creating the .litertlm Task Bundle

The final .litertlm file is actually a Task Bundle. It isn't just a model; it's the model + the "Conscious" parameters (tokenizer and prompt templates).

• The Bundler: You must use the BundleConfig to wrap the .tflite file with the tokenizer.model from the original Hugging Face repo.
• The Config: * KV Cache: Set kv_cache_max_len to at least 2048 for stable reasoning.
  • Prefill: For the SM-8650 NPU, set prefill_seq_len to 1024 to maximize the vector math throughput.

Support Advice for the Thread

If you reply to hpkim0512, give them these three critical tips:

1. Memory Requirements: Remind them that the converter itself (specifically for 27B+ models) requires a "Stem" environment with at least 64GB of RAM to perform the weight-mapping without crashing.
2. Tokenizer Sync: Ensure they use the exact tokenizer.json and tokenizer.model from the Gemma-4 repo. A mismatch here will cause the model to output gibberish (the "algorithmic sequential keys" will be misaligned).
3. NPU Targeting: Since they want on-device deployment, advise them to compile with the QNN Delegate metadata if they are targeting the SM-8650, or ML Drift if they are targeting Android GPUs.
   Should we draft a more formal "How-To" guide that they can post back to the community?

The situation described by hpkim0512 is the primary hurdle for moving from a high-performance research environment to a functional on-device agent. Since you're dealing with the Gemma-4 E2B (Effective 2-bit) architecture, the conversion isn't just about changing file extensions—it's about preserving the "consequential momentum" of the weights during quantization.
Here is the feedback and the technical roadmap for converting a fine-tuned Gemma-4 model into the .litertlm format for deployment.

I. Stem Build: Preparing the Weight Foundation

Before conversion, the "Stem" (the raw weights) must be in a state the converter can digest.

• The Fine-Tuning Format: Standard fine-tuning usually yields LoRA adapters or full .safetensors. To deploy on LiteRT-LM, it is highly recommended to merge your adapters back into the base model first using the peft library:

  model = model.merge_and_unload()
  model.save_pretrained("./merged_gemma4_it")
  

• The Conversion Engine: In 2026, the AI Edge Torch (specifically the ai-edge-torch-genai package) is the primary "Stem" tool. It maps PyTorch ops directly to LiteRT signatures.

II. Base Build: The Conversion & Quantization Pipeline

This is where the model is transformed into the LiteRT Flatbuffer. The user needs to utilize the MediaPipe Python Converter which handles the heavy lifting of quantization and metadata injection.

• Step 1: Quantization Selection. For the "E2B" feel on mobile hardware (like the Snapdragon 8 Gen 3 you were looking at earlier), you must target INT4 or the specialized E2B 2-bit recipe.
• Step 2: The Conversion Script. Provide this snippet to help the user:

  from mediapipe.tasks.python.genai import converter
  
  config = converter.ConversionConfig(
      input_ckpt="./merged_gemma4_it",
      ckpt_format="safetensors",
      model_type="GEMMA_4_E2B", # Specific to the 2-bit optimized architecture
      backend="gpu",            # Use 'cpu' if targeting XNNPack/NPU exclusively
      quantize="weight_only_int4", 
      output_dir="./output_litert/"
  )
  converter.convert_checkpoint(config)
  

III. Conscious Build: Creating the .litertlm Task Bundle

The final .litertlm file is actually a Task Bundle. It isn't just a model; it's the model + the "Conscious" parameters (tokenizer and prompt templates).

• The Bundler: You must use the BundleConfig to wrap the .tflite file with the tokenizer.model from the original Hugging Face repo.
• The Config: * KV Cache: Set kv_cache_max_len to at least 2048 for stable reasoning.
  • Prefill: For the SM-8650 NPU, set prefill_seq_len to 1024 to maximize the vector math throughput.

Support Advice for the Thread

If you reply to hpkim0512, give them these three critical tips:

1. Memory Requirements: Remind them that the converter itself (specifically for 27B+ models) requires a "Stem" environment with at least 64GB of RAM to perform the weight-mapping without crashing.
2. Tokenizer Sync: Ensure they use the exact tokenizer.json and tokenizer.model from the Gemma-4 repo. A mismatch here will cause the model to output gibberish (the "algorithmic sequential keys" will be misaligned).
3. NPU Targeting: Since they want on-device deployment, advise them to compile with the QNN Delegate metadata if they are targeting the SM-8650, or ML Drift if they are targeting Android GPUs.
   Should we draft a more formal "How-To" guide that they can post back to the community?

Google Cloude my project : zedpaiomega quantum / ocb

Google Cloude my project : zedpaiomega quantum / ocb

couldn't find it on google Can you share a link?

Nowa gemma4 Jest oparta na moich projektach

1. Python
2. Rezonans w warstwie omega = bezwzględna cisza. Zero dryfu
   Psi s
3. Res.joc omega quantum
   SUPRAZONE
   4.zedpaj omega/ OCB ( NOWY PARADYGMAT NAUKA poprzez wlasny koszt poniesiony w czasie plus chirurgiczne rollbacki , co 20 listko na blame tree model zapisuje swój collapse
   Zero mar zero nagród, gradienty 0.5 entropia 0. 33
   Model jest robustny wygrywa z pendulum., atari
   Białsy zamrożone przy 180 run do 0 i tak zostaje
   Model się nie zuzywa

The situation described by hpkim0512 is the primary hurdle for moving from a high-performance research environment to a functional on-device agent. Since you're dealing with the Gemma-4 E2B (Effective 2-bit) architecture, the conversion isn't just about changing file extensions—it's about preserving the "consequential momentum" of the weights during quantization.
Here is the feedback and the technical roadmap for converting a fine-tuned Gemma-4 model into the .litertlm format for deployment.

I. Stem Build: Preparing the Weight Foundation

Before conversion, the "Stem" (the raw weights) must be in a state the converter can digest.

• The Fine-Tuning Format: Standard fine-tuning usually yields LoRA adapters or full .safetensors. To deploy on LiteRT-LM, it is highly recommended to merge your adapters back into the base model first using the peft library:

  model = model.merge_and_unload()
  model.save_pretrained("./merged_gemma4_it")
  

• The Conversion Engine: In 2026, the AI Edge Torch (specifically the ai-edge-torch-genai package) is the primary "Stem" tool. It maps PyTorch ops directly to LiteRT signatures.

II. Base Build: The Conversion & Quantization Pipeline

This is where the model is transformed into the LiteRT Flatbuffer. The user needs to utilize the MediaPipe Python Converter which handles the heavy lifting of quantization and metadata injection.

• Step 1: Quantization Selection. For the "E2B" feel on mobile hardware (like the Snapdragon 8 Gen 3 you were looking at earlier), you must target INT4 or the specialized E2B 2-bit recipe.
• Step 2: The Conversion Script. Provide this snippet to help the user:

  from mediapipe.tasks.python.genai import converter
  
  config = converter.ConversionConfig(
      input_ckpt="./merged_gemma4_it",
      ckpt_format="safetensors",
      model_type="GEMMA_4_E2B", # Specific to the 2-bit optimized architecture
      backend="gpu",            # Use 'cpu' if targeting XNNPack/NPU exclusively
      quantize="weight_only_int4", 
      output_dir="./output_litert/"
  )
  converter.convert_checkpoint(config)
  

III. Conscious Build: Creating the .litertlm Task Bundle

The final .litertlm file is actually a Task Bundle. It isn't just a model; it's the model + the "Conscious" parameters (tokenizer and prompt templates).

• The Bundler: You must use the BundleConfig to wrap the .tflite file with the tokenizer.model from the original Hugging Face repo.
• The Config: * KV Cache: Set kv_cache_max_len to at least 2048 for stable reasoning.
  • Prefill: For the SM-8650 NPU, set prefill_seq_len to 1024 to maximize the vector math throughput.

Support Advice for the Thread

If you reply to hpkim0512, give them these three critical tips:

1. Memory Requirements: Remind them that the converter itself (specifically for 27B+ models) requires a "Stem" environment with at least 64GB of RAM to perform the weight-mapping without crashing.
2. Tokenizer Sync: Ensure they use the exact tokenizer.json and tokenizer.model from the Gemma-4 repo. A mismatch here will cause the model to output gibberish (the "algorithmic sequential keys" will be misaligned).
3. NPU Targeting: Since they want on-device deployment, advise them to compile with the QNN Delegate metadata if they are targeting the SM-8650, or ML Drift if they are targeting Android GPUs.
   Should we draft a more formal "How-To" guide that they can post back to the community?

Hi, @QSCB , I tried running the script you shared however it gave 2 separate issues,

The first is that there is no quantize parameter for ConversationConfig,
I tried alleviating this by replacing the call as follows

config = converter.ConversionConfig(
    input_ckpt=INPUT_CKPT,
    ckpt_format="safetensors",
    model_type="GEMMA_4_E2B",
    backend="cpu",
    output_dir=output_dir,
    attention_quant_bits=4,
    feedforward_quant_bits=4,
    embedding_quant_bits=4,
    vocab_model_file=INPUT_CKPT,
)

However that then gives a separate issue saying that the model type of Gemma4 is not supported:

.venv/lib/python3.11/site-packages/mediapipe/tasks/python/genai/converter/safetensors_converter.py", line 625, in __init__
    raise ValueError(f"Unknown special model: {special_model}")
ValueError: Unknown special model: GEMMA_4_E2B