---
language:
- en
license: apache-2.0
tags:
- gguf
- llama-cpp
- granite
- ibm
- reasoning
- tool-use
- quantized
- conversational
- on-device
pipeline_tag: text-generation
base_model: ibm-granite/granite-4.2-8b
---

# Granite 4.2 8B · GGUF Quantizations

Ready-to-run GGUF quantizations of **IBM Granite 4.2 8B** for local inference with **llama.cpp**, **Ollama**, **LM Studio**, **Open WebUI**, and other GGUF-compatible runtimes.

This repository provides a complete range of quantization formats, from near-lossless BF16 and Q8 models to highly compressed IQ variants for memory-constrained devices.

## About the Base Model

Granite 4.2 8B is IBM's dense reasoning language model designed for agentic workflows, coding, tool use, structured outputs, and enterprise-grade reasoning.

### Key Properties

| Property | Value |
|----------|--------|
| Base Model | ibm-granite/granite-4.2-8b |
| Parameters | ~8B |
| Architecture | GraniteForCausalLM |
| Context Window | 128K tokens |
| Reasoning Mode | Switchable thinking / non-thinking |
| Tool Calling | Supported |
| License | Apache 2.0 |
| Format | GGUF |

## Features

- 🧠 Native reasoning with optional `<think>` mode
- 🛠️ Tool calling and agent workflows
- 💻 Strong coding and software engineering capabilities
- 📊 Structured JSON generation
- ⚡ Compatible with llama.cpp and GGUF ecosystems
- 📱 Efficient local deployment on consumer hardware
- 🌍 Multilingual support

## Recommended Quantizations

| Quant | Recommendation |
|---------|---------------|
| Q6_K | Highest quality without full precision |
| Q5_K_M | Excellent balance of quality and VRAM |
| Q4_K_M | Default recommendation for most users |
| IQ4_XS | Compact deployment with good quality |
| Q3_K_M | Low-memory environments |
| Q2_K | Maximum compression |

## Usage with llama.cpp

```bash
./llama-cli \
  -m granite-4.2-8b-Q4_K_M.gguf \
  -c 8192 \
  -p "Explain transformer attention."
```

## Prompt Format

```text
<|im_start|>system
You are a helpful assistant.
<|im_end|>

<|im_start|>user
Explain transformer attention.
<|im_end|>

<|im_start|>assistant
```

## Hardware Guidance

| Quant | Approximate RAM / VRAM |
|---------|----------------------|
| Q2_K | 4 GB |
| Q3_K_M | 5 GB |
| Q4_K_M | 6 GB |
| Q5_K_M | 7 GB |
| Q6_K | 9 GB |
| Q8_0 | 11 GB |
| BF16 | 18+ GB |

Actual memory consumption depends on context length, batch size, and runtime configuration.

## Intended Use

This model is suitable for:

- Conversational AI
- Coding assistants
- Software engineering agents
- Retrieval-Augmented Generation (RAG)
- Local AI deployments
- Research and experimentation
- Tool-calling workflows

## Limitations

- Quantization may slightly reduce model quality compared to the original BF16 checkpoint.
- Smaller quantizations prioritize memory efficiency over accuracy.
- Model outputs may contain inaccuracies or hallucinations.
- Performance depends heavily on the selected quantization level and hardware.

## License

This repository distributes quantized versions of **IBM Granite 4.2 8B**.

The original model is licensed under **Apache 2.0**. Please refer to the upstream model card for complete licensing and usage information.

## Acknowledgements

- IBM Granite Team for the original Granite 4.2 model.
- llama.cpp contributors for GGUF support and quantization tooling.
- Hugging Face for model hosting and distribution.

## Citation

If you use these quantizations in research or production, please cite the original Granite 4.2 model from IBM.