Grunder/kev-0.8b-conversation-vision

Kev-0.8B Conversation + Vision

Local conversation decisions with adapted turn-taking and pause heads, plus a restored Qwen visual input path.

This community release was developed for ZoE / Grunder and is published by Grunder Labs, maintained by StealthySalamander47. It extends Jared Palmer's Kev-0.8B with two stages of actual LoRA and pointer-head fine-tuning for English/German conversation control. It also reconnects the original Qwen3.5 vision encoder and merger/projector to the same language backbone, allowing Kev to classify real image content together with a supplied state and question.

Kev selects among supplied options and returns probability distributions. It does not generate the assistant's spoken answers, transcribe audio, synthesize speech, or independently provide a complete voice-call application.

Release: 1.0.0 · Date: 2026-10-08 · Runtime: Python 3.12, PyTorch, NVIDIA CUDA · No additional vision training

At a glance

Capability	Implementation	Evidence in this release
Conversational intent	Adapted LoRA + pointer head; respond, redirect, continue, listen, quiet, away, unfinished, resume	97/120 held-out conversation scenarios
Respecting the conversational floor	Separately selected pause adapter/head	64/64 authored pause holdout scenarios
Visual decisions	Original frozen Qwen vision encoder/projector feeding Kev's shared backbone	12/12 synthetic color/shape decisions; 13/13 questions over five photographs
Image-conditioned conversation	Text intent first, image evidence for explicit visual conditions	14/14 explicit visual conditions in the final integration replay
Sustained application operation	Historical text-only integration soak, including answer-model calls	7,500 seconds; 806 inputs; 996/1,028 checks passed

These are local, task-specific measurements, not a claim of general visual understanding or production-wide accuracy. The visual integration sets were revisited during development. The conversation holdout and sustained test retain known failures. Full scope, denominators, raw results and limitations appear below and in BENCHMARKS.md.

Architecture

Text state + supplied questions ───────────────────────┐
                                                     v
Images → frozen Qwen vision encoder → merger → ONE shared Qwen text backbone
                                                     |
                         selected LoRA + pointer head + fitted temperature
                                                     |
                                      choices and probability distributions

Three small adapters share one resident language backbone:

Route	Checkpoint directory	Temperature
Exact conversation question contract	models/kev-0.8b-conversation-candidate	4.4139146805
Exact pause/floor question contract	models/kev-0.8b-conversation-local	4.3553471565
Other questions, including general visual classification	models/kev-0.8b — unchanged upstream adapter/head	2.3510958126

The historical directory names are retained so the evaluated adapter selection and integrity checks continue to work. Keep all three directories. Loading only the final-stage adapter with a generic PEFT loader does not reproduce this composite runtime. Changing a question contract also changes which adapter the runtime selects; use the supplied constants for the adapted tasks.

Quick start

Download this repository to a writable directory. The example below pins the release tag:

hf download Grunder/kev-0.8b-conversation-vision --revision v1.0.0 --local-dir kev-conversation-vision
cd kev-conversation-vision
python -m venv .venv

Activate the environment (.venv\Scripts\activate on Windows; source .venv/bin/activate on Linux), then install the tested CUDA stack and runtime dependencies:

python -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r requirements.txt
python Scripts/prepare-base.py
python examples/classify.py --check-only
python examples/classify.py --text "Zoe, please stop talking." --speaking --device cuda:0

prepare-base.py downloads the pinned original Qwen base and verifies its hashes. The base contains both language and original vision weights and is intentionally not duplicated in this repository. It is about 1.75 GB for the main weight shard. Subsequent model operation is offline. CUDA is required by the supplied worker; a CPU fallback is not implemented in this release.

A generic visual decision:

python examples/classify.py --image path/to/photo.jpg --question "What is the main subject in the image?" --options cat dog car other

The vision tower loads on the first explicit image request. Up to four local PNG/JPEG/WebP/BMP images are accepted. Filenames and paths are not inserted into the model's state. See examples/classify.py, examples/request.json and VISION.md for the API and preprocessing details.

For a long-running host, keep one KevClient alive and reuse it. examples/classify.py is a short CLI demonstration which starts and closes its own worker. Do not repeatedly invoke it for every turn in a real-time application.

What was fine-tuned, and how?

The Qwen base remained frozen. The local training updated 10,822,656 LoRA parameters and Kev's pointer readout head, initialized from the released Kev checkpoint. No Qwen language-model output head, image encoder, or projector was trained.

Local stage	Training records	Calibration records	Epochs	Learning rate	Recorded training + calibration time
Conversation adaptation	872	184	2	2e-5	1,927.94 s
Pause/floor adaptation with conversation replay	888	232	1	1e-5	516.59 s

Common settings: rank 16, alpha 32, LoRA dropout 0.05, AdamW, weight decay 0.01, accumulation 8, gradient clipping 1.0, seed 4104, frozen BF16 backbone and FP32 trainable LoRA parameters. Loss is the mean cross-entropy over each record's question options. Training used a local RTX 3090 workflow; total recorded stage time is about 40 minutes 45 seconds, excluding environment setup and initial loading/encoding.

Training examples are authored synthetic utterances and conversation states, primarily English/German. Splits are made by utterance before state augmentation. Stage 2 contains 616 conversation records (400 replayed plus 216 new) and 272 pause records. Calibration contains the 184 conversation records plus 48 pause records. These counts describe records, not independent speakers or audio recordings.

A scalar temperature was fitted by minimizing calibration negative log-likelihood. Conversation-stage NLL changed from 0.8724 to 0.3243; second-stage NLL from 0.7327 to 0.2798. Those reductions are measured on the fitting data and are not independent calibration results. Visual probabilities were not calibrated on images.

The original trainer, data generators, train/calibration JSONL, exact question contracts, reports and hashes are included. See TRAINING.md for the full method and reproduction commands.

Vision: what was restored

The visual input path is a runtime integration, not a newly trained “vision head.” It restores the original pinned Qwen3.5 vision tower and merger/projector, shares the existing language backbone, inserts image placeholders into Kev's state prefix, preserves pointer indices, applies the multimodal rotary positions, and copies both attention and DeltaNet cache state for isolated question branches.

The small diagnostic sets show that it can distinguish colors/shapes, identify photographed objects and gestures, count the visible cats/hands in the tested images, and distinguish the first and second image. Cache invalidation also passed checks for changed bytes at the same path and reversed image order. Cached and full-row calculations selected the same answers in the recorded checks.

The conversation integration keeps text intent authoritative for direct stops, listening feedback and unfinished turns. For a complex image-conditioned request, the original application used a separate answer LLM as a text-only condition parser, then let Kev inspect the actual images. Its 14/14 result therefore measures that integration, not standalone Kev language parsing. This repository includes the decision helpers and original integration harnesses; it does not bundle or automatically start that answer LLM.

Details, measured image-path latency, cache invariants and remaining limitations: VISION.md.

Evaluation

Evaluation	Result	Interpretation
Conversation development	100/104 — 96.15%	Used during development/selection
Conversation holdout	97/120 — 80.83%	Authored synthetic cases; action and interruption must both match
Pause holdout	64/64 — 100%	Small authored binary floor-state panel
Color/shape diagnostic	12/12	Six synthetic images, two questions each
Photo diagnostic	13/13	Five public example photographs; small, non-independent question set
Final visual integration replay	81/85 — 95.29%	Development/challenge/release panels; revisited during development
Explicit image-conditioned cases within that replay	14/14	Includes the external text-only condition parser
Rated pause cases within that replay	24/24	Includes deterministic activity/quiet guards
Subsequent continuous visual dialogue	8/8 expected actions	One authored dialogue; answer model used separately
Text-only sustained integration run	996/1,028 — 96.89%	125 minutes, 806 inputs, no ASR or TTS calls

In the sustained run, Kev decisions took 390 ms median / 859 ms p95. The application closed 67/67 targeted output stops, with 234 ms median / 516 ms p95 / 563 ms maximum in the simulated text playback setup. The run produced 24 autonomous contributions and recorded no runtime errors, while retaining 32 semantic/check failures. These are not physical audio-stop latency measurements.

Do not combine these rows into a single accuracy number: the units, test protocols and dependencies differ. Full raw scenario reports and the recorded two-hour summary are under benchmarks/. BENCHMARKS.md explains which tests are reproducible from this model-only package and which require the original application harness.

Known limits

• Some unfinished sentences, temporary-away requests, resumes, quoted/negated stops and task replacements remain misclassified. The holdout and sustained-run failures are included.
• Images do not make the text-trained turn adapter universally more accurate. Sending images through every turn decision caused regressions during development; the final integration uses selective image evaluation.
• Visual probabilities use a text-fitted temperature and are not calibrated visual confidence. No broad OCR, spatial-reasoning, robustness or open-domain vision benchmark was run.
• The supplied default is an 8,192-token state budget plus a 2,048-token question allowance. Image tokens consume the state budget. Local adaptation training used a 2,048-token state cap; this is not evidence of reliable conversation decisions at every supported length.
• Validation used synthetic conversations and public/sample images. It does not establish diarization, multi-user Discord behavior, microphone robustness, or end-to-end ASR/TTS performance.
• The adapter selects conversation actions; an application still owns scheduling, cancellation, playback acknowledgements, permission checks and call lifecycle. A quiet/away decision should not itself hang up a call.

Reproducibility and attribution

Component	Pinned source
Parent Kev checkpoint	jaredpalmer/kev-0.8b@9a45d25e
Frozen multimodal base	Qwen/Qwen3.5-0.8B-Base@dc7cdfe2
Vendored Kev library snapshot	jaredpalmer/kev@fe64b127

This is an independent community adaptation, not an official Kev, Qwen or TypeSafe release. Upstream model architecture and the original decision-model work are credited to their authors. Local changes are the conversational adaptations, calibration, restored visual path and integration helpers documented here.

Model weight files are byte-identical to the selected local checkpoints. provenance/source-files.json records their source hashes and the sanitization of report paths. provenance/release-files.json inventories the published files. Release preparation ran package/integrity tests without loading additional model instances; the inference results above are the retained historical measurements, not fresh publication-day reruns.

Apache-2.0; see LICENSE and NOTICE. Referenced third-party photographs are not redistributed; their URLs and content hashes are listed separately. See CITATION.cff for citation metadata.