Datasets:

VIRUS374
/

ai-developer-dataset

messages listlengths 3 3	metadata dict
[ { "role": "system", "content": "Category: programming\nSubcategory: python_basics\nComplexity: beginner\nSource: handcrafted" }, { "role": "user", "content": "Объясни, что такое list comprehension в Python и покажи примеры." }, { "role": "assistant", "content": "List comprehension — ...	{ "category": "programming", "subcategory": "python_basics", "complexity": "beginner", "source": "handcrafted" }
[ { "role": "system", "content": "Category: programming\nSubcategory: python_basics\nComplexity: beginner\nSource: handcrafted" }, { "role": "user", "content": "Как работают args и kwargs в Python?" }, { "role": "assistant", "content": "`args` и `**kwargs` позволяют функции принима...	{ "category": "programming", "subcategory": "python_basics", "complexity": "beginner", "source": "handcrafted" }
[ { "role": "system", "content": "Category: programming\nSubcategory: python_advanced\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "Напиши асинхронный HTTP-клиент на aiohttp с ограничением параллелизма и retry-логикой." }, { "role": "assistant", "cont...	{ "category": "programming", "subcategory": "python_advanced", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: programming\nSubcategory: python_advanced\nComplexity: advanced\nSource: handcrafted" }, { "role": "user", "content": "Реализуй LRU-кэш на Python с поддержкой TTL без использования functools.lru_cache." }, { "role": "assistant", "content": "Р...	{ "category": "programming", "subcategory": "python_advanced", "complexity": "advanced", "source": "handcrafted" }
[ { "role": "system", "content": "Category: programming\nSubcategory: typescript\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "Напиши типобезопасный fetch-обёртку на TypeScript, которая валидирует ответ через Zod и бросает типизированные ошибки." }, { "ro...	{ "category": "programming", "subcategory": "typescript", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: programming\nSubcategory: bash\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "Напиши bash-скрипт, который мониторит лог-файл и шлёт уведомление в webhook при появлении ERROR." }, { "role": "assistant", "conte...	{ "category": "programming", "subcategory": "bash", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: programming\nSubcategory: sql\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "Есть таблицы users(id, name, created_at) и orders(id, user_id, amount, created_at). Найди топ-10 пользователей по сумме заказов за последние 30...	{ "category": "programming", "subcategory": "sql", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: programming\nSubcategory: go\nComplexity: advanced\nSource: handcrafted" }, { "role": "user", "content": "Напиши на Go HTTP-сервер с graceful shutdown, health-check и middleware-логированием." }, { "role": "assistant", "content": "Используем ...	{ "category": "programming", "subcategory": "go", "complexity": "advanced", "source": "handcrafted" }
[ { "role": "system", "content": "Category: programming\nSubcategory: code_review\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "Сделай code review этого Python-кода:\n\n```python\ndef get_user_data(id, db):\n cur = db.cursor()\n cur.execute(f'SELECT * FROM ...	{ "category": "programming", "subcategory": "code_review", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: programming\nSubcategory: refactoring\nComplexity: advanced\nSource: handcrafted" }, { "role": "user", "content": "Отрефактори этот код так, чтобы он был тестируемым и не зависел от глобального состояния:\n\n```python\nimport requests\nimport os\n\nAPI_U...	{ "category": "programming", "subcategory": "refactoring", "complexity": "advanced", "source": "handcrafted" }
[ { "role": "system", "content": "Category: web_development\nSubcategory: react\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "Напиши кастомный хук useDebounce на TypeScript и покажи пример использования в поисковой строке с запросом к API." }, { "role": "...	{ "category": "web_development", "subcategory": "react", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: web_development\nSubcategory: nextjs\nComplexity: advanced\nSource: handcrafted" }, { "role": "user", "content": "Создай Next.js 14 (App Router) страницу с server component для списка постов и client component для лайков с optimistic update." }, { ...	{ "category": "web_development", "subcategory": "nextjs", "complexity": "advanced", "source": "handcrafted" }
[ { "role": "system", "content": "Category: web_development\nSubcategory: tailwind\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "Напиши на Tailwind CSS красивую карточку товара для e-commerce с hover-эффектом, badge \"-20%\" и кнопкой \"В корзину\". Responsive." ...	{ "category": "web_development", "subcategory": "tailwind", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: web_development\nSubcategory: dashboard\nComplexity: advanced\nSource: handcrafted" }, { "role": "user", "content": "Создай React + TypeScript dashboard для управления серверами: список серверов с CPU/RAM/disk, фильтры по статусу, обновление в реальном в...	{ "category": "web_development", "subcategory": "dashboard", "complexity": "advanced", "source": "handcrafted" }
[ { "role": "system", "content": "Category: web_development\nSubcategory: websockets\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "Напиши WebSocket-сервер на Node.js + ws, который держит комнаты и бродкастит сообщения только внутри комнаты." }, { "role": ...	{ "category": "web_development", "subcategory": "websockets", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: web_development\nSubcategory: api_integration\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "Напиши React-компонент для загрузки файла на сервер с прогресс-баром, отменой и обработкой ошибок." }, { "role": "assis...	{ "category": "web_development", "subcategory": "api_integration", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: web_development\nSubcategory: landing_page\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "Сделай одностраничный landing для SaaS-продукта \"AI Code Reviewer\" с hero, фичами, ценами и CTA. Tailwind, responsive, тёмная те...	{ "category": "web_development", "subcategory": "landing_page", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: web_development\nSubcategory: forms_auth\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "Напиши на React + TypeScript форму логина с валидацией через react-hook-form + zod, показом серверных ошибок и disabled-состоянием к...	{ "category": "web_development", "subcategory": "forms_auth", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: web_development\nSubcategory: vite\nComplexity: beginner\nSource: handcrafted" }, { "role": "user", "content": "Объясни, что такое Vite и чем он лучше webpack. Покажи базовую настройку React + TypeScript проекта." }, { "role": "assistant", "c...	{ "category": "web_development", "subcategory": "vite", "complexity": "beginner", "source": "handcrafted" }
[ { "role": "system", "content": "Category: debugging\nSubcategory: python_exceptions\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "Мой код падает с ошибкой:\n\n```\nTraceback (most recent call last):\n File \"app.py\", line 23, in <module>\n user = users[0][...	{ "category": "debugging", "subcategory": "python_exceptions", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: debugging\nSubcategory: javascript_errors\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "React-компонент падает:\n\n```\nTypeError: Cannot read properties of undefined (reading 'map')\n at UserList (UserList.tsx:12:23...	{ "category": "debugging", "subcategory": "javascript_errors", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: debugging\nSubcategory: react_errors\nComplexity: advanced\nSource: handcrafted" }, { "role": "user", "content": "Бесконечный re-render. В консоли:\n\n```\nWarning: Maximum update depth exceeded. This can happen when a component calls setState inside com...	{ "category": "debugging", "subcategory": "react_errors", "complexity": "advanced", "source": "handcrafted" }
[ { "role": "system", "content": "Category: debugging\nSubcategory: npm_errors\nComplexity: intermediate\nSource: handcrafted" }, { "role": "user", "content": "При `npm install` получаю:\n\n```\nnpm error ERESOLVE unable to resolve dependency tree\nnpm error Found: react@18.2.0\nnpm error Could no...	{ "category": "debugging", "subcategory": "npm_errors", "complexity": "intermediate", "source": "handcrafted" }
[ { "role": "system", "content": "Category: debugging\nSubcategory: http_api_errors\nComplexity: beginner\nSource: handcrafted" }, { "role": "user", "content": "Мой fetch возвращает ошибку CORS:\n\n```\nAccess to fetch at 'https://api.example.com/data' from origin 'http://localhost:3000'\nhas been...	{ "category": "debugging", "subcategory": "http_api_errors", "complexity": "beginner", "source": "handcrafted" }

End of preview. Expand in Data Studio

AI Developer Dataset

A large-scale instruction-tuning dataset for fine-tuning an open-weight LLM into a universal AI developer assistant.

The model trained on this dataset should be especially good at: PROGRAMMING + WEB DEVELOPMENT + UI/UX + ANIMATIONS + BOTS + SCRIPTS + AUTOMATION + BACKEND + API + DATABASES + DEBUGGING + LINUX + DEPLOYMENT + AI DEVELOPMENT + SECURITY.

📊 Statistics

Total examples: 1,124,699
File size: 2.13 GB
Format: JSONL (conversational)
Average assistant length: ~1,222 characters
Schema: {"messages":[{"role":"system|user|assistant","content":"..."}, ...], "metadata":{"category","subcategory","complexity","source"}}

📚 Sources

This dataset is a curated merge of high-quality open-source code instruction datasets from HuggingFace, plus handcrafted examples in Russian covering UI/UX, security, and other specialized topics.

Source	Examples
`hf:glaiveai/glaive-code-assistant-v2`	213,670
`hf:glaiveai/glaive-code-assistant`	135,195
`hf:mwitiderrick/glaive-code-assistant`	104,336
`hf:b-mc2/sql-create-context`	78,573
`hf:TokenBender/code_instructions_122k_alpaca_style`	78,433
`hf:iamtarun/code_instructions_120k_alpaca`	78,433
`hf:ajibawa-2023/Code-74k-ShareGPT`	68,790
`hf:Locutusque/code-feedback-sharegpt`	62,866
`hf:m-a-p/Code-Feedback`	62,866
`hf:HuggingFaceH4/Code-Feedback`	61,923
`hf:WizardLMTeam/WizardLM_evol_instruct_70k`	60,411
`hf:nickrosh/Evol-Instruct-Code-80k-v1`	58,265
`hf:Data-Agora/magpie_code_gpt4o_mini_50000`	36,818
`hf:ajibawa-2023/Python-Code-23k-ShareGPT`	17,737
`hf:HuggingFaceH4/CodeAlpaca_20K`	6,307
`handcrafted`	76

📁 Categories

Category	Examples	%
programming	1,046,060	93.0%
backend_api_db	78,579	7.0%
ui_ux_design	10	0.0%
web_development	9	0.0%
bots_automation	8	0.0%
security	8	0.0%
ai_ml	7	0.0%
debugging	6	0.0%
linux_server_deploy	6	0.0%
web_search_docs	6	0.0%

🎯 Complexity Distribution

Level	Examples	%
intermediate	686,519	61.0%
advanced	438,166	39.0%
beginner	12	0.0%
expert	2	0.0%

📋 Schema

Each line is a single JSON object:

{
  "messages": [
    {"role": "system", "content": "Category: programming\nSubcategory: code_alpaca\n..."},
    {"role": "user", "content": "Write a Python function to..."},
    {"role": "assistant", "content": "Here is the implementation..."}
  ],
  "metadata": {
    "category": "programming",
    "subcategory": "code_alpaca",
    "complexity": "intermediate",
    "source": "hf:HuggingFaceH4/CodeAlpaca_20K"
  }
}

🛠️ Quality Control

Every example passes:

JSON-serializable (validated on 100% of examples)
Non-empty user and assistant content
Assistant length between 200 and 25,000 characters
Per-source deduplication by user-content fingerprint (SHA-256, first 16 hex chars)
Schema validation (messages + metadata required)

🚀 Usage

from datasets import load_dataset

# Load the full dataset
ds = load_dataset("VIRUS374/ai-developer-dataset", data_files="dataset.jsonl", split="train")
print(ds)
# Dataset({
#     features: ["messages", "metadata"],
#     num_rows: 1,124,699
# })

# Or use streaming for memory-efficient iteration
ds = load_dataset("VIRUS374/ai-developer-dataset", data_files="dataset.jsonl", split="train", streaming=True)
for example in ds:
    messages = example["messages"]
    # ... feed to tokenizer with chat template

🏋️ Fine-tuning Recipe (QLoRA)

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
from datasets import load_dataset
import torch

MODEL = "meta-llama/Llama-3.1-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(MODEL)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL,
    torch_dtype=torch.bfloat16,
    load_in_4bit=True,
    device_map="auto",
)
model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16, lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)

ds = load_dataset("VIRUS374/ai-developer-dataset", data_files="dataset.jsonl", split="train")

def format_example(example):
    text = tokenizer.apply_chat_template(
        example["messages"], tokenize=False, add_generation_prompt=False,
    )
    return {"text": text}

ds = ds.map(format_example, remove_columns=ds.column_names)

trainer = SFTTrainer(
    model=model,
    train_dataset=ds,
    args=TrainingArguments(
        output_dir="./output",
        num_train_epochs=3,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,
        warmup_steps=50,
        learning_rate=2e-4,
        bf16=True,
        logging_steps=10,
        save_steps=500,
    ),
    max_seq_length=4096,
)
trainer.train()

📜 License & Attribution

This dataset aggregates multiple open-source datasets. Each retains its original license:

HuggingFaceH4/CodeAlpaca_20K — Apache 2.0
HuggingFaceH4/Code-Feedback — Apache 2.0
nickrosh/Evol-Instruct-Code-80k-v1 — Apache 2.0
WizardLMTeam/WizardLM_evol_instruct_70k — Apache 2.0 (Non-commercial)
TokenBender/code_instructions_122k_alpaca_style — Apache 2.0
iamtarun/code_instructions_120k_alpaca — Apache 2.0
m-a-p/Code-Feedback — CC BY-NC 4.0
Locutusque/code-feedback-sharegpt — Apache 2.0
ajibawa-2023/Code-74k-ShareGPT — Apache 2.0
ajibawa-2023/Python-Code-23k-ShareGPT — Apache 2.0
b-mc2/sql-create-context — CC BY 4.0
glaiveai/glaive-code-assistant — Apache 2.0
glaiveai/glaive-code-assistant-v2 — Apache 2.0
mwitiderrick/glaive-code-assistant — Apache 2.0
Data-Agora/magpie_code_gpt4o_mini_50000 — Apache 2.0
Handcrafted examples (VIRUS374) — MIT

Important: If you use this dataset for commercial purposes, please review the WizardLM and m-a-p licenses — they may have non-commercial restrictions.

🙏 Acknowledgements

Thanks to all the original dataset creators:

HuggingFace H4 team
WizardLM team
TokenBender
iamtarun
m-a-p
Locutusque
ajibawa-2023
b-mc2
glaiveai
mwitiderrick
Data-Agora

📝 Citation

@dataset{virus374_ai_developer_2024,
  title  = {AI Developer Dataset},
  author = {VIRUS374},
  year   = {2024},
  url    = {https://huggingface.co/datasets/VIRUS374/ai-developer-dataset},
  note   = {Aggregated from open-source code instruction datasets + handcrafted examples}
}

Downloads last month: 109