ai-toolkit/ui/README_EXTENDED.md

Extended Features Documentation

This document describes all the extended features added to the AI Toolkit that are not part of the original codebase. It includes instructions on how to apply these changes to an unchanged AI Toolkit installation.

Table of Contents

1. New Features Overview
2. Files to Override
3. New Files to Add
4. Feature Details
5. Installation Instructions

---

New Features Overview

1. Advanced Queue (New Feature)

An enhanced version of the Training Queue with advanced filtering, pagination, search, and bulk operations.

Location: /jobs-extended

Features:

• Pagination for idle entries with options: 25, 50, 100, 250, 500, 1000, 2000, 5000, or "all"
• Search by job name
• Status filter dropdown
• Bulk actions:
  • Stop All Queued Jobs: Marks all queued jobs as stopped (leaves running jobs intact)
  • Resume Stopped Jobs: Changes all stopped jobs back to queued status

2. From Dataset to Queue (New Feature)

Allows users to batch-create training jobs from datasets using templates.

Location: /batch-queue

Features:

• Select training template (wan, zimage, etc.)
• View available datasets that haven't been trained yet
• Select multiple datasets and submit them to the training queue
• Automatically redirects to Advanced Queue after submission

3. UI Improvements

• Fixed context menu text color (dark gray → white) for better visibility
• Added tooltips to all action buttons in JobActionBar
• Improved user experience with hover hints

---

Files to Override

These files were modified from the original AI Toolkit. You need to replace them with the extended versions:

1. ui/src/components/Sidebar.tsx

Changes:

• Added "Advanced Queue" navigation item linking to /jobs-extended
• Added "From Dataset to Queue" navigation item linking to /batch-queue
• Uses BrainCircuit icon for Advanced Queue and ListPlus icon for From Dataset to Queue

Location in navigation:

• "Advanced Queue" appears between "Training Queue" and "From Dataset to Queue"
• "From Dataset to Queue" appears after "Advanced Queue"

2. ui/src/components/JobActionBar.tsx

Changes:

• Fixed context menu text color: Added text-gray-200 class to menu items for better visibility on dark background
• Added title attributes (tooltips) to all action buttons:
  • Play button: "Start Job"
  • Pause button: "Stop Job"
  • X button: "Remove from Queue"
  • Eye button: "View Job"
  • Pen button: "Edit Job"
  • Trash button: "Delete Job"
  • Cog button: "More Options"

---

New Files to Add

These are completely new files that don't exist in the original AI Toolkit:

1. ui/src/app/batch-queue/page.tsx

Purpose: Main page component for the "From Dataset to Queue" feature

Description:

• Allows users to select a training template and multiple datasets
• Filters datasets to show only those that haven't been trained yet
• Creates batch jobs and redirects to Advanced Queue after submission
• Includes template selection, dataset filtering, and multi-select functionality

Note: This file redirects to /jobs-extended after batch job creation (not the original /jobs route)

2. ui/src/app/jobs-extended/page.tsx

Purpose: Main page component for the Advanced Queue route

Description:

• Renders the Advanced Queue page with TopBar and MainContent layout
• Uses JobsTableExtended component to display the enhanced queue interface

3. ui/src/components/JobsTableExtended.tsx

Purpose: Enhanced version of JobsTable with advanced features

Features:

• Pagination controls for idle entries
• Search input for filtering by job name
• Status filter dropdown
• Bulk action buttons (Stop All Queued, Resume Stopped)
• All original JobsTable functionality (GPU sections, queue controls, etc.)

Dependencies:

• Uses existing hooks: useJobsList, useQueueList, useGPUInfo
• Uses existing components: UniversalTable, JobActionBar
• Uses existing utilities: startQueue, stopQueue, apiClient

4. ui/src/app/api/jobs/bulk/route.ts

Purpose: API endpoint for bulk job operations

Endpoints:

• POST /api/jobs/bulk with action parameter:
  • action: 'stop-queued' - Stops all queued jobs (marks as stopped)
  • action: 'resume-stopped' - Resumes all stopped jobs (changes to queued)

Request Body:

{
  "action": "stop-queued" | "resume-stopped"
}

Response:

{
  "success": true,
  "count": 5,
  "message": "Stopped 5 queued job(s)"
}

5. ui/src/app/api/batch-queue/route.ts

Purpose: API endpoint for batch job creation from datasets

Endpoints:

• POST /api/batch-queue - Creates multiple training jobs from template and dataset list

Request Body:

{
  "template": "wan",
  "datasets": ["aaliyah", "abbiecornish"],
  "gpu_ids": "0"
}

Response:

{
  "created": 2,
  "errors": 0,
  "errorDetails": []
}

6. ui/src/app/api/templates/list/route.ts

Purpose: API endpoint to list available training templates

Endpoints:

• GET /api/templates/list - Returns array of template names (e.g., ["wan", "zimage"])

7. ui/src/app/api/templates/[name]/route.ts

Purpose: API endpoint to get template content

Endpoints:

• GET /api/templates/[name] - Returns YAML content of the specified template

8. ui/src/app/api/output/list/route.ts

Purpose: API endpoint to list existing output folders

Endpoints:

• GET /api/output/list - Returns array of output folder names

9. ui/src/hooks/useTemplateList.tsx

Purpose: React hook to fetch and cache available templates list

10. ui/src/hooks/useOutputList.tsx

Purpose: React hook to fetch and cache output folder names

---

Feature Details

Advanced Queue Features

Pagination

• Default page size: 25 entries
• Options: 25, 50, 100, 250, 500, 1000, 2000, 5000, or "all"
• Only applies to the "Idle" section (completed/stopped/failed jobs)
• Active jobs (running/queued) are shown in GPU-specific sections without pagination

Search

• Real-time filtering by job name
• Case-insensitive search
• Works in combination with status filter
• Resets to page 1 when search query changes

Status Filter

• Filter options: "all" plus all unique statuses found in idle jobs
• Common statuses: completed, stopped, failed, error
• Works in combination with search

Bulk Actions

Stop All Queued Jobs:

• Finds all jobs with status 'queued'
• Marks them as 'stopped' (does NOT affect running jobs)
• Shows confirmation dialog before execution
• Displays success message with count of stopped jobs

Resume Stopped Jobs:

• Finds all jobs with status 'stopped'
• Changes status to 'queued'
• Assigns proper queue positions (increments of 1000)
• Ensures queue exists for each job's GPU
• Shows confirmation dialog before execution
• Displays success message with count of resumed jobs

From Dataset to Queue Features

This is a new feature that allows batch creation of training jobs from datasets using templates (see BATCH_QUEUE_FEATURE.md for full details).

Key Components:

• Template selection dropdown
• Dataset filtering (excludes already trained datasets)
• Multi-select checkboxes
• Batch job creation
• Error handling for duplicate job names
• Redirects to Advanced Queue after submission (not the original /jobs route)

API Dependencies:

• /api/templates/list - Lists available templates
• /api/templates/[name] - Gets template YAML content
• /api/output/list - Lists existing output folders
• /api/batch-queue - Creates batch jobs from template and datasets

---

Installation Instructions

To apply these extended features to an unchanged AI Toolkit:

Step 1: Backup Original Files

Before making changes, backup the following files:

cp ui/src/components/Sidebar.tsx ui/src/components/Sidebar.tsx.backup
cp ui/src/components/JobActionBar.tsx ui/src/components/JobActionBar.tsx.backup

Step 2: Override Modified Files

Replace the following files with the extended versions:

1. ui/src/components/Sidebar.tsx

   • Copy the extended version that includes both "Advanced Queue" and "From Dataset to Queue" menu items

2. ui/src/components/JobActionBar.tsx

   • Copy the extended version with fixed menu colors and tooltips

Step 3: Add New Files

Create the following new files:

Frontend Pages:

1. ui/src/app/jobs-extended/page.tsx

   • Create the directory if it doesn't exist: ui/src/app/jobs-extended/
   • Add the page component file

2. ui/src/app/batch-queue/page.tsx

   • Create the directory if it doesn't exist: ui/src/app/batch-queue/
   • Add the batch queue page component

Frontend Components:

3. ui/src/components/JobsTableExtended.tsx
   • Add the enhanced table component

API Routes:

4. ui/src/app/api/jobs/bulk/route.ts

   • Create the directory structure: ui/src/app/api/jobs/bulk/
   • Add the bulk operations API route

5. ui/src/app/api/batch-queue/route.ts

   • Create the directory structure: ui/src/app/api/batch-queue/
   • Add the batch queue API route

6. ui/src/app/api/templates/list/route.ts

   • Create the directory structure: ui/src/app/api/templates/list/
   • Add the templates list API route

7. ui/src/app/api/templates/[name]/route.ts

   • Create the directory structure: ui/src/app/api/templates/[name]/
   • Add the template content API route

8. ui/src/app/api/output/list/route.ts

   • Create the directory structure: ui/src/app/api/output/list/
   • Add the output list API route

React Hooks:

9. ui/src/hooks/useTemplateList.tsx

   • Add the template list hook

10. ui/src/hooks/useOutputList.tsx

    • Add the output list hook

Step 4: Verify Dependencies

Ensure all dependencies are installed. The extended features use:

• Existing React hooks (no new dependencies)
• Existing UI components (no new dependencies)
• Existing API utilities (no new dependencies)
• Headless UI components (should already be installed)

Step 5: Test Installation

1. Start the development server: npm run dev
2. Navigate to the sidebar and verify:
   • "Advanced Queue" menu item appears
   • "From Dataset to Queue" menu item exists
3. Test the Advanced Queue:
   • Navigate to /jobs-extended
   • Verify pagination controls appear
   • Test search functionality
   • Test status filter
   • Test bulk action buttons
4. Test batch-queue redirect:
   • Submit jobs from batch-queue page
   • Verify redirect to /jobs-extended instead of /jobs

Step 6: Database

No database migrations are required. The extended features use existing database schema and tables.

---

File Structure Summary

ui/
├── src/
│   ├── app/
│   │   ├── api/
│   │   │   ├── batch-queue/
│   │   │   │   └── route.ts              [NEW]
│   │   │   ├── jobs/
│   │   │   │   └── bulk/
│   │   │   │       └── route.ts          [NEW]
│   │   │   ├── output/
│   │   │   │   └── list/
│   │   │   │       └── route.ts          [NEW]
│   │   │   └── templates/
│   │   │       ├── list/
│   │   │       │   └── route.ts          [NEW]
│   │   │       └── [name]/
│   │   │           └── route.ts           [NEW]
│   │   ├── batch-queue/
│   │   │   └── page.tsx                  [NEW]
│   │   └── jobs-extended/
│   │       └── page.tsx                  [NEW]
│   ├── components/
│   │   ├── JobActionBar.tsx              [MODIFIED - colors + tooltips]
│   │   ├── JobsTableExtended.tsx         [NEW]
│   │   └── Sidebar.tsx                   [MODIFIED - new menu items]
│   └── hooks/
│       ├── useTemplateList.tsx           [NEW]
│       └── useOutputList.tsx             [NEW]
└── README_EXTENDED.md                    [NEW - this file]

---

Notes

• Minimal Changes Philosophy: All changes were designed to minimize modifications to the original codebase
• No Breaking Changes: All extended features are additive and don't break existing functionality
• Backward Compatible: The original /jobs route and JobsTable component remain unchanged
• Shared Components: Extended features reuse existing components, hooks, and utilities where possible

---

Troubleshooting

Issue: Advanced Queue page not found (404)

Solution: Ensure ui/src/app/jobs-extended/page.tsx exists and the directory structure is correct.

Issue: Bulk actions not working

Solution:

• Verify ui/src/app/api/jobs/bulk/route.ts exists
• Check that the API route is accessible at /api/jobs/bulk
• Check browser console for errors

Issue: Menu text not visible in context menu

Solution: Ensure JobActionBar.tsx has text-gray-200 class on menu items.

Issue: Tooltips not showing

Solution: Verify title attributes are present on all buttons in JobActionBar.tsx.

---

Future Enhancements

Potential improvements for the Advanced Queue:

• Export filtered results
• Advanced sorting options
• Column visibility toggles
• Bulk edit capabilities
• Job grouping by template or dataset