```
### GPU Metrics Chart Component
The `GPUMetricsChart` component visualizes monitoring data:
```tsx
import GPUMetricsChart from '@/components/GPUMetricsChart';
// In experiment details modal:
{selectedExperiment.metrics?.gpu_monitoring && (
GPU Monitoring
)}
```
#### Chart Features
1. **Monitoring Summary**
- Duration and sample count
- Displayed in blue info box
2. **GPU Stats Table**
- Per-GPU statistics
- Columns: GPU ID, Model, Utilization, Memory, Temperature, Power
- Shows mean values with min-max ranges
3. **Interactive Charts** (Recharts LineChart)
- GPU Utilization (%)
- Memory Usage (%)
- Temperature (°C)
- Power Draw (W)
- Responsive design (adapts to container width)
### TypeScript Types
```typescript
// frontend/src/types/api.ts
export interface Experiment {
// ... other fields
gpu_info?: {
model: string;
count: number;
device_ids?: string[];
world_size?: number;
gpu_info?: {
count: number;
indices: number[];
allocation_method: string;
details?: Array<{
index: number;
name: string;
memory_free_mb: number;
utilization_percent: number;
availability_score: number;
}>;
};
};
metrics?: {
// ... other metrics
gpu_monitoring?: {
monitoring_duration_seconds: number;
sample_count: number;
gpu_stats: {
[gpu_index: string]: {
name: string;
utilization: { min: number; max: number; mean: number; samples: number };
memory_used_mb: { min: number; max: number; mean: number };
memory_usage_percent: { min: number; max: number; mean: number };
temperature_c: { min: number; max: number; mean: number };
power_draw_w: { min: number; max: number; mean: number };
};
};
};
};
}
```
## API Reference
### gpu_monitor.py
#### GPUMonitor Class
**`get_gpu_monitor() -> GPUMonitor`**
- Returns the global GPUMonitor singleton instance
- Thread-safe initialization
**`is_available() -> bool`**
- Check if nvidia-smi is available on the system
- Returns False if nvidia-smi not found or execution fails
**`query_gpus(use_cache: bool = True) -> Optional[GPUSnapshot]`**
- Query current GPU status
- Args:
- `use_cache`: Use cached data if available and recent (default: True)
- Returns: GPUSnapshot with GPU data or None if query fails
- Cache TTL: 5 seconds
**`get_available_gpus(min_memory_mb: Optional[int] = None, max_utilization: int = 50) -> List[int]`**
- Get list of available GPU indices
- Args:
- `min_memory_mb`: Minimum free memory required (optional)
- `max_utilization`: Maximum utilization percentage (default: 50)
- Returns: List of GPU indices sorted by availability score (descending)
**`get_gpu_info(gpu_index: int) -> Optional[GPUInfo]`**
- Get information for a specific GPU
- Args:
- `gpu_index`: GPU index (0-based)
- Returns: GPUInfo object or None if not found
**`start_monitoring(interval_seconds: float = 1.0) -> None`**
- Start continuous GPU monitoring thread
- Args:
- `interval_seconds`: Sampling interval (default: 1.0)
- Clears any previous monitoring data
**`stop_monitoring() -> Optional[Dict[str, Any]]`**
- Stop monitoring thread and return aggregated statistics
- Returns: Dictionary with monitoring data (see data structure above)
- Returns None if monitoring was not started
#### Data Classes
**`GPUInfo`**
```python
@dataclass
class GPUInfo:
index: int # GPU index (0-based)
name: str # GPU model name
memory_total_mb: int # Total memory in MB
memory_used_mb: int # Used memory in MB
memory_free_mb: int # Free memory in MB
utilization_percent: int # GPU utilization (0-100)
temperature_c: int # Temperature in Celsius
power_draw_w: float # Power draw in Watts
score: float # Availability score (0.0-1.0)
```
**`GPUSnapshot`**
```python
@dataclass
class GPUSnapshot:
timestamp: datetime # When snapshot was taken
gpus: List[GPUInfo] # List of GPU information
```
### gpu_scheduler.py
**`estimate_gpu_requirements(task_config: Dict[str, Any]) -> Tuple[int, int]`**
- Estimate GPU requirements from task configuration
- Args:
- `task_config`: Task configuration dictionary
- Returns: Tuple of (min_gpus_required, estimated_memory_mb_per_gpu)
- Calculation: `world_size = tp × pp × max(dp, dcp, cp)`
**`check_gpu_availability(required_gpus: int, min_memory_mb: Optional[int] = None) -> Tuple[bool, str]`**
- Check if sufficient GPUs are available
- Args:
- `required_gpus`: Number of GPUs required
- `min_memory_mb`: Minimum memory per GPU (optional)
- Returns: Tuple of (is_available, message)
- Message contains detailed status or error information
**`wait_for_gpu_availability(required_gpus: int, min_memory_mb: Optional[int] = None, timeout_seconds: int = 300, check_interval: int = 30) -> Tuple[bool, str]`**
- Wait for sufficient GPUs to become available
- Args:
- `required_gpus`: Number of GPUs required
- `min_memory_mb`: Minimum memory per GPU (optional)
- `timeout_seconds`: Maximum wait time (default: 300 = 5 minutes)
- `check_interval`: Polling interval (default: 30 seconds)
- Returns: Tuple of (is_available, message)
- Logs check attempts and status periodically
### docker_controller.py
**`select_gpus_for_task(required_gpus: int, min_memory_mb: int = 8000) -> List[int]`**
- Select optimal GPUs for task execution
- Args:
- `required_gpus`: Number of GPUs needed
- `min_memory_mb`: Minimum free memory per GPU (default: 8000)
- Returns: List of GPU indices
- Raises: RuntimeError if insufficient GPUs available
### direct_benchmark_controller.py
**`run_benchmark_job(endpoint_url: str, benchmark_spec: Dict[str, Any], gpu_indices: Optional[List[int]] = None) -> Dict[str, Any]`**
- Run benchmark with GPU monitoring
- Args:
- `endpoint_url`: Inference service endpoint
- `benchmark_spec`: Benchmark configuration
- `gpu_indices`: GPU indices being used (optional, for logging)
- Returns: Dictionary with benchmark results and `gpu_monitoring` field
## Configuration
### Environment Variables
No specific environment variables required. GPU monitoring uses nvidia-smi from PATH.
### Task Configuration
Specify parallel configuration in task JSON:
```json
{
"parameters": {
"tp-size": [1, 2, 4], // Tensor parallelism
"pp-size": [1], // Pipeline parallelism
"dp-size": [1], // Data parallelism
"cp-size": [1], // Context parallelism
"dcp-size": [1] // Decode context parallelism
}
}
```
### Scheduler Configuration
Modify timeout and interval in `autotuner_worker.py`:
```python
# Wait for GPUs with custom timeout
is_available, message = wait_for_gpu_availability(
required_gpus=required_gpus,
min_memory_mb=estimated_memory_mb,
timeout_seconds=600, # 10 minutes (default: 300)
check_interval=60 # Check every minute (default: 30)
)
```
### Monitoring Configuration
Adjust sampling interval for continuous monitoring:
```python
# Sample every 2 seconds instead of 1
gpu_monitor.start_monitoring(interval_seconds=2.0)
```
## Troubleshooting
### nvidia-smi Not Found
**Symptom**: Warnings like "nvidia-smi not available"
**Cause**: nvidia-smi not in PATH or NVIDIA drivers not installed
**Solution**:
- System proceeds without GPU monitoring (graceful degradation)
- Install NVIDIA drivers and CUDA toolkit
- Verify: `nvidia-smi` command works in terminal
### No GPUs Available
**Symptom**: Task fails with "Insufficient GPUs after waiting"
**Cause**: All GPUs are busy or don't meet memory requirements
**Solutions**:
1. Wait for running tasks to complete
2. Reduce `min_memory_mb` requirement
3. Reduce parallel configuration (tp-size, pp-size)
4. Increase timeout: `timeout_seconds=600`
### Incorrect GPU Count Estimation
**Symptom**: Task requests wrong number of GPUs
**Cause**: Parameter names not recognized or misconfigured
**Solutions**:
1. Use standard hyphenated format: `tp-size` not `tp_size`
2. Check parameter values are lists: `[1, 2, 4]` not `1`
3. Verify task config JSON structure
4. Check logs for "Estimated requirements" message
### GPU Allocation Failures
**Symptom**: RuntimeError during GPU selection
**Cause**: Insufficient GPUs with required memory
**Solutions**:
1. Lower memory requirement: `min_memory_mb=6000`
2. Free up GPU memory (stop other processes)
3. Use fewer GPUs (reduce tp-size)
### Monitoring Data Not Appearing
**Symptom**: No GPU charts in experiment details
**Cause**: Monitoring not enabled or failed to collect data
**Solutions**:
1. Verify nvidia-smi works: `nvidia-smi` in terminal
2. Check experiment metrics in database has `gpu_monitoring` field
3. Ensure DirectBenchmarkController is being used (Docker mode)
4. Check worker logs for monitoring errors
### Docker Container Can't Access GPUs
**Symptom**: "No accelerator available" in container logs
**Cause**: Incorrect Docker GPU configuration
**Solutions**:
1. Verify `device_requests` is used, not `CUDA_VISIBLE_DEVICES`
2. Check nvidia-container-toolkit installed: `docker run --gpus all ...`
3. Verify GPU indices are valid: `nvidia-smi -L`
4. Don't mix `device_requests` and `CUDA_VISIBLE_DEVICES`
### Frontend Not Showing GPU Info
**Symptom**: "N/A" in GPU column
**Cause**: Experiment doesn't have gpu_info
**Solutions**:
1. Verify task uses Docker mode (OME mode has limited GPU tracking)
2. Check experiment record in database has `gpu_info` field
3. Ensure DockerController's `select_gpus_for_task` was called
4. Verify frontend TypeScript types are up to date
## Performance Considerations
### Cache Usage
- **Query Cache**: 5-second TTL reduces nvidia-smi overhead
- **Recommendation**: Use default cache for frequent queries
- **Force Refresh**: Use `use_cache=False` for critical decisions (GPU allocation)
### Monitoring Overhead
- **Sampling Rate**: Default 1 second is good for most workloads
- **Overhead**: Minimal (~1% CPU per GPU monitored)
- **Recommendation**: Increase interval to 2-5 seconds for very long benchmarks (>10 minutes)
### Scheduler Polling
- **Default Interval**: 30 seconds balances responsiveness and overhead
- **Recommendation**: Use shorter interval (10-15s) for high-priority tasks
- **Recommendation**: Use longer interval (60s) when many tasks in queue
### GPU Allocation Strategy
- **Scoring Algorithm**: Prioritizes memory (60%) over utilization (40%)
- **Rationale**: Memory is hard constraint, utilization is soft
- **Recommendation**: Adjust if workload is compute-bound rather than memory-bound
### Database Storage
- **GPU Info**: Stored as JSON in experiment record (~1-2 KB per experiment)
- **Monitoring Data**: Can be large for long benchmarks (~50 KB for 1000 samples)
- **Recommendation**: Consider cleanup policy for old experiment monitoring data
## Best Practices
1. **Use Docker Mode**: Full GPU tracking support (OME mode has limited support)
2. **Set Realistic Memory Requirements**: Over-estimation causes unnecessary waits
3. **Configure Timeouts Appropriately**:
- Short tasks (< 5 min): 300s timeout
- Long tasks (> 10 min): 600-900s timeout
4. **Monitor System Load**: Use `watch -n 1 nvidia-smi` to understand GPU usage patterns
5. **Tune Scoring Algorithm**: Adjust weights in `_calculate_gpu_score()` for your workload
6. **Archive Monitoring Data**: Consider moving old monitoring data to separate storage
7. **Use Graceful Degradation**: System works without nvidia-smi, but with reduced visibility
8. **Check Logs**: Worker logs contain detailed GPU scheduling information
9. **Test Parallel Configs**: Verify world_size calculation matches your expectation
10. **Frontend Caching**: TanStack Query caches experiment data to reduce API calls
## Future Enhancements
Potential areas for improvement:
- **Multi-Node GPU Scheduling**: Support for distributed GPU allocation across nodes
- **Predictive Scheduling**: ML-based prediction of task duration and GPU requirements
- **Dynamic Reallocation**: Move tasks between GPUs based on load
- **WebSocket Updates**: Real-time GPU metrics streaming to frontend
- **GPU Affinity**: Pin specific experiments to specific GPUs
- **Power Capping**: Enforce power limits for energy efficiency
- **Historical Analytics**: Track GPU utilization trends over time
---
## Intelligent GPU Allocation (OME/Kubernetes)
For Kubernetes deployments, the system includes cluster-wide GPU discovery and intelligent node selection.
### Features
1. **Cluster-wide GPU Discovery**
- Queries all nodes in Kubernetes cluster
- Collects GPU capacity, utilization, memory, temperature
- Node-level GPU availability tracking
2. **Intelligent Node Selection**
- Determines GPU requirements from task parameters (tp-size)
- Ranks nodes based on idle GPU availability
- Idle criteria: <30% utilization AND <50% memory
- Applies node affinity to InferenceService deployments
3. **Automatic Fallback**
- Falls back to K8s scheduler if no metrics available
- Graceful degradation if no idle GPUs found
- Can disable with `enable_gpu_selection=False`
### Implementation
See `src/controllers/gpu_allocator.py`:
- `get_cluster_gpu_status()`: Cluster-wide discovery
- `select_best_node()`: Node ranking algorithm
- Integrates with OMEController for deployments
### Benefits
- Balanced GPU utilization across cluster
- Avoids overloaded nodes
- Reduces deployment failures from resource contention
- Works with dynamic Kubernetes clusters