🤖 GPU Backend Overview
AI-Powered Video Analysis Engine
:::tip TL;DR
- Python 3.11.x only (not 3.12+) — CatBoost + MediaPipe wheels
- Pipeline: Ball → Court → Players → Bounces → Shots → Analysis → Export
- Runs on the Lambda pull-worker fleet in prod (RunPod removed 2026-08-16), local MPS/CUDA/CPU in dev
--devflag enables a 96%-faster cache for iteration- v46 (April 2026): 5-phase accuracy overhaul — real court coords in meters, parabolic trajectory interpolation, fixed shot classifier ONNX path
- Phase 6: insight engine (
src/insights/) emits rankedinsights.json, also embedded in the analysis JSON :::
The GPU Backend is the core AI processing engine that transforms tennis videos into structured analysis data using computer vision and pose estimation.
:::info Tennis-only (multi-sport deferred)
The shipped product is tennis-only. Both main.py entry points hardcode sport="tennis" and there is no --sport CLI flag. Multi-sport work is deferred; add a new games/<sport>/config.yaml only when real sport-specific models exist.
:::
🎯 System Overview
:::info Tech Stack
| Repository | Commit | Docker Image |
|---|---|---|
| acesense-gpu-backend | daaddb2 | akshaysarode/acesense:v46 |
- Language: Python 3.11.x (required for catboost + mediapipe)
- Object Detection: YOLO v8 / v11 (Ultralytics)
- Pose Estimation: MediaPipe
- Video Processing: OpenCV, FFmpeg
- Bounce Detection: CatBoost
- Hosting: Lambda Cloud, autoscaled by
lambdaScalerTick(RunPod removed 2026-08-16) - GPU: NVIDIA RTX 4090 :::
🏗️ Processing Architecture
🎯 AI Models
YOLO v8 Object Detection
| Class | Precision | Recall | mAP@0.5 |
|---|---|---|---|
| Ball | 94.2% | 91.8% | 93.0% |
| Player | 98.5% | 97.2% | 97.8% |
| Court | 96.1% | 95.4% | 95.7% |
MediaPipe Pose Estimation
📊 Shot Detection Algorithm
Shot Classification
| Shot Type | Detection Method | Accuracy |
|---|---|---|
| Forehand | Wrist position + swing direction | 96.3% |
| Backhand | Arm cross + shoulder rotation | 94.8% |
| Serve | Overhead motion + ball toss | 98.2% |
| Volley | Net proximity + punch motion | 91.5% |
📋 Output Schema (v1.5.0)
The analysis export writes a canonical JSON file (*_analysis.json, schema 1.5.0) plus a standalone insights.json. Spatial coordinates are in meters (ITF court standard), and the analysis JSON embeds an "insights" object.
See the dedicated Output Schema reference for the full, authoritative field-by-field specification — it is the single source of truth and replaces the abbreviated block that used to live here.
🖥️ RunPod Configuration (historical)
:::warning Not in use
RunPod was removed on 2026-08-16 — it repeatedly sat at
402 Insufficient Balance, which turned a dead Lambda worker into a failed job
rather than a slower one. The adapter and this section are kept for the day the
account is funded and runpod is named in GPU_PROVIDER_POLICY.enabled.
:::
Environment Variables
| Variable | Description |
|---|---|
FIREBASE_CREDENTIALS | Service account JSON |
STORAGE_BUCKET | GCS bucket name |
MODEL_PATH | YOLO weights path |
LOG_LEVEL | Logging verbosity |
⚡ Performance
| Metric | Value |
|---|---|
| Processing Speed | ~2x real-time |
| Frames per Second | 60 FPS |
| Memory Usage | ~18 GB VRAM |
| Cold Start | ~15 seconds |
| Warm Inference | ~10ms/frame |
Optimization Techniques
- Batch Processing: Process 8 frames at once
- TensorRT: Optimized YOLO inference
- Half Precision: FP16 for pose estimation
- Stream Processing: Decode while processing
🔄 Error Handling
🎯 Next Steps
- 🚀 Backend Dev Quickstart — 5-min setup
- 🔧 GPU Backend Setup — full install walkthrough
- 🚢 Deployment Guide — Docker → RunPod
- 🐛 Troubleshooting — CUDA OOM, MPS, FFmpeg
📚 Related
- Architecture Overview — System design
- Firebase Functions — Job triggers
- Frontend — Mobile app display