Skip to main content

GPU Backend — Pipeline Stages

:::tip TL;DR

  • 10 sequential stages from video load to report generation
  • GPU-accelerated stages: Ball Detection, Court Detection, Player Detection, Pose Analysis
  • --dev cache skips stages 1–5 after the first run (96% faster iteration)
  • Stage 9 (Video Rendering) is optional — skipped in cloud/serverless mode :::

The active orchestrator is process_video(sport, ...) in src/pipeline/game_processor.py, composing stage classes from src/stages/{ball_tracking,surface,events,physics}/. Each stage writes to a shared result dict; downstream stages read from it. The dev cache persists the output of each stage so subsequent runs skip completed work. A final Phase 6 insight step writes insights.json and embeds it in the analysis JSON.

Stage Overview​

Green stages run on GPU (CUDA / MPS / CPU fallback).

Stage 1 — Video Loading​

Module: src/io/video_reader.py Cache key: none (always runs)

Loads the input video with OpenCV and downscales to 720p (MAX_VIDEO_RESOLUTION_HEIGHT = 720 in src/core/constants.py) to reduce GPU memory pressure. Extracts video metadata used by all downstream stages:

Output fieldDescription
fpsFrame rate (used for all timestamp calculations)
total_framesFrame count
width, heightResolution after optional resize
frames_originalDecoded frame array
python main.py --path_input_video match.mp4   # always downscaled to 720p internally

Downscaling is automatic and fixed at 720p — there is no --max_width / resolution flag.

Stage 2 — Scene Detection​

Module: internal Cache key: none (fast, no cache)

Identifies distinct shots/segments within the video using frame-difference heuristics. For standard tennis match footage this stage usually returns a single scene; it becomes relevant for edited highlight clips or footage with hard cuts.

Output feeds the rally segmentation logic used later in shot grouping.

Stage 3 — Ball Detection​

Module: src/tracking/ball_detector.py (TrackNet) Cache key: ball_tracking.pkl

Runs TrackNet — a convolutional network trained specifically for tennis ball detection — across all frames in batches. Returns a list of (x, y) positions per frame (or None when the ball is occluded).

Key details:

  • Batch size is auto-tuned to GPU memory; OOM triggers adaptive reduction
  • Interpolation: parabolic trajectory fill-in for short occlusion windows (v46+)
  • Output coordinates are in normalized video space (0–1); court transformation happens later
.dev/cache/<video_hash>/ball_tracking.pkl

Stage 4 — Court Detection​

Module: src/surface/surface_detector.py (CourtDetectorNet) Cache key: court_detection.pkl

Detects 14 court keypoints (baselines, service lines, sidelines, net posts) and computes a homography matrix per frame mapping video pixels → real court coordinates in meters (ITF standard: 10.97 m × 23.77 m).

Keypoints detectedCount
Corners (baseline × sideline)4
Service box corners4
Net posts2
Center marks4

The homography matrix is the foundation for all metric-space calculations (ball speed in km/h, court position heatmaps, minimap rendering).

Stage 5 — Player Detection​

Module: src/players/pose_detector.py (FasterRCNN + MediaPipe) Cache key: player_detection.pkl

Two-step detection:

  1. FasterRCNN detects player bounding boxes per frame
  2. Court homography maps bounding box feet-points to court coordinates → assigns players to near_court or far_court

Output: persons_top and persons_bottom — sparse lists of detections per frame. Only frames with detected players are stored (memory-efficient).

Stage 6 — Bounce Detection​

Module: src/events/contact_detector.py (CatBoost) Cache key: bounces.pkl

Two detection strategies run in parallel and their results are merged:

  1. CatBoost classifier — trained on ball trajectory features (velocity, acceleration, curvature change)
  2. Direction-change heuristic — identifies frames where vertical velocity sign flips (ball hits ground)

Post-processing removes bounces that overlap with detected shots within an 80ms window (shot contact and ground bounce produce similar trajectory signals — shot label wins).

Stage 7 — Shot Detection​

Module: src/events/action_detector.py Cache key: shots.pkl

Identifies shot events by combining:

  • Ball trajectory direction change near a player
  • Horizontal travel across the net after contact
  • Occlusion + swing physics (for shots where ball is momentarily hidden by the player)

Rally-aware shot cap: within each rally (period between consecutive serves), at most 6 shots are kept to suppress false positives.

Serve detection: serves are identified separately by an overhead motion classifier and stored as is_serve: true on the shot dict.

Stage 8 — Pose Analysis​

Module: src/events/action_detector.py + src/classification/shot_classifier.py (MediaPipe + ONNX) Cache key: none (runs post-shot detection)

For each detected shot, MediaPipe extracts 33 pose keypoints from the player crop. A sequence classifier (ONNX) classifies the shot type from the pose sequence.

Frame skip: pose extraction runs on every Nth frame (default pose_frame_skip=2) to reduce GPU memory. The ONNX model path is read from GameConfig (fixed in v46 — previously hardcoded incorrectly).

Shot types produced: forehand, backhand, serve, volley, unknown.

Stage 9 — Video Rendering​

Module: src/render/ (skipped in cloud) Cache key: none

Renders the annotated output video with overlays:

  • Ball trajectory trace
  • Player bounding boxes
  • Shot markers and labels
  • Court keypoint dots

This stage is skipped when running in serverless mode (--cloud) to reduce processing time. The annotated video is only generated for local/dev runs.

Stage 10 — Report Generation​

Module: src/export/ (report_generator.py, analysis_export.py, minimap_generator.py) Cache key: none (always runs)

Produces the full output bundle:

Output fileDescription
*_analysis.jsonCanonical analysis JSON (schema 1.5.0)
events_timeline.jsonCompact event frame list for annotation comparison
ball_trajectory_3d.jsonPhysics-based 3D trajectory
analysis_report.pdfHuman-readable performance report
shots/Per-shot metadata, pose JSON, optional shot clips

See Output Schema for the full JSON structure.

Dev Cache​

Run with --dev to enable caching. On first run all stages execute; subsequent runs load from cache for stages 1–5, making iteration ~96% faster.

# First run — ~90s, builds cache
python main.py --path_input_video match.mp4 --dev

# Subsequent runs — ~3s
python main.py --path_input_video match.mp4 --dev

Cache location:

.dev/cache/<video_hash>/
├── ball_tracking.pkl
├── court_detection.pkl
├── player_detection.pkl
├── bounces.pkl
└── shots.pkl

Clear cache: rm -rf .dev/cache/

Performance Reference​

Measured on RTX 3080 with a 30-second video:

StageTime
Ball Detection~30s
Court Detection~10s
Player Detection~40s
Analysis + Export~10s
Total (no cache)~90s
Total (cached)~3s

Next Steps​