Natural language safety queries on live video, zero training data required.
Workplace safety monitoring traditionally requires expensive labeled video datasets to train a custom detector for each violation type. If a new hazard emerges (a new machine, a new PPE requirement), you're back to data collection. VectorGaze eliminates this bottleneck by using CLIP's zero-shot visual-language alignment: instead of training a detector, you describe the violation in natural language, and CLIP finds frames that semantically match. This makes the system instantly configurable for any safety scenario without a single labeled example.
Frames are sampled from the live or recorded video stream at configurable intervals (e.g., 1 frame per second for real-time, denser for analysis).
Each frame is encoded by CLIP's vision encoder into a 512-dimensional embedding that captures the visual semantics of the scene.
The natural language safety query (e.g., "worker not wearing hard hat") is encoded by CLIP's text encoder. LanceDB performs approximate nearest-neighbor search across all frame embeddings.
Top candidate frames are passed to Moondream2 VLM for detailed visual reasoning, reducing false positives before alerting.
Entire pipeline runs on Apple Silicon (M-series) with Metal acceleration, no cloud calls, keeping footage private and latency sub-second.
Zero-shot detection of arbitrary safety violations with no labeled training data
LanceDB enables real-time vector search across thousands of frame embeddings
Moondream2 reasoning step reduces false positive rate significantly
Fully on-device on Apple Silicon, so no data leaves the local network