Overview
While taking a course in digital image processing, I built a simple prototype for an LLM driving assistant. The goal was to build a pipeline to get an LLM to understand a driver’s surroundings from a dashcam feed, and then alert the user of any potential traffic incidents, such as being aware of a pedestrian or a car drifting into their lane.
What I built
- I wrote a python script to scrape videos from multiple subreddits. There are many dashcam videos on Reddit of people almost getting into accidents, and are the perfect data to testing our pipeline.
- I designed an entire pipeline to allow a large-language model to understand what was in view of the dashcam.
- I wrote code to run the pipeline with over 40 videos to measure its real-time performance, and roughly, how well it understood its surrounds and if there would be a potential collision.
Technologies
Python, PyTorch, OpenCV, HuggingFace, LLMs, Real-Time Detection Transformers (RT-DETRs), YOLO CV Models