One of the project which involved me building pipelines and was somewhat applied ml was this project. Back in my third year, a few friends and I put together SoccArt; an end-to-end computer vision pipeline that takes raw broadcast footage of a soccer match and turns it into structured, analyzable data: player and ball positions, team assignments, jersey numbers, and a top-down minimap of the game.
What made this project memorable was that it got picked up beyond just a course submission. We were invited by Prof. Deepak Subramani to present it as a poster, and it ended up going to the 1st Conference on Applied AI and Scientific Machine Learning (CASML 2024), held from 16th-18th December 2024, under the title "State of the Art Soccer Analysis Pipeline using Artificial Intelligence." It was a great excuse to take something that started as a hacky course project and push it to a level where it could hold up in front of a room full of people asking hard questions about it.
Motivation
A lot of us on the team just enjoy watching football, and like most fans we wanted to keep track of scores and maybe get a bit predictive about it; using past data to reason about how a team or player is likely to perform. Doing that properly means keeping information about every player up to date: where they played on the pitch, how many goals they scored, how their game evolved over a season. You can maintain all of that by hand, but it becomes a lot easier if you can just point a model at existing match footage and have it keep track of everyone for you.
This line of thinking is what pushed us toward wanting a top-down view of the game rather than just annotating the broadcast feed. Watching the broadcast angle, it's genuinely hard to tell how the ball is actually moving between the two teams. A bird's-eye minimap makes that immediately obvious, and that was really the core intuition. If somehow we can manage to get bird's eye view of the game, we can use that map to keep track of player's position in the ground or track any sensible information. So the technical bet became: if we could track the jersey number, the player, and the ball closely enough to reconstruct an equivalent top-down view of the match, a lot of interesting statistics would start to fall out almost for free.
That was more or less the motivation that led us to propose this as a course project. I quite enjoyed working on it back then.
What the pipeline actually does
The system is built around a few stages, each of which was its own rabbit hole of tuning and engineering:
- Detection & tracking: YOLOv10 for detecting players, referees, and the ball, combined with ByteTrack to keep consistent identities across frames.
- Team identification & jersey recognition: SigLIP-based color clustering to split players into teams, plus ResNet34 and ViTPose-based models to read jersey numbers off moving players.
- Field calibration: homography estimation to calibrate the broadcast camera against the pitch and project every detection onto a 2D minimap.
- Temporal smoothing: interpolation with sliding-window smoothing so tracks don't jitter frame to frame, which matters a lot once you start computing things like distance covered or passing lanes.
Here's the pipeline running on an actual clip — raw frame, detections with tracking IDs, and the resulting minimap:
Here's the tracking demo video from the repo:
Team
This was very much a group effort with Aditya Gupta, Armaan Khetarpaul, R. K. Shishir, S. Sharath, and Umang Majumder; each of us owned a different stage of the pipeline, from detection and tracking to calibration and jersey recognition, and the poster was really a synthesis of all of it.
Code, the trained weights, the full poster, and the abstract are all up on the GitHub repo if you want to dig into the implementation.