2D Lego Tracking - Object Counter - Multiple Object Tracking

AI Laboratory Work (ELTE 2025 - Summer Work + Autumn Semester)

It is a research based project in which object detection, segmentation, and tracking algorithms focusing on LEGO bricks in real-world video recordings are used. Using YOLOv8 and RT-DETR models for the detection, DeepLabv3 for semantic segmentation and DeepSORT for tracking, the research evaluates performance in building an object counter. For the detailed information, you can read the report.

Technology

• Python: Main programming language.
• PyTorch: Machine Learning framework.
• Ultralytics: AI platform that makes the usage of YOLOv8 and RT-DETR models easy.
• OpenCV: Used for controlling video frames and processing them.
• RoboFlow: It is a platform used to annotate images with boundary boxes.
• Kaggle: This platform has been used to store data.

Google Colab

• To train or fine-tune detection and segmentation models, usage of GPU is recommended.
• During research, codes have been executed in Google Colab as it is a hosted Jupyter Notebook service that provides free access to computing resources, including GPUs and TPUs.

Dataset

• In this research, the dataset was manually created using a smartphone, consisting of 12 videos — 10 for training and 2 for testing. These videos capture conveyor belts with LEGO bricks from multiple angles with varying complexities for diversity. LEGO bricks were annotated frame-by-frame using RoboFlow (https://roboflow.com/), which was the most time-consuming step.
• Dataset is publicly available in Kaggle: https://www.kaggle.com/datasets/hbahruz/multiple-lego-tracking-dataset

Pipeline

Pre-trained parameters

• The folder contains .pt files of YOLOv8 and RT-DETR detection models which are the saved parameters of best YOLOv8 and best RT-DETR models represented in Lego_Results.xlsx (located under Results folder)

Models

• Jupyter Notebook named "Fine-Tuning Detection Models" is used to fine-tune the YOLOv8 and RT-DETR and save the parameters
• data.yaml file is used in the training procedure of the detection models, which contains data path, augmentation types and other needy information
• Jupyter Notebook named "Fine-Tuned RTDETR + DeepSORT" is a combination of the Detection-Tracking pipeline. YOLOv8 can also be used instead of RT-DETR with some minor changes.

Report

• Actual report and its editable format in LatEx is provided under this folder

Results

• The folder contains Lego_Results.xlsx which has all saved values during the execution of the codes
• There are also some figures underlining important points obtained from the experiments

Watch Test Videos obtained from the execution of the pipelines