The goal of this project is to familiarize you with the essential building blocks that make up modern large language and multimodal models.
To prepare you for this, we will distribute homeworks that teach you the basics of PyTorch and computing clusters like SCITAS, which will be used throughout this course. Additionally, you will explore the 4M multimodal foundation model, gaining an understanding of its applications in generation and retrieval tasks.
The main part of the project will be about implementing nano4M, a minimal version of 4M, which will give you a practical understanding of how to design and train multimodal foundation models. In addition, you will implement a series of extensions of your choice on top of the nano4M codebase.
To ensure you grasp these concepts, we have designed exercises in the provided Jupyter notebooks. These exercises will help you verify your understanding as you progress through the materials.
- Week 1 to week 4: PyTorch tutorial, SCITAS setup, 4M tutorial
- Week 5 to week 9: Nano-4M project
- Week 10 to week 15: Nano-4M extension
Follow the provided instructions at SCITAS Tutorial to set up your SCITAS environment.
PyTorch is an open-source deep learning framework, which provides a flexible and intuitive way to build deep learning models. In the PyTorch_Tutorial folder, you will find three tutorials covering the basic usage of PyTorch and the corresponding exercises. If you are already familiar with PyTorch, you can proceed directly to the exercises.
After gaining familiarity with PyTorch and SCITAS, you will explore the multimodal foundation model 4M. This hands-on experience will help you understand the model's key components and how to utilize its pipeline for generation and retrieval tasks.
To get started, follow the instructions in the 4M_Tutorial folder to learn more about the model, set up the required environment, and experiment with the provided Jupyter notebooks!
We will share more details about this soon! For now, please see COM_304_Spring_2025_nano4M_Project_Guidelines.pdf for nano4M project guidelines (last updated 18.02.2025).