Maximillian Mandel Research Notebok

[kemandel]

Research notebook for senior comp

[kemandel]

9-14-2023

Project Overview

Conceptualize and implement a non-photorealistic rendering algorithm to be used with real-time applications such as video games. This algorithm will be a core part of the art direction of the application, and the goal of its use and implementation would be to substantially reduce the cost of developing complex visual art assets for 3D real-time programs.

Completed

• Research existing NPR algorithms used for non real-time programs and their implementations
  • Gooch Shading
  • Chinese Painting of Animals
  • Edge Enhancement

To Do

• Conduct further research on existing NPR algorithms
  • Implement them as described to deepen understanding
• Identify a creative or technical gap in existing real time algorithms
  • Convert non real-time to real-time through optimizations?
  • Find a unique artistic style not commonly used in digital media?

[kemandel]

9-19-2023

Completed

• Spoke with Professor Kapfhammer about moving forward with my project
  • Found new resources to help research existing ideas and papers
  • Discussed performance metrics
    • Algorithm efficiency in terms of real time performance (frames per second)
    • Visual clarity from a subjective perspective (how good does it look/feel?)
    • Accuracy (Does the algorithm emulate the desired effect accurately?)

To Do

• Find more papers and examples of non-photorealistic rendering
• Find potential substitutes for expensive rendering effect such as Gaussian Blur
• Research unique art/shading techniques to help determine the visual goal of the algorithm

[kemandel]

9-21-2023

Completed

• Found more sources and documentation on NPR effect
  • Post-Processing NPR Effects for Video Games
    • Surface level overview of the general potential for NPR effects in video games
  • Ray Tracing NPR-Style Feature Lines
    • Discusses the potential to create NPR-like effect through ray-tracing

To Do

• Further research on NPR algorithms and effects

[kemandel]

9-26-2023

Completed

• Started implementing my findings into a short Unity3D demo
  • Integrated Depth/Normal Edge detection
  • Integrated blue noise dithering for a stipple effect
  • Experimented with different noise textures and variables

To Do

• Demonstrate short demo on 9/29
• Conduct further research
• Continue prototype development

[kemandel]

9-28-2023

Completed

• Spoke with Professor Kapfhammer about further implementations
  • How can I modularize the different shader techniques to individually test components?

[kemandel]

9-29-2023

Alumni Panel Discussion & Social

• Talked with peers about ongoing progress and comps
  • Discussed the Unity engine changes and how that may affect my ongoing projects
  • Discussed how current and past courses adjusted our comps
    • How the AI course influenced what our comps were going to be
• Talked with an alumni named Rachel about grad school and moving forward after graduation
  • How to get support and find schools to go to for a graduates degree

[kemandel]

9-30-23 & 10-2-23

Completed

• Extended demo in functionality to allow easily stacking and saving various shader effects and combinations
  • Works by creating an object called a ProcessorStack that saves the load order for various ImageProcessor objects
    • ImageProcessor Objects simply initialize a shader to a specific set of variables. This allows us to save two types of shaders with different parameters for the various functions
  • Allows me to save unique combinations of effects that create interesting or unique NPR visuals
• Implemented an Identity shader to help test the core runtime of the program on the CPU side of things
• Implemented Dither shader
  • This shader uses ordered dithering with a bayer 4x4 matrix to dither colors on screen
  • Useful for preventing color banding when limiting the color palette
• Implemented Palette Compression shader
  • This shader takes a predefined color palette as a parameter and compresses all of the colors on screen to match the palette.
    • This is done by converting all of the colors to the OkLab color space
      • Allows us to better calculate a delta value which can then be used to determine the closest color in the palette
    • Lots of application in altering pre-existing assets in a unique way, or to supplement the creation of other visual effects
    • Works with a palette of up to 256 colors

To Do

• Prepare a couple different ProcessorStack with unique visuals for an in-class demo on 10-6
  • One utilizing the Edge Detection and stipple shaders and the other using the Palette and Dither shaders
• Compare the runtime cost of each of the individual shader effects
• Expand demo to include more unique and varied geometry

[kemandel]

10-3-23

Completed

• Refactor code to improve optimization and real-time editing of parameters
  • Limited the transfer of information from CPU to GPU of every frame whilst still maintaining the flexibility to edit variables at runtime
• Implemented Binary Shader
  • Converts each pixel of an image to black or white based on the specified threshold
• Implemented a Stipple processor stack
• Changed scene lighting to function more accurately (Global -> Spot light)

To Do

• Prepare for an in-class demo on 10-6
• Research and implement addition image processing effects

[kemandel]

10-4-2023

Completed

• Spoke with Professor Kapfhammer about completed work
  • How can I more accurately define my algorithm?
  • What would I name the technique or style I am searching for?
    • Does it have a name already?
    • Is it unique enough for me to name it myself
  • What makes my technique stand apart from others?
    • Real time implementation of image processing techniques?
      • Could I use this to add a temporal element?

[kemandel]

10-7-23 & 10-9-23

Completed

• Attempted to implement a temporal-based shader
• Implemented a gaussian-based depth of field shader
  • Allows other, stylistic shaders to express more detail
  • Generated by calculating the distance from the focal point our camera is, and comparing it with other pixel's depth
    • Allows us the mix a pixel with another, blurred texture to create a sort of unfocused effect

To Do

• Identify the final scope and style for the thesis
• Research existing types of image processing and art styles

[kemandel]

10-13-23

Completed

• Initial Implementation of Artifact prototype
  • Completed Unity build
  • Completed README
    • Included instructions on running both build and Unity project
    • Included high level descriptions of NPR effects and algorithm ideas
    • Added sources of help for algorithm implementations
  • Added Github CI
    • Lint md files

To Do

• Prepare pitch presentation
  • Demo Video
  • Slides

[kemandel]

10-19-23

Completed

• Met with professor Kapfhammer and discussed README details
  • Talked about making a github releases page for project builds
    • Implemented
  • Talked about how to use this information for pitch

To Do

• Make pitch presentation

[kemandel]

10-22-23

Completed

• Made pitch presentation slides
  • Includes high level descriptions of the field I am studying and contributing to
  • Explains how GPU shaders and real-time applications work
  • Shows examples of NPR effects and post-processing ideas

To Do

• Present on 10/27

[kemandel]

11-30-23

Completed

• Met with Doctor Kapfhammer and discussed writing first 2 chapters
  • Talked about repository, best practices, minimum word count for final thesis

To Do

• Start writing first chapter

[kemandel]

12-3-23

Completed

• Started writing introduction chapter of thesis
  • Wrote Motivation
  • Tested Github Actions

To Do

• Finish Introduction Chapter

[kemandel]

12-6-23

Completed

• Continued writing Introduction

To Do

• Finish Introduction Chapter

[kemandel]

12-7-23

Completed

• Met with Dr. Kapfhammer and discussed introduction chapter
• Continued writing introduction
  • Rewrote goals

To Do

• Finish Introduction Chapter

[gkapfham]

Hello @kemandel, overall your research notebook is very good. I especially appreciated your initial entries that contained numerous details and follow-ups on your status. Thanks! I encourage you to ensure that, going forward during the upcoming semester, all of your entries contain sufficient detail. In particular, some of the final entries do not contain enough information about the tasks that you completed.

[kemandel]

1-18-2024 to 1-26-2024

Completed

• Revised Introduction Chapter
  • Added to motivation
    • Outlined precisely what the project aims to do
    • Discussed how the project contributes to the field of real-time image processing
      • How others can benefit from the thesis
  • Added references throughout chapter
    • References on post-processing, shaders, npr effects in games
• Revised Related Works Chapter
  • Added references throughout
    • References primarily on NPR effects, dithering, color, noise generation

To Do

• Start developing artifact into a completed, demo-ready product
• Continue writing additional chapters

[kemandel]

1-26-2024 to 2-1-2024

Completed

• Started expanding demo into a full experience
  • Started crafting a new level with rooms/hallways
    • Different geometry for rooms
    • Different Lighting in rooms and hallways
    • Made level use more neutral colors to show the dynamic effects of the light with the shaders
  • Modified scene lighting to be more dynamic and impactful
    • Global light largely eliminated
    • Individual, localized lights of different colors are used to highlight scenes
  • Build an interact-based system for the player to interact with objects in the world
    • Objects can move/create lighting to allow the shader to show its flexibility

To Do

• Continue developing demo to include more dynamic areas with new lighting & geometry
• Start writing and implementing the "methods" chapter

[kemandel]

2-2-2024 to 2-8-2024

Completed

• Outlined methods draft
  • Determined what content to prioritize
  • Determined order of discussion
  • Figured out what levels of detail to use when describing algorithms

To Do

• Continue developing demo to include more dynamic areas with new lighting & geometry
• Finish methods draft

[kemandel]

2-9-2024 to 2-15-2024

Completed

• Draft of Methods Chapter
  • High level description of graphics pipeline algorithm
    • Justification of using a pipeline
    • Advantages and disadvantages of this approach
  • Detailed overview of edge detection algorithm
    • Depth-based edge detection psuedo code
    • Depth-based edge detection image examples

To Do

• Continue developing demo to include more dynamic areas with new lighting & geometry
• Finalize Methods Chapter
  • Dithering Section
  • Color Quantization Section

[kemandel]

2-16-2024

Added issue: https://github.com/ReadyResearchers-2023-24/YanqiaoChenArtifact/issues/3

[kemandel]

2-16-2024 to 2-22-2024

Completed

• 2nd Draft of Methods
  • ~1600 words
    • Expected ~2500
  • In-depth descriptions and figures for core algorithms
    • Depth-based edge detection
    • Color Quantization
• Beginning outlines for experiments chapter

To Do

• Continue developing demo to include more dynamic areas with new lighting & geometry
• Finalize Methods Chapter
  • Dithering Section
  • Stack overview with figures

[kemandel]

2-23-2024 to 2-29-2024

Completed

• Final Draft of Methods
  • ~3300 words
  • Full sections for each algorithm
    • Technical descriptions and pseudo code for algorithms
    • Visual examples of the algorithm effects on images
  • Pipeline section for full image effect
    • Images taken at each section of pipeline
    • Written descriptions of the algorithms' effects at each step

To Do

• Add references to methods (if applicable)
• Add Flow chart to methods
• Expand experiments chapter

[kemandel]

3-8-2024 to 3-14-2024

Completed

• Experimental Testing scripts
  • Iteratively run through different graphics stacks
  • Record fps at each step
  • Save results to files
• Start outlining and considering writings
• Added to methods
  • Added references
  • Added flow chart for graphics pipeline

To Do

• Finish experiments chapter
• Finish conclusion chapter

[kemandel]

3-15-2024 to 3-21-2024

Completed

• Artifact
  • Finished experiment capabilities
    • Added Standard deviation data
    • Automated full experimentation process
• Experiments Chapter
  • Ran various tests on 2 different machines
    • Recorded results
  • Analyzed results of experimentation
    • Wrote about various trends
    • Compared performance of different algorithms
      • Compared individual graphics processing algorithms
      • Compared pipelines or 'stacks' of processing algorithms
      • Used an Identity test as a control case
• Conclusions Chapter
  • Summarized goals of the project
  • Summarized method of approach
  • Discussed results and their implications
  • Discussed ethical concerns
• Abstract
  • Described thesis overview at a high level

To Do

• Prepare for presentation

[kemandel]

3-22-2024 to 3-28-2024

Completed

• Prepare presentation
  • Created slides to illustrate ideas
    • Included images, flow charts, tables
    • Included short descriptions of algorithms
  • Created demonstration video
    • ~2 minutes long

To Do

• Practice presentation

[kemandel]

4-26-2024 to 5-3-2024

Completed

• Implemented Senior Thesis feedback
  • Retitled project to be more specific
  • Added empirical results summarization to abstract
  • Reviewed fair use policies for video game imagery
  • Added 'impact' section to introduction
  • Fixed reference citing
  • Described how bayer 4x4 matrix is used in my work
  • Extended each section of chapter 2 to provide addition context to the work in relation to the overall thesis
  • Extended chapter 3 to describe how to run the tool both independently and through Unity
  • Described how experimental variables are calculate for standard deviation and relative mean identity
  • Fixed formatting in some pages
  • Added to references and fixed references formatting and capitalization

[gkapfham]

Hello @kemandel you have again submitted very good materials for your research notebook. Thank you!