10_colab

Introduction to RunwayML

Session A: Hosted Models and Networking

Objectives

• Learn how to integrate Colab with JavaScript and other software applications.

Resources

• Hosted Models
• Interact with models in Colab over the network

Session B: Generative Adversarial Networks, Semantic Maps

Objectives

• Understand the basics of how the GANs and Image Colorization models work
• Develop high-level understandings of nonlinear dimensionality reductions and the latent space.
• Understand how GANs can be applied to interactive systems to generate imagery.
• Slides

Additional references

• Google's A.I. Experiments: Visualizing High-Dimensional Space
• Using Artificial Intelligence to Augment Human Intelligence by Shan Carter (see the first three sections for latent space descriptions)
• Octavio Good on Generative Adversarial Networks (21:54 - 28:35)
• GAN Lab by Minsuk Kahng, Nikhil Thorat, Polo Chau, Fernanda Viégas, and Martin Wattenberg
• In the Age of A.I., Is Seeing Still Believing? by Joshua Rothman

Creative GAN projects

• Playing a game of GANstruction by Helena Sarin (2019 Eyeo Talk]
• Misremembering and Mistranslating: GANs in Art Context and Fall of the House of Usher by Anna Ridler
• Using AI to Produce “Impossible” Tulips: Anna Ridler uses AI to bring “tulipmania” into the future by Elain Ayers
• Mario Klingemann’s Neurography: Cameraless Photography with Neural Networks
• Booksby.ai by Andreas Refsgaard and Mikkel Thybo Loose
• Unfinished by Roman Lipski
• Blackberry Winter by Christian Mio Loclair
• Meet AICAN, a machine that operates as an autonomous artist

Semantic Maps / Image Synthesis

• Semantic Image Synthesis with Spatially-Adaptive Normalization, original SPADE paper paper
• GAUGan online demo
• Learning to See by Memo Akten
• Uncanny Road by Anastasis Germanidis and Cristóbal Valenzuela
• AI Lab Workshop: Painting Landscapes with the Body

Code Examples

• Glitch! RunwayML template - StyleGAN
• Glitch! RunwayML StyleGAN walk
• p5 web editor: send text to RunwayML: Generate an image with AttnGAN
• p5 web editor: send webcam image to RunwayML: Photo booth with Fast Style Transfer
• p5 web editor: send uploaded image file to RunwayML: Generate a caption with im2txt

Assignment (Due Sunday, Nov 9):

1. Create a p5.js sketch that receives data from RunwayML (using any model). You can use this glitch RunwayML template which hides the keys in a .env file. (If you work with the web editor only, be careful about leaving your token in your code and the model active in the RunwayML interface!)
2. Optionally, send data to RunwayML or use another programming environment or software tool besides p5.js.
3. Continue your reflection on RunwayML in a blog post. How is working with RunwayML from your code compared to the web interface? Include screenshots and screen captures of your workflow.