[Lesson 10] GANs - rociorey/cci-2020 GitHub Wiki
Extracts from lesson 10 chat:
Geomancer by Lawrence Lek
Student: I remember this movie i saw,i think it would be interesting to watch if u didnt yet its about AI who wants to become an artist 😃 Called geomancer by lawrence lek
Alexander Fefegha-Etta: vimeo.com/210494259 geomancer trailer
GANs discriminator
Student: quick question, in GANs only the discriminator has access to the data? like was the generator previously trained or it just does random stuff and it learns from the discriminator how to make what it is supposed to
Alexander Fefegha-Etta: To train the discriminator, its data come from two sources (real data ie pictures of people) and fake data (which is generated by the generator)
Rebecca Fiebrink: The generator uses information about the data through the discriminator. There isn't a sense of the generator looking at the original data on its own and attempting to reconstruct it, independently of the discriminator. However, the generator is literally created using backpropogation of error from the discriminator, so this is a very strong signal that will help the generator model the data.
It's not like it's just getting a "real" or "fake" label from the discriminator and trying to randomly change its behaviour to get a better result-- the information it gets from the discriminator is more informative then that with regards to how it should change its behaviour
Alexander Fefegha-Etta: The portion of the GAN that trains the generator includes:
- random input
- generator network, which transforms the random input into a data instance
- discriminator network, which classifies the generated data
- discriminator output
- generator loss, which penalizes the generator for failing to fool the discriminator
Backpropagation
Alexander Fefegha-Etta: if anyone who want to understand backpropagation, this is a good video - youtube.com/watch?v=Ilg3gGewQ5U
GANs generator
Student: can i just check i've understood this correctly: a generator is successful when the discriminator is running at 50% accuracy, but this 50% accuracy will feed junk feedback back to the generator which will in turn start to make it less successful?
Alexander Fefegha-Etta: As the generator improves with training, the discriminator performance gets worse because the discriminator can't easily tell the difference between real and fake. If the generator succeeds perfectly, then the discriminator has a 50% accuracy. this becomes an issue as the feedback from the discriminator becomes meaningless over time if the GAN trains past the point when the discriminator is giving completely random feedback as at if its running 50% accuracy, it is akin to flipping a coin - so the generator trains on this random feedback meaning its quality will fail good feedback from the discriminator is key to good performance from the generator as I quote @Rebecca Fiebrink "It's not like it's just getting a "real" or "fake" label from the discriminator and trying to randomly change its behaviour to get a better result-- the information it gets from the discriminator is more informative then that with regards to how it should change its behaviour"
Runway
Student: Wondering why runway ml costs?
Rebecca Fiebrink: It's because you're using GPUs on a cloud machine which can take a fair amount of electricity to run
Student: Ahh is there any way to access for example bigGAN without using runway ml website? Is the models on there open source or all locked to runway ml
Rebecca Fiebrink: Plus to buy the hardware to do this reasonably can cost many thousands of pounds. Most ML researchers (and even creative folks using ML) outside of very large companies like Google use cloud computing for this, on platforms like Amazon AWS or paperspace etc., rather than pay a huge hardware cost up front. These platforms allow you to pay only for the compute time that you use, and to have access to more whenever you need it. BigGAN is available outside of Runway but I don't know of any platform that allows you to interact with it without writing code or running on a coding platform like Colab Most of the basic model types on Runway exist outside the Runway ecosystem, though there are a growing number (e.g., GANs trained on specific datasets like the Santa dataset) that are just within Runway.
Train a model to generate 3D form paintings
Student: Do you think it's possible to train a model which generates a 3D model from a painting? I found this tool called PiFuHD which creates a 3D model from a photo of a standing human so wondering if its possible with other images
Rebecca Fiebrink: Yes, this is an active area of research in computer vision and ML
Alexander Fefegha-Etta: I was saying earlier, it seems to be a new area of development for GANs medium.com/analytics-vidhya/applying-generative-adversarial-network-to-generate-novel-3d-images-ba70e1176dac so new that most of the stuff that coming in this area was done last year - xingangpan.github.io/projects/GAN2Shape.html
Student: Would it be realistic us as students to create our own models like this? Yet to see work with this specific application, mostly different sets of real-world photo
Alexander Fefegha-Etta: I think it depends on your competency with code right and how well you get yourself immersed with it. I even think about why would you use GANs to make a 3d model, is it to create a new shape that you can't even think of
Student: Just to speed up process of having to manually model a figure from a painting, which would usually be abstract and not follow exact anatomy (usually seen in photography)
Alexander Fefegha-Etta: would the generated image take the form of 2d or 3d?
Student: Plus a lot of art is moving to crypto world at the moment, I think traditional artists are sort of left out mix as their work adds nothing new in digital space Ohh 3d
Alexander Fefegha-Etta: I guess holoGAN could attempt that crypto art is an area I been following atm loopifyyy.medium.com/the-proof-of-nfts-3-5m-beeple-drop-37955867d789 this is a good good guide - justincone.com/posts/nft-skeptics-guide/?utm_campaign=10%20Things%2020210108&utm_medium=email&utm_source=Revue%20newsletter I guess the uptake of crypto art is due to the pandemic and the hype that is associated with bitcoin atm
Alexander Fefegha-Etta: Digital Artist Beeple Sells $582,000 of Crypto Art in 5 Minutes decrypt.co/51270/beeple-nft-sale (for those who are not familiar)
GANs creating sound and text
Student: Is there any references for GANs/NN creating sound with text?
Alexander Fefegha-Etta: medium.com/syncedreview/deepmind-uses-gans-to-convert-text-to-speech-ec30500b72a3 magenta.tensorflow.org/gansynth
Resources
Student: any more resources on the ethical side?
Alexander Fefegha-Etta: arxiv.org/pdf/2001.09528.pdf
Rebecca Fiebrink: Biggest venues for AI/ML + ethics papers: dl.acm.org/conference/fat and facctconference.org (both will be accessible if you log in with UAL credentials). These aren't GAN-specific but will have a lot of relevant stuff
Rebecca Fiebrink: neurips2020creativity.github.io Here are the art submissions from previous years: aiartonline.com