AI-painted BG + Traditional Character Animation
This is a proof of concept I made to test using AI for animation. The aim for this is to create a clip that uses a deep learning model but preserves a consistent background, character motion, and Japanese anime art style.
This shows promise for quick, not-too-artistic animations like product commercials where the backgrounds and details do not matter too much since the target is more for the general audience and a faster production cycle.
However, I think this technology is still not mature enough to be used for more art-centered films, since creating backgrounds isn't merely basing it on a real location and copying. There are a lot of complex processes such as addition of secondary objects for character and scene exposition (e.g. - for a scene where a war veteran is introduced, it pays to add a lot of "hints" to the audience such as a flag, guns, photos in the background of the character's room) where simply drawing them or creating 3d models from scratch gives more control and breathing room to the creator without having the need to buy actual objects. Moreover, the level of control for lighting and shadows is also diminished.
Also, most importantly, licenses. Most AI models are licensed for non-commercial use only, and even the training data for these high-quality AI models are usually taken from copyrighted material (the dataset used to train the model in this post uses a lot of Makoto Shinkai's backgrounds from movies).
For Japanese anime specifically, I think AI is good for research in its current state, but current technology especially with regards to licenses are still very restrictive. I'd give it more time to mature before using in actual production, so I will be focusing on learning and reviewing other more manual skills such as anatomy, perspective, creative writing, and direction rather than deep learning in the meantime.
AI model was based on research by Chen et al.