“Hollywood and Silicon Valley are in the same business: producing algorithms,” writes AI pioneer Yves Bergquist, one of a new breed of data scientists focused upon the entertainment and media business.
These scientists believe that to survive and thrive, the media and entertainment industry needs to embrace cognitive science, if they hope to compete with tech companies and address its failing business models.
They use a cluster of technologies generally called artificial intelligence (AI) or machine learning – such fields as big data analytics, deep machine learning, semantics and natural language processing, visual and auditory recognition, prediction and personalization, conversational agents among others.
They are creating software that can be taught to learn and program itself – to automate repetitive tasks and to provide insights that were never before possible.
Tech-assisted content development
One active area is content development. A studio-funded think tank, USC Entertainment Technology Center, where Bergquist leads an AI and neuroscience group is mapping box office returns against elements of the film narrative. Bergquist is working on data breakdowns of movies, as shown in this demo, the work of two Bergquist AI startups Corto and Novamente.
Greenlight Essentials, a member of IDEABOOST Network Connect, has broken down decades of film screenplays into 40,000+ unique plot elements and has analyzed over 200 million audience profiles to help filmmakers improve scripts, target audiences, and improve marketing. The product’s Analytic Terminal allows users with neither programming nor mathematics background to explore and discover repeatable patterns from decades of film data.
Scriptonomics is an ML application that breaks down movie scripts by scene, character, location, and other components. Writers and producers can leverage insights and comparisons that the tool extracts from its massive database of past successful movies to improve subsequent drafts and to aid in pitches and audience targeting. Here is an example of Scriptonomics breakdown for Titanic.
Founder Tammuz Dubnov says that Scriptonomics generates a geometric model of a screenplay – its DNA if you will -- in order to compare and improve elements as compared with financially successful films of the past. Dubnov believes that this data-driven, quantitative filmmaking process will give rise to a new generation of data-assistant content studio that will help create more hits and fewer flops.
RivetAI offers Agile Producer, a pre-production platform that automates script breakdown, storyboard, shot lists, scheduling and budgeting. Before RivetAI Toronto native Debajyoti Ray built two earlier AI startups. Video AMP, an AI-powered video advertising solution helped him to understand how much commercials owe to storytelling. So he decided to build an AI engine based upon thousands of movie scripts, both produced and unproduced that became RivetAI.
RivetAI was influenced by its production partner End Cue which had produced Sunspring, a short film starting Thomas Middleditch, and a script credited to “Benjamin, an artificially intelligent neural network." Also: “Bubbles,” an animated acquired by Netflix about Michael Jackson’s chimpanzee that Ray found while analyzing unproduced screenplays.
RivetAI’s 500 production companies customers will feed ever more data to its self-learning system to augment their storytelling efforts. Ray compares RivetAI to AutoCAD – software that began as a drafting tool, and has become a central platform for many creative professionals. To that end, RivetAI is developing products for screenwriters, corporate branded content, series television and reality shows.
Content Creation with an A.I. Assist
Computer-generated visual effects are widely used in blockbuster movies, TV shows and games. Sensei, Adobe’s AI is now being deployed across the company’s cloud platforms to automate functions and provide intelligence. The more Sensei is used, the ‘smarter’ it gets.
3D software giant AutoDesk is moving towards AI-assisted generative design, which the company used on its own new facility in downtown Toronto. Massive Software, which has been used on Peter Jackson’s massive CG films, now uses AI to automate crowd simulation and other time-sucking tasks. Its Ready to Run Agents are prefabricated AI agents that can be dropped into scenes by visual effects artists, saving time in the creation of CGI characters.
Arraiy is a well-funded Silicon Valley startup that uses computer vision and machine learning to automate time-consuming visual effects like rotosoping, to separate layers of an image to allow manipulation. The Black-Eyed Peas music video ‘Street Living’ utilized Arraiy to superimpose band members over images from the civil rights era.
The work involved in modeling, texturing, lighting, animation and performance will ultimately be automated with machine learning, says Derek Spears, Emmy winning VFX artist for Game of Thrones. “Then, the next frontier will be A.I.-driven actor performances.”
Simulating People
We’ve seen Carrie Fischer exhumed into Star Wars movies, using past performances. Now we’re seeing the emergence of simulated video and voice. Rival Theory's RAIN AI creates human-like AI for more than 100,000 game developers and agencies. Lyrebird is an tool for the creation of artificial voices. Adobe has demoed Voco, a prototype that generates speech that sounds like a specific person.
Clarifai is a platform that uses ‘computer vision,’ a form of machine learning, to help customers detect and predict demographics of faces, identify celebrities and much more. Face2Face offers real-time facial capture and reenactment. Check out this clip of a speech by President Barack Obama which he never gave.
Software pioneer Marc Canter has developed a new AI-based storytelling platform called Instigate that takes an Instagram or Snapchat story, adds intelligence and interactivity to create what he calls ‘beings’ that can be have content-enabled conversations with friends.
Canter, who developed Micromind Director multimedia authoring tools, sees Instigator as an AI authoring environment for a new form of storytelling. AI makes Instigate’s ‘beings’ more intelligent than the standard-issue bots that perform repetitive, pre-defined tasks.
The Ubiquity of AI and ML
Over time, this new layer of AI/ML capabilities will become standard for every company and every product’s technology stack and will generate billions of dollars for companies across the global business value chain. We can see how media businesses such as digital video, advertising, marketing, and VR/AR are already fundamentally driven by AI and ML capabilities.
In the end, Hollywood is just like any other industry -- as investor Ben Evans put it, “eventually, pretty much everything will have ML somewhere inside and no-one will care.”