How mixed reality is shaping a new era for content creator

PRESS RELEASE FOR IMMEDIATE RELEASE

Montreal April 20, 2021 - Neweb Labs is pleased to announce the presentation of a free online conference - Perspectives Production Virtuelle - on May 13, 2021, from 12 pm to 3 pm. The event will cover the hot topic that is virtual production: the integration of mixed reality technologies on sets.

Aimed at the community of producers, directors and technical teams in the Canadian film, television and advertising sector, this event will bring together several experts in the field of content production and technology - Epic Games, Zero Density, Ncam, Dreamwall, TFOLuv, Mtl Studios Grande and Neweb Labs.

Get more information and register

Guest Speakers and Panelists

• David Morin, Head, Epic Games Los Angeles Lab
• Thibault Baras, CEO, Dreamwall
• Paul Hurteau, DOP, MTL Studios Grande
• Mike Ruddell, Global Director of Business Development, Ncam
• Jerry Henroteaux, Real-time VFX specialist, TFO LUV
• Onur Gulenc, Business Development Manager, Zero Density
• Frederic MacDonald, Director, Strategic Development, Neweb Labs
• Catherine Mathys, Director of Industry & Market Trends, Canada Media Fund

About virtual production

Virtual production is a new production technique inspired by the gaming industry that has been adapted in recent years to the television and film industry. It allows crews - producers, directors and technicians - to preview complex scenes in a movie before filming (previs) and to have real-time rendering of visual effects directly in-camera on set. With a virtual production pipeline, technical and creative teams can streamline the entire process, make real-time decisions, reduce post-production efforts and cut production costs.

About Epic Games Lab - Unreal Engine

Unreal Engine is a complete suite of development tools for anyone working with real-time technology. From design visualizations and cinematic experiences to high-quality games across PC, console, mobile, VR, and AR, Unreal Engine gives you everything you need to start, ship, grow, and stand out from the crowd.
With virtual production poised to be one of the biggest technology disruptors of the visual medium, Unreal Engine has been at the forefront of virtual production since the first workflows began to emerge. Today, virtual production can influence every aspect of the production pipeline and it is fast becoming an integral part of film, video, and broadcast pipelines, unlocking new doors for innovators seeking to carve out their foothold in the competitive visual space. By combining CG, motion capture, and real-time rendering with traditional techniques, production teams are already discovering they can achieve the director’s vision faster than ever before, and this is only the beginning.

About TFO - LUV

Innovation is part of Groupe Média TFO’s DNA. Its Virtual Worlds Laboratory (Laboratoires d’univers virtuels, or LUV) best demonstrates this. The LUV is the only studio of its kind. Located in the heart of Toronto, this dynamic new content creation process was designed by Groupe Média TFO and its partners. This unique incubator for a whole new generation of content combines the virtual worlds in real time with live-action captures, using the latest innovations in the videogame, television broadcasting and entertainment industries. The Virtual Worlds Laboratory is also marketed so its cutting-edge technology can serve and benefit the creative ecosystem and external productions.

About Zero Density

Zero Density is an international technology company dedicated to develop creative products for broadcasting, augmented reality, live events and e-sports industries. Zero Density offers the next level of virtual production with real time visual effects. It provides Unreal Engine native platform, “Reality Engine®”, with advanced real-time compositing tools and its proprietary keying technology. Reality Engine® is the most photo-realistic real-time 3D Virtual Studio and Augmented Reality platform in the industry.

About Ncam

Ncam are the creators of Ncam Reality, the most advanced real-time camera tracker in the world. From Star Wars to the Super Bowl, Ncam Reality is used throughout the broadcast, film and live events industries by some of the biggest brands in the world to visualize photorealistic graphics in real-time. Customers include: Amazon, CNN, Disney, ESPN, Netflix, the NFL and Sky TV.

You may not have heard of Maya Kodes, but over the past year, the slender blond singer has released a dance pop song on iTunes, recorded an EP set for release in June, performed 30 concerts and amassed 5,500 Facebook followers.

Pretty impressive for a hologram.

Billed as the world’s first interactive real-time virtual pop star, Kodes is the creation of Montreal’s Neweb Labs.

In development for 18 months, she’s the brainchild of Yves St-Gelais, producer of the popular Radio-Canada TV series ICI Laflaque, and a former Cirque du Soleil comedian and director.

Kodes has already been road-tested at the company’s custom-designed holographic Prince Theatre, where she performed a fluffy dance-pop confection called “Boomerang” — the song currently for sale on iTunes — amid a flesh and blood dance crew and in front of a largely tween audience, kibitzing with the show’s emcee in a fromage-filled routine that bordered on juvenile.

It’s also the venue from where Kodes will present, on June 13, a 360-degree.

A Toronto performance is expected before the end of the year, says Neweb spokesperson Élodie Lorrain-Martin.

“We really want to do a world tour and also have her perform in multiple venues at the same time,” says Lorrain-Martin.

Since holograms have already performed — late rapper Tupac Shakur was resurrected at Coachella in 2012 as was the late Michael Jackson at the 2014 Billboard Music Awards — Neweb’s ambition is plausible.

Onstage, Kodes is a chameleon of special effects, changing costumes and skin pigmentation in the blink of an eye; shooting sparkles from her hands and instantly cloning herself into a Maya army.

But what makes Kodes stand apart from Shakur, Jackson and the 3D Japanese anime star Hatsune Miku — who has already toured Japan and North America as a singing hologram — is that she isn’t strictly a playback apparition.

“Every other holographic project on the market right now is all playback,” Lorrain-Martin says. “She’s the first one in the world to be able to interact in real time.”

How Kodes manages to do this involves a bit of smoke-and-mirrors: the onstage Maya is embodied by two people hidden from view.

The first, Erika Prevost, best known as Sloane on the Family Channel’s The Next Step, portrays the physical Maya, her choreographed movements relayed by motion capture cameras and a bank of five computers as she cavorts with onstage dancers. Prevost also provides Kodes’ speaking voice, keeping an eye on her audience through an onstage camera, which lets her gauge the crowd’s response and even indulge in a Q&A if desired.

Then there’s a woman who provides Kodes’ singing voice, whom Lorrain-Martin refuses to identify.

Kodes’ virtual existence even has a back story.

Copyright Toronto Star - By Nick Krewen Music

In some respects, the future of French-Canadian pop music resembles many present-day divas: blonde, slim, attractive. But there’s one very important difference between Maya Kodes and the Keshas of this world—Kodes is a hologram. And while she may not be the world’s first virtual pop star, she’s the only one so far who moves, sings and talks to people in real time.

Kodes is the brainchild of animator Yves St-Gelais and his Montreal-based start-up, Neweb Labs. She has a prototype of sorts in Miku Hatsune, a Japanese virtual pop star with a huge back catalog and a large and breathless following. But unlike Hatsune, whose performances are prerecorded and played like a DVD, Kodes delivers her pop routine live. So far, Kodes has performed around 30 times. Marketed toward tweens, each gig marries uncomplicated dance-pop with digital pyrotechnics, ranging from color-changing costumes to the dazzling electronic sparks that shoot out from her digital palms.

Like any musician, each performance is subtly different, and the product of much work behind the scenes. It’s just the nature of that work that’s a little different: Off-stage, dancer Elise Boileau provides the moves, which an army of designers, illustrators and computer programmers translate into digital gestures. Kodes’ songs are also performed live, though the identity of the human behind her voice is guarded under lock and key, to maintain the illusion. She can even answer questions from members of the audience, or respond to what’s going on around her. 

Of course, she isn’t perfect. The animation recalls the video game The Sims, and even her “look” is fairly derivative of the most basic kind of Disney princess. But St-Gelais says this is just the beginning. Already, he’s adjusted the design in response to negative feedback from women about her pneumatic figure, which he characterized as “a man’s version of a woman.” Now, he envisages a future for Kodes with multiple characters, an updated aesthetic, and live performances taking place simultaneously all over the world. “That will bring another form of communication, of intimacy in which Maya Kodes will be the core element,” he told Quartz.

For now, the important thing may be building a flesh and blood fan base to support this virtual star. So far, Kodes has an EP, a backstory, and an effervescent Twitter presence—but she’s still awaiting the one key to success that can’t be conjured up online.

Copyright Quartz - Natasha Frost, Reporter

Inside the Montreal company engineering virtual-reality pop star Maya Kodes.

A trip to the South by Southwest festival in Austin, Tex., this month could mark the start of a mixed-reality phenomenon.

Near the turn of the millennium, Yves St-Gelais had a vision. A part of the founding team behind Montreal entertainment juggernaut Cirque du Soleil and a fledgling animation producer, St-Gelais conceived a mixed-reality future where celebrities were not fallible objects of fetish, but ubiquitous, geo-customizable and interactive avatars.

At the heart of this vision was a pop star – a force for good who would speak primarily through music, the world's lingua franca, as well as interact with her audience in their mother tongue, a feat he describes as "bringing art to the soulless." He would name her Maya Kodes and he envisioned a superhero's arc, with story beats similar to that of Spider-Man. Early demos proved promising but fruitless. Never mind the animation quality or the music, computers simply were not fast enough to create a viable, likeable, real-time reactive humanoid.

Two decades later, St-Gelais believes the tech has caught up with his vision.

Armed with a story and a songwriting team, 12-person production crew headed by an Emmy Award-winning animator, her own A&R man, a mountain of proprietary software and hardware, a touring holographic spectacle and two EPs of chart-ready pop songs available on iTunes and Spotify that bring to mind Sia and Lady Gaga, Kodes (pronounced "codes" as in computer, not "kode-es," like a Russian spy) is already the world's first interactive virtual singer. This month, starting with a trip to the South by Southwest festival in Austin, Tex., St-Gelais will set out to discover if Kodes can become its first mixed-reality phenomenon.

On a recent morning, a cheery crew of technicians, programmers and performers arrived at the office of Neweb Labs, which St-Gelais founded in 2015, to run a dress rehearsal for the touring show. A small ground-floor unit in a sprawling commercial park surrounded by body shops in Montreal's industrial Côte-Saint-Paul neighbourhood, the studio space – bifurcated into a test theatre and a work area – is hardly the slick cradle of innovation one might expect from such a company, but charmingly reflects the scrappy mentality of its occupants.

Trickling in, the team sipped coffee "borrowed" from an upstairs cafeteria while setting up their respective work stations. For the technology-inclined, this meant helming computers, assuring programs were booting and code was running smoothly, while others slipped into bodysuits strapped with ball-sensors or donned a head rig resembling the skull of an aardvark.

Gently prodded by Neweb's artistic director, Véronique Bossé, the group – all women save for a technical artist – exchanged knowing glances as the hum of processors grew louder. Before long, a blond humanoid appeared on multiple screens inside the studio and began shadowing a body-suited performer's routine.

"There she is," Bossé exclaimed, addressing the nearest screen. "Hello, Maya."

On the surface, a virtual pop star may seem tame compared to the recent holographic resurrections of legendary artists such as Tupac or Bob Marley. What sets Kodes apart is that she's being performed and rendered in real time.

Kodes's team performs this "soul insertion" through the synchronization of highly guarded proprietary software and a puppeteering duo: Adancer in a motion-capture rig controls her body, while a stationary singer handles her facial movements and voice; both have audience monitors allowing them to react and interact with whatever's happening in the theatre. This occasionally allows Kodes, who is projected onto a holographic screen stretched out across the stage, to slip out of human form and interact with her real and digital surroundings, allowing her to participate in elaborate, otherworldly stage routines – multiplying herself in a psychedelic haze, for example – then answer off-the-cuff audience questions.

At Neweb, Kodes is referred to as if she were actually in the room, not unlike when addressing Santa Claus around children. And, like Santa, she comes with her own convenient if slightly confusing back story: Created to combat the Y2K bug, a glitch turned her code to human. Greeted to the world with music, the new form henceforth documents its existential anthropological study via song.

Arriving at the tour dress rehearsal fresh from the Pollstar Live! convention in Los Angeles, St-Gelais is reservedly excited. In his late 50s, soft-featured with dark-rimmed glasses, he appears like a cross between Walt Disney and late-era Steve Jobs and speaks softly with a pronounced Québécois accent.

A veteran of the entertainment field, St-Gelais now splits his time between Neweb and its big sister, Vox Populi Productions, which has produced the Quebec political satire series ICI Laflaque for the past 15 years. It was working on Laflaque, which uses quick-turnaround 3-D animation to skewer politicians on a weekly basis, that convinced the budding producer that creating real-time, interactive animation was not only possible, but likely the future of entertainment.

In that sense, Maya Kodes is very much St-Gelais's Mickey Mouse, a likable emissary he can use to capture the imagination of a new generation and launch an empire of interactive holograms and AI bots. But first, he has to create his Steamboat Willie. To do this, he's bet big, putting his own money alongside that of private investors into Neweb, as well as receiving $1.2-million from the Canadian Media Fund to have Kodes and her team perform at industry showcases and 1,000-person-capacity venues in Switzerland, Sweden, Germany, Canada and the United States. The future, he says shortly after the rehearsal wraps, is interactive, global and beyond the capabilities of a single human. After all, there can only be one Britney Spears, Beyoncé or Taylor Swift, and each comes with their own very human problems. In contrast, multiple versions of Kodes can work 24/7 around the world without the fear of burnout or language issues.

"We had every big name in the Canadian music business pass through here. They loved Maya, but they said they didn't know what to do with her," St-Gelais explains. "The traditional music-industry model is just that: traditional. [Kodes] is fundamentally an attraction. The market model is not Warner, Sony or Universal, it's Cirque du Soleil."

If Neweb's mission appears out of movies, books or literature, it's because it more or less is. A 2013 episode of Black Mirror titled The Waldo Moment featured an interactive animated avatar running for Parliament; in the 2002 film S1M0NE, a successful director secretly creates an interactive character to star in his films; and, perhaps most famously, Dr. Frankenstein created a humanoid experiment in Mary Shelley's updated Prometheus myth. In all cases, the creations – often born of frustration and their creator's own existential dread – ended up turning on their makers, leaving them desolate or dead.

When I bring this up to St-Gelais, he laughs it off. Kodes is not an extension of himself as much as an evolution of a movement, he explains. The next generation of tech-savvy consumers will not only want to be entertained but also be part of their entertainment – whether through augmented, virtual or mixed reality.

And judging by market trends, St-Gelais has a point.

According to industry forecaster Digi-Capital, augmented reality and virtual reality combined will be a US$110-billion business within five years. And the entertainment sector, second only to gaming, has been a maven in the industry. Although headset technology is still too nascent to create a full-on tipping point, industry trends indicate music will be a leading way in for Generation Z, due to both its sensory immersion and short running time. But while some forward-thinking artists, such as Future Islands, Black Eyed Peas and Run the Jewels, have released VR experiences, for the most part, marquee pop stars have stayed out of the sphere – preferring to keep as much manicured control as possible.

"The technology is a factor, but so is the marketing," explains Jon Riera, one half of Combo Bravo, the Toronto-based creative team behind VR videos for A Tribe Called Red and Jazz Cartier. Riera argues that until the marketing budgets match those of traditional video and there's a centralized digital location where audiences know they can turn to to find music-based VR, the medium will always suffer from the "cheesy-wow" factor. However, he points out that at least one big-name, owl-loving Toronto artist has expressed an interest in working in the medium. "He just needs Apple to release its own VR headset."

Harold Price at Occupied VR, one of the leading companies working in the field in Canada, sympathizes with Riera's argument. We've already accepted the tools to this future – AI assistants and ARKit-enabled phones – into our home, he says. "What we need is a 'must-see' cultural moment. We need our Star Wars."

St-Gelais believes Kodes could be such a moment. "Look at Hatsune Miku," he says. "She plays to 200,000 people in Japan. And she is just a playback."

Miku, whose name literally translates to the voice of the future, is Kodes's most obvious predecessor, and much of the expectations of avatar-based musical success can be traced back to her effect on the industry. Invented as a marketing avatar in 2006, Miku is a "voicaloid" – meaning her voice is competently synthesized – who shot to fame in the late aughts, selling out stadiums in Japan, opening for Lady Gaga's 2014 North American tour and even appearing on The Late Show with David Letterman. As St-Gelais points out, Kodes, who currently has no comparative competition in the market, can not only do everything Miku can, but also be adapted for different geopolitical regions, while feeling just as organic and interactive in each one.

"For me, holography is not the last platform," St-Gelais says, revealing that plans are already in place for a weekly 30-minute talk show with live Q&A on Facebook. "The next step is connecting her to an artificial intelligence, so it's possible to have Maya on your phone or virtual reality."

First though, the Neweb team has to focus on bringing Kodes to the world "in person." The exercise almost already feels antiquated, but St-Gelais has been waiting nearly 20 years for the technology to be ready and he's not going to let this moment pass.

"We're on her ninth body," Bossé says, pointing out the speed in which the software develops, while mentioning there's already talk among the crew that the 10th will be a little more reflective of "real" women.

While the pomp and circumstance surrounding Kodes and Neweb's ambition is admirable, the question of her success still rests on finding a dedicated audience. And, for all the talk of connecting to international, tech-savvy youth waiting for tech to catch up to its demand, for me, Kodes's and Neweb's success lies squarely in its ability to appeal to children. Always the salesman, St-Gelais is keen to discuss the strong international reaction so far.

But it's when I ask him why he created Kodes that his mouth curves up and he affects a Disney glint. "I have children," he says in his Québécois accent. "I make her for my children."

© Copyright 2020 The Globe and Mail Inc. All rights reserved.

Ready to step into the future with a very different kind of performance or brand activation? Check out interactive holographic pop artist Maya Kodes, an innovative superstar from Neweb Labs.

Neweb Labs uses artificial intelligence, holography and motion capture technology to produce 3-D animations and virtual personalities like Maya. “Our augmented reality productions create a real emotional connection to audiences and deliver tremendous wow factor,” says Candace Steinberg, Neweb Lab’s marketing and sales director. “We leverage motion capture for real-time interaction.”

Maya Kodes is a holographic superstar created by Neweb Labs

Maya, hot on the heels of her Boomerang CD release, is an on-stage dynamo, entertaining with spectacular special effects, lively, in-the-moment banter, and choreographed sets alongside her human back-up dancers. Behind the scenes, it’s Neweb’s crew working her moves and on-point audience interaction via the body-motion capture suit and live voice audio that brings her to life. The crew works her moves and interacts with the audience to create an immersive experience. In addition to Maya, Neweb develops live interactive 3-D holograms, including living or deceased celebrities and public figures, and animated caricature-based characters, for any type of custom-branded event such as conferences, trade shows or product launches. The company also provides artificial intelligence technology that pairs a human interface with a holographic terminal or mobile app for virtual customer service greeters, tour guides, and more.

Maya has taken the world by storm. With her incredible voice and cutting-edge dance moves, she wows audiences everywhere she goes. In a future filled with endless possibilities, Maya Kodes is one star that shines bright. Some people say that holographic performers are the future of entertainment, and she is certainly leading the pack. She is sure to keep fans coming back for more, and her live shows are not to be missed! If you're lucky enough to see her in concert, you'll be blown away by her presence and stage presence. Maya is truly a one-of-a-kind performer, and we can't wait to see what she does next!

Click here to watch Maya Kodes.

Copyright Karen Orme TSEvents