How AI and XR Technology Are Changing the Face of Film

The night after my trip to the Venice Film Festival’s Immersive Island, I awoke with a frightening thought: what if I’d never been unplugged from the XR projects I’d spent the day absorbed in, and was still living inside a simulation? It’s the kind of premise Charlie Brooker would reject out of hand as too trite for Black Mirror, but the effect it had on my semi-conscious brain was real enough.

When cinema came of age as a medium in the early 20th century, people fretted about its ability to render us powerless to parse fiction from fact. With Extended Reality (XR), all these questions come bubbling back to the surface, but with an added twist: by offering us the chance to step through the looking glass and partake in experiences inside of the screen, does reality threaten to recede further into the rearview than ever before?

As a digital agnostic, the thought makes me uncomfortable. But the Venice Immersive island is a fabulous testing ground for these ideas. Taking place in an island hospital built to house plague victims in the 15th century, it’s the largest and most prestigious showcase for extended reality (XR) art in the world. 

A date with Tulpamancer, an AI project creating bespoke experiences based on prompts inputted by the user, underlines questions about XR’s role in the world to come. Users are asked to step into a room with an old green-screen computer which asks you questions about your life. Then, you move across to a second room with a chair and a VR headset, and a soothing voiceover acts as your guide through scenes from your past, present and future, as rendered by AI.

The project's co-creator, Brooklyn-based Matthew Niederhauser, says he has had plenty of feedback from users who tell him how moved they’ve been by the experience – and to allay fears about how their data might be used, participants are reminded that their information will be wiped from memory upon completing the project. But that in itself begs a question of how such a project's power might be harnessed by big tech companies, who have people’s entire life stories inscribed in the cloud.

“As artists we have definitely thought about these scenarios,” says Niederhauser. “You can start imagining what these tools can do if combined with your digital history. The implications for this data being combined with AI that can talk to us, manipulate us, show us what we want to see are scary. But the future is not yet written. We hope that through our art we can contribute to people’s understanding of the potential of these tools so that we can collectively decide what we as a society want to do about them.”

One quality often ascribed to cinema is its ability to foster empathy, by putting us in the shoes of characters whose circumstances may be very different from our own. If that’s true, then two more projects in competition at this year’s Immersive event demonstrate XR’s potential to enhance these connections. In Emperor, a 40-minute interactive experience  from Marion Burger and Ilan Cohen, you play a man with aphasia, a neurological condition which affects your ability to understand and formulate language. As you attempt to communicate with your daughter through speech therapy sessions, you become lost in moments from your past, in scenes devised by Burger as a way to feel closer to her own dad. It all adds up to a disorienting yet deeply touching experience.

Another ambitious project, Body of Mine, was developed by Cameron Kostopoulos as a way to take us inside the minds and bodies of people struggling with gender dysphoria. In it, you are ushered inside a room where you are fitted with various sensors across your body which, once you’ve donned your VR headset, give you an avatar whose movements track your own. Touching various parts of your body when prompted, you hear moving testimony from trans people about their experiences – and see yourself as a stranger in the mirror just as many of these people do.

“This kind of technology can be a game-changer for young people wrestling with their identity,” says Kostopolous. “It offers them a safe space to explore and understand feelings they might not yet have the language to express. For me, being able to connect with and being inspired by other queer stories of people in my shoes would have helped tremendously with my mental health. While a young trans girl in Texas might get kicked out of the house for wearing a dress, in VR, she can realise herself as anyone she wants, dress how she wants, even try out new pronouns.”

Indeed, a degree of disorientation is inbuilt into many of the 68 projects at this year’s showcase: one minute, you might be an abandoned robot dog left to wander a posthuman world (My Name Is O90), the next, you’re a mushroom spore floating up into the treetops, the smell of decay all around you (Forager).

For Liz Rosenthal, who has been a curator on the Immersive event since its inception in 2016, the most exciting thing about XR is “how it changes the way we create and engage with stories”. A lifelong cinephile who began her career working in film, she says she struggles to get excited by a lot of new cinema, a form she says has remained virtually unchanged for the last 100 years.

“I see [filmmakers] trying to get more and more immersive in their storytelling; it’s almost like they’re trying to burst out of the screen because they can’t achieve what they [want to],” Rosenthal explains. “I got excited about this medium because I think films have to evolve. The film business is going through all kinds of changes but it’s not all about money and engagement; it’s about how audiences evolve culturally and socially as technology changes the platforms you engage with. And so I think as an artist it’s important to adapt and try out different mediums.”

For participants, that means an opportunity to feel a wonder akin to audiences at the Lumière Brothers’ short film of a train pulling into a station, ducking out the way as it approached the screen. Perhaps my most abiding memory of the day is also the weirdest: in Tulpamancer, having shared memories of my daughter being born in response to a question about when I was happiest, I encounter a scene full of Elizabethan courtiers with distorted faces. In the middle sits a queen with a golden, protruding belly. Where on earth did that come from? It’s impossible to say, but at the Immersive island, the beginnings of a new, weird reality are already swimming into focus.