Avatar 2: What is the future for visual effects

Thirteen years later, Cameron, his production company Lightstorm Entertainment and Wētā have returned audiences to the world of Pandora with Avatar: The Way of Water, the first of four planned sequels in the franchise. It’s once again a photorealistic game changer, with awe-inspiring CGI feats to put the superhero movie industrial complex to shame. And, according to Jon Landau, Cameron’s producing partner on the Avatar franchise – as well as on Titanic and Alita: Battle Angel – the course of this instalment was charted shortly after the original film’s release.

“In 2010, after the movie came out, we held a retreat where we brought in our department heads, and said, ‘If we’re crazy enough to want to make a sequel or sequels, what are the things that we need to improve upon in our process that allows us to create an even better, immersive cinematic experience for audiences?'” Landau tells BBC Culture.

Reaching the next level

First on the agenda was the facial performances. For Avatar, Wētā used their FACET system to capture distinct details of the actors’ faces  – from Zoe Saldaña as native Na’vi warrior Neytiri to Sam Worthington’s marine Jake Sully and Sigourney Weaver’s xenobotanist Grace – when in their Na’vi avatars (genetically engineered Na’vi bodies that human consciousness can inhabit, in order to interact with Pandorans and survive the atmosphere). By putting the actors in performance-capture suits featuring reflective markers, as well as attaching a head camera rig to them, the VFX artists were able to intricately track how muscle, hair and skin moves in real time so that they could be mapped on to their virtual characters.

For Avatar: The Way of Water, major advancements – such as the use of more facial cameras to capture the actors’ expressions and an increased number of polygons for the method of texture mapping on to a computer-generated shape – allowed for even more infinitesimal details of the actors’ performances to come through in their digital avatars in close-ups, both on land and in the water.

“We wanted to capture more nuanced, high-fidelity performances, to realise them in even greater photo-reality than we did on the first movie,” says Landau. “That was an approach that involved a lot of people. It wasn’t just Wētā effects. It wasn’t just people on the production. It was those two working together to create more captivating characters on the screen. We were very focused on creating an actor-director-centric process.”

Using performance-capture stages in Los Angeles, in September 2017, Cameron began directing his actors before any live-action scenes were shot, capturing as many as 23 members of the cast simultaneously for particular sequences. The original film was around 70-75% computer generated but for this sequel that percentage is said to be 90%. If a Na’vi character was interacting with a human character in a scene – like for example, Miles “Spider” Socorro (Jake Champion), the human boy who lives with Jake (now a fully-embodied Na’vi) and Neytiri – the latter actor would be on set to perform with the performance-capture stars. “Jack Champion played all of those scenes that you see him in twice,” says Landau. “He played them once when we were capturing Sigourney [Weaver as Kiri], Stephen [Lang as Colonel Miles Quaritch], Sam, Zoe and the kids. We weren’t capturing his performance but he was there for them to react to and then to act with. Then, a year and a half later, we brought Jack to New Zealand, [and] had him do the live-action portion of that shot.”

The live-action sets were constructed in studios between the city of Wellington and the rural town of Kumeu, with production kicking off in 2019. But rather than relying on a tennis ball on a stick to provide an eyeline guidance for Champion and other actors to perform opposite, in lieu of castmates playing digitally-rendered characters, they created a cable system “like you might see at a sporting event, [strung up] over the stadium with a camera attached,” says Landau. “We replaced that camera with a monitor and a speaker and then we programmed them to be in the correct spot where the actor was for their CG character, so they can have the correct eyeline and the correct performance [to act against]. Then we married those two [performances] seamlessly.”

The biggest challenge is in the title: The Way of Water. The storyline sees Jake and his family targeted by human colonisers, who want to make Pandora the new home for Earth’s populations, so they flee their forest-based home and seek refuge with an ocean community of amphibious Na’vi, known as Metkayina. This means the vast majority of the film takes place in, under or around water, which involves a new level of environmental and technological understanding. “Light travels differently underwater,” says Landau. “So we needed to create a whole different capture system to work with the reflective markers and a different system above the water, and for those to work together.”

During pre-production, a research trip to the Bahamas gave the camera operators and digital engineers the chance to dive into these uncharted waters of CGI. “We had people go underwater on SEABOB water sleds that were propelling them at the same speed as the Ilu [water creature] would,” says Landau. “How does their face ripple? What is the water impact on their face when they come out of the water? We put them in wigs so that when they were underwater and going at a certain speed, we could see how their hair moved. Even the bubbles when someone dives in the water: what do those bubbles look like around them? And that’s not something you can do without an artist’s eye to figure that out.”

The actors were flown to Hawaii to take part in a breath hold training camp so they could shoot as much as possible of the underwater scenes in actual water tanks (rather than using CGI to create the illusion of being in water) with a “sense memory” of what it was like in the ocean. “They’re wearing the suits with the markers and they have head [camera] rigs on their faces, and we’re capturing their performances underwater,” explains Landau. “All the crew who were in the tanks with them also had a breath hold because we couldn’t have the bubbles in the tank.” Kate Winslet held the record in the cast for holding her breath for just over seven minutes though that was not while exerting herself in scenes. “That’s static apnea,” Cameron explained to Deadline. “She’s not swimming around, she’s face down, you go into a Zen trans-like state; you slow your heart rate down. She was taught how to do that.

“Now, what that translates to in practical terms is about two-and-a-half to three minutes of swimming and acting. Active brain function uses up a lot of oxygen; swimming and moving around uses up a lot of oxygen, so you don’t get those kinds of times in an actual scene.”

3D is another major element of Avatar’s visual world-building, Landau says was designed to create a “voyeuristic” experience for the audience. To achieve this, Cameron made use of a system called Sony CineAlta VENICE 3D where the camera is rigged to a specially made 3D stereoscopic beam splitter. “The goal of 3D is to have the screen plane disappear for the audience and to create a window into a world so that you forget that you’re sitting in a theatre,” says Landau. “We don’t want the [images] coming off the screen, because [if that happens] you’re interrupting the suspension of disbelief. If you can put them underwater with the Sullys, put them at Jake’s speech, you’ve done something that immerses the audience more in your narrative storytelling. For us, 3D is [even] more important in a dramatic scene than it is in an action [one].”

One of those action sequences in The Way of Water involves a train raid which combines 3D and higher frame rate shots played back at 48 frames per second to create a smoother and more realistic image than the standard 24fps. “That is the level at which Wētā is able to render high dynamic range shots at a high frame rate that you go, that is like a real train crashing,” adds Landau. “That’s a real person there and a close-up. Those are the things that are important to us.”

As are so-called flaws in the cinematography. VFX artists have the ability to streamline scenes in order to remove any lens flare or camera wobble, but that would hinder the sense of grounded reality that Cameron and the team wanted to achieve. “Visual effects can be a perfect art form but filmmaking is not,” says Landau. “We allow some of those imperfections to come through. They’d ask, ‘You don’t want us to fix the camera?’ No, because that wouldn’t be real. If I was really operating a camera and a truck was driving by me, I’d flinch. So that’s the cameraman flinching to represent that. We want those imperfections in it to make it feel real, cinematography wise.”

A troubled industry

The Avatar sequel comes at a time of great debate in the VFX world about the working environment and treatment of VFX artists in the blockbuster movie space. Untenable working hours, tight deadlines and late-stage edits have allegedly caused rampant burnout for VFX company employees trying to keep up with the unrealistic expectations from studios producing an increasing number of CGI-heavy movies and series. Superhero projects have reportedly caused the biggest headache, with Marvel Studios (a subsidiary of Walt Disney Studios) being called out by VFX artists through news outlets and otherwise, claiming execs make increasingly complex demands but don’t adequately compensate for the multiple new renderings that come from endless studio notes. Scott Squires is a veteran visual effects supervisor who worked for Industrial, Light & Magic (ILM) for 20 years and whose credits include The Mask and Star Wars: Episode 1 – The Phantom Menace. Squires tells BBC Culture that while a positive for the VFX artists working in the industry is the sheer amount of work out there – “digital visual effects have provided incredibly powerful tools for manipulation and creation by VFX artists [and] that has created an explosion of  VFX in anything and everything” – the downside has been “more exploitation”. “When I started out, those of us working in visual effects on large films were employed directly by the studios and were in Hollywood unions,” he says. “Today there are no VFX unions so no protections. No one making sure overtime isn’t crazy [and] that overtime is paid [for example].”

Talking to the Guardian in August, award-winning VFX artist Joe Pavlo spoke of his experience working on the Marvel film Guardians of the Galaxy, describing it allegedly as “a mess” and “crazy”. “The visual effects industry is filled with terrific people with lots of goodwill who really care but, at the end of the day, there’s nothing in place when their backs are up against the wall and Disney is making crazy demands,” he added. Working on a Marvel production was the straw that broke the camel’s back for another visual effects professional who agreed to speak with BBC Culture anonymously, having now left the industry. “I was doing seven days a week, 10 to 12 hours a day,” he alleges. “We were redoing the last act of the movie, a month or two before it was delivered; they’re doing it so late, but they don’t shift the [release] dates around at all. Marvel projects make every [VFX] studio underbid or compete, so they have very low margins. You have to work with a smaller team [than on most blockbusters] as well. It’s very high stress because everyone has to do more and work longer hours. It was six months of that and I just preferred a work-life balance.”

A second VFX artist BBC Culture spoke to, who has worked on multiple Marvel film and TV projects, and also asked for anonymity because they still work in the industry, claims too that studios like Marvel and Disney facilitate less than ideal working conditions for VFX artists and companies. “For the most part, nearly every major client delays decisions, changes their minds, gives muddy direction in a tight timeline, but Marvel does all of those things consistently and to a higher degree of intensity,” he alleges to BBC Culture. “Institutionally, their interactions with visual effects vendors have gotten worse and worse over the years because of their ever-changing direction, constant re-writes and U-turns, all of which need to be turned around under smaller and smaller timelines. The result for visual effects artists is major overtime for months at a time.”

BBC Culture has reached out to Walt Disney Studios for comment on all the above allegations.

The first anonymous VFX professional also criticises the lack of understanding by certain directors who don’t engage with or offer creative instruction to VFX artists tasked with generating complicated sequences. “After it’s done shooting, they are barely involved,” they explain. “They’ll get the [VFX shots] at one point and give notes, but it’s sort of like a factory floor. Sometimes you get a director that isn’t familiar with the whole visual effects work so they want finished quality right off the bat. I’ve worked on a lot of projects where they asked for finished frames and that’s just not how it works.”

He does, however, point to filmmakers like Cameron who have a far better understanding of how visual effects work, and collaborate better with VFX teams in post-production. “James Cameron and David Fincher are guys that go in with a very strong vision and they plan it out pretty meticulously. They’ll have changes from it, obviously, but they still have a great idea,” he says.

For Avatar: The Way of Water, Landau says there was a strong collaboration between the filmmaking and visual effects departments throughout the shoot to ensure the VFX artists had exactly what they needed to realise Cameron’s vision. “We gave them the exact camera angles we wanted, the exact performances at a lower resolution, and then expected them to make it look photographic,” the producer says, adding that access to Wētā’s internal computer network was a big help to give feedback. “They allowed me to be on that same system with them so that I could do up to seven hours of [footage] reviews a day and challenge them: ‘Why is it this way? Is that the best way to present it to Jim?'”

Over a decade in the making, Avatar: The Way of Water can’t be accused of being a rushed production either. The release date was shifted from 17 December 2021 to 16 December 2022 to accommodate the creative demands, especially with the pandemic causing live-action production to be delayed. “We needed the [extra] time to do it,” says Landau. “Someone said to me once, ‘when will the movie be finished?’ I said, ‘the day before it comes out’. [Production senior visual effects supervisor] Joe Letteri, [senior visual effects supervisor] Eric Saindon and the rest of the team at Wētā effects, they wanted to push themselves so they never wanted to rush anything. There are shots that would take two to three weeks to render, but they wanted to do it to get it right and that’s why we love working with them. They are about delivering a calibre of excellence.”

However the VFX outfit has also not been free from criticism when it comes to its historic workplace culture. A 2020 independent review by Miriam Dean QC, where more than 200 current and former staff were interviewed, found that staff crew were not protected from bullying, sexual harassment and discrimination, and that its HR practices, policies and processes were “immature”. Eighty complaints about bullying and 120 of poor conduct were made. Company founders, Sir Peter Jackson and Dame Fran Walsh, assured employees at the time that they would “action the recommendations and build the culture Wētā Digital needs to maintain its status as a leader in the industry”.

A leader it is, and the innovative VFX  it has created for Avatar: The Way of Water ranks as one of the most impressive VFX achievements of the past 10 years, alongside Industrial Light & Magic (ILM)’s Stagecraft units, as seen in Star Wars series The Mandalorian, a virtual filmmaking environment using LED-based wall displays where directors can see in-situ what the digital backdrop of a scene will look like, and direct camera angles and actors accordingly. “The coolest things coming out recently,” adds the first visual effects professional, “is the adoption of AI to accelerate work. There’s a lot of machine learning going on, especially for the recreation of actors. James Cameron talked about it a lot for Avatar, [of the potential] to make the Na’vi where you’re taking the actor’s performance and then retargeting it onto the CG model. That takes a lot of engineering, and Wētā especially has a lot of great research in that area.”

Squires agrees about “new technology enabling VFX artists to make an even higher quality level of work and provide many more options” for creativity. “Beyond digital development, including computer graphics, in the 90s, today there are LED volumes, real-time rendering, AI-associated tools, and more,” he adds. “Much of the very time-consuming and tedious hand work VFX done in the past has been at least partially automated.”

Yet as advancement in technology gives filmmakers greater freedom to realise their wildest cinematic dreams, progress needs to be made to create a better, healthier working culture for the VFX artists to deliver increasingly complex demands. “With high-speed video communication and the expansion of VFX companies around the globe, it means someone, somewhere is working on the VFX of your film at any hour of the day,” says Squires. “Those supervising and monitoring the work may now have to have reviews at 8am, 5pm, midnight and 3am.”

He suggests unionisation as the solution, and that seems to be the consensus for making the VFX workplace healthy again. “The lack of international unionisation and any cooperation by the visual effects shops is hurting workers,” says the second VFX artist. “Visual effects shops frequently underbid each other, and Marvel (and other clients) know this and pit them against each other, consistently leading to massive overtime and burnout from the artists,” he alleges, “and then the visual effects shop eats losses, and the cycle begins again.”

“Right now, there’s no incentive for a visual effects studio to give their employees better hours, better benefits or a better life balance,” adds the first VFX professional. “There’s no way for a single employee to push back but if you get a bunch of them, VFX studios have to prioritise that.”

Avatar: The Way of Water is out in cinemas internationally now

Love film and TV? Join BBC Culture Film and TV Club on Facebook, a community for cinephiles all over the world.

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.

And if you liked this story, sign up for the weekly bbc.com features newsletter, called The Essential List. A handpicked selection of stories from BBC Future, Culture, Worklife and Travel, delivered to your inbox every Friday.