Flickers of Light in Dark Times — Motion Capture Live Performance
A mocap beginner’s guide for putting on a live performance. 🤔🤔🤔
Overview
We were a team of four of New York University (NYU) Integrated Digital Media (IDM) and Interactive Telecommunications Program (ITP) students grouped together as Team Ono to showcase a live motion dance capture performance.
The Premise
Flickers of Light in Dark Times is a live motion capture dance performance about getting crushed by the adversity in urban life, and regaining strength by accepting kindness and support.
The piece plays with notions of the visible and invisible structures that surrounds us in urban life. The tracked dancer navigates the spaces in between the audience members, vexing from completely ignoring them, and acknowledging their presence as a source of strength. The audience members takes an active role as they offer her support through motion tracked wisps that rattle and sprout lights in the darkness.
With Flickers of Light in Dark Times, the artists create an immersive live experience through movement, projected interactive 3D graphics and music, crafting a cohesive narrative with cutting edge technology.
Team Ono and Timeline
Once we were assigned into groups as Team Ono, we had 1.5 months time span working together until the performance on December 14th. The team was named after Yoko Ono a multimedia artist who is known for her artist performance. The first half of the semester was focused on understanding the basics of motion capture and integrating three different softwares — OptiTrack Motive, Unreal engine(UE4) and Maya. The second half was focused on a the class final assignment, which is to create a live motion capture performance.
In our first meeting, we brainstormed ideas and discussed how our skill sets can contribute to this piece in UE4. We realised there were obstacles of assigning roles when everyone has diverse backgrounds and working together— Nancy and I (Man-Ping) are IDM graduate students with UX and graphic design backgrounds, Chester is an IDM undergraduate student who is a 3D modeller and Louise, an ITP graduate student, is an experienced interaction designer who is proficient in programming, installations, and Unity. We naturally leaned towards Louise as the leader as she has the most experience in the field, however we were all determined to take on the challenge and gather all the needed hands for the project.
Nancy Chiu — Technical Director
Louise Lessel — Director of Mocap and Tech Integration
Chester Ma — Director of Graphics and Model Design
Man-Ping Wu — Art Director.
In addition, we partnered up with Aaron Moreno, a Sound Designer from ITP, and Katie Brady, the Performer from Barnard College.
Tasks at hand
With a big task at hand, we split our responsibilities into different aspects of the piece. Louise was in charge of being the project manager with handling the macro tasks with the technology, world integration and communicating with Aaron and Katie. Chester was in charge of applying the particles on to the dancer. Nancy was responsible for three wall interactions and reaction setups. My tasks (Man-Ping) involved being responsible for the promotion materials, rigid bodies and wand particles.
Aaron helped by incorporating sound design that allows the performance space to become an immersive environment and Katie was responsible for the choreography.
Storyboard
Our piece’s premise is to demonstrate one getting crushed by the adversity in urban life, and regaining strength by accepting kindness and support. The storyboard went through multiple alterations after gaining feedback from our professors and industry professors tweaking the piece into something more cohesive and fascinating to the audience members. Little details were asked and explained to test whether or not our core message would come across.
Mood board and Ideation
During the process of storyboarding, we gathered and compared our individual mood boards to see if we all had a collective art direction for our projection design. We all collectively agreed to work towards a soft floating-like dim water particle effect to outline for the particles of the dancer and wand particles. In addition we wanted other visual elements to be simple like the white vector outlined walls and black background, giving a better contrast between the different elements and avoiding complicated visuals.
Process
The setup and the process for the live performance through UE4 has its ups and downs. Our professors, Todd and Kat, even nicknamed us as the Team Oh No! as there were many moments when one variable was working and the next moment it crashed. We kept in mind about the potentials of both software and human error, so we have backups and saved often. We updated each other by the minute with how things were going with the task at hand and what we needed to complete.
Environmental Setup
Louise was in charge of the full integration for the digital and physical space, and making sure details like the map level environment, audio setup (with Aaron), sequencing and equipment had an orderly set up. During the performance, she was on the monitors making sure the system was debugged and virtual cameras were working.
Wall Setup
Nancy was in charge of the walls with trial and error. Todd was readily available on slack and office hours when she needed the help. Factors to consider as a successful interaction are the wall collision, live motion capture effects and ray casting. The walls interaction requirement were collision and the dancer’s ability to interact through ray casting.
Sequencing required wall actions
- Bricks of the wall spawn one at a time, and one row at a time, from the left wall to the middle, then to the right wall.
- Once all 3 walls are formed, bricks start moving in and out organically
Bricks should “ripple“ as dancer pushes the walls. - Rows of bricks will start pushing in towards the center (dancer) from above.
- Walls fall after dancer gained strength and particles from the audience members Particle Dancer Setup.
Dancer Particles
Chester was responsible for the dancer particles with the Niagara VFX system. It’s a tool where users can create and adjust particle systems and effects on objects. We were taught about this system in class, whereas in previous versions particles were created and edited using Cascade. We wanted our particles to be consistent of soft floating minimal particles to both the dancer and the wand particles and used Matt DesLausier’s reflective series as a reference point.
We spent a lot of time tweaking the softness of the particles with the challenges of extreme colors and high opacity. With some tweaks by changing the color and decreasing the opacity, we were able to capture the delicate aesthetic we wanted to portray.
Wand Particles Setup
My primary responsibilities were the rigid bodies (wands), wand particles and the promotional materials for the show.
Rigid Bodies
Rigid bodies are objects with three markers that can create movement or be another camera in the digital scenery. We were interested in creating wands to give the audience members an opportunity to interact in the scene and connect with the dancer. It was an obstacle coming up with a plan to make it as we wanted to create an organic magical like object. There were discussions with using a 3D printer or wood to create, but the problem came around to spend the time and money when there were ready available rigid bodies in the stock room.
After some thinking, I realized from my graphic design background that thick 100lb paper stock will do the trick. It is easier to manipulate and strong enough to withstand the performance. Many mockups later we created three circular wands and a backup. We placed three velcro pieces on the wands for the markers to be placed on.
Wand Particles
The particles were created with Niagara system plugin. There were some issues with the plugin being a bit unpredictable and how I handled it.
Promotional Materials
In addition to creating the live performance, we had to create promotional materials with a teaser video and poster to promote our live performances.
It was a tight deadline within 2 weeks into the project to have a teaser video ready. Despite not having a setup ready, it did helped speed the creative direction.
Sound Design and Choreography Setup
Louise brought along Aaron and Katie to help out for the piece, as we realized elements of sound and performance are the missing key components for this piece. We may not see each other in every meeting, but when we meet up for the occasional rehearsal at the black box, it was a magical experience bringing all the elements together. We provided feedback to one another and discussed how it could be further improved.
Integration
Since this big project, involving two instructors and four team members working on the project, we decided to use SVN for version control.That way everyone had access UE4 project and could work simultaneously. We gave each other heads up if one of us check out one of the levels to avoid conflict files. Thanks to our professor Todd for lending us his system. 🙏🏻🙏🏻🙏🏻
Preparation
Stage Setup
The IDM department was moved into a new building and we had to set up in one of the working NYU IDM labs. Setting up the stage was a monumental task that required the whole class on a Thursday night. All of equipment setup like floor padding, monitors, projection screens, cables and more was quite daunting as the equipment’s value ranges in the thousands of dollars. Everyone had to be careful, while working very quickly, or be known as the person who broke expensive equipment. The macro tasks with setting up the big pieces was done quicker than expected by micro tasks like organizing the cables, safety check, camera setup and sensors, which took up the majority of the time.
The setup was Thursday night and the live performance was on Saturday. We had to juggle with the time limitations for rehearsals and testing any technical issues. In addition, everyone’s main project was still a work in progress.
For our performance setup, thanks to Louise, we had projection of interactive 3D graphics and 4-channel audio.
We did an entire system integration with a spout plugin to connect monitors to the projectors and Optitrack cameras to UE4.
Rehearsals and Performances
For each group performance, we all had two runs through to give everyone a chance to fix any minor mishaps and document the process. We had three spots for rehearsals in specific time slots. During those times, we realized the main project was giving us some hiccups. One of them was when we imported the project through spout into another computer that lead to some potential bugs. It created some inefficiency with our rehearsal time as we had to delay it in order to fix these issues. Despite all of that these rehearsals helped us being adjusted to the system and catch some details before we even realized it.
Minor Mishap
During the performance there was a minor mishap with the wand particles not showing up in the UE4 play through and being reflected on screen. We did notice the trouble through rehearsals as we were attempting to have the particles show up and the rigid bodies create some movement for the cameras to capture. Prior to the performance, we attempted to fix it through our local individual laptops with multiple playbacks and added key press when the worse comes to worst. This was happening while the other groups were rehearsing on the main stage. Despite the two run-throughs and all attempts to fix, sadly it didn’t showed up.
Response
Despite the minor mishap, the audience members response was extremely positive and supportive, which was encouraging to see. It showed despite some parts may not be working, the audience members will not really realize the difference. The performance was thought-provoking and combined our ideas of the physical and digital realities. Both elements being a critical for the piece.
Next Step
Where would you take this project if you had more time/resources?
If we had more opportunities to rehearse and debugging the new setup it will be fully appreciated, as we were limited in time and space to allow other groups to rehearse. The time crunch for the space being set up two days before the final performance was intimidating and daunting, while handling our projects at the same time. The new setup can easily cause some errors by human error, since Optitrack and UE4 are sensitive software. It would had been better if we had performed in a space where the system was already setup for a period of time, instead of building a new one from scratch. More time also means prevention of the wand particles mishap, if we had an extra week setting the stage and do more testing with the system.
Despite the errors that were caused and only having 1.5 months to create the entire piece, I would say it was a success for a group of mocap beginners. Creatives can be one of the harshest critics of their work, but not everything can be spotless and goes without complications. The reception itself is already proof of our accomplishment and the experience behind it made me appreciate process, the people I’d worked with and solutions behind making the piece come to life.
I want to first give thanks to our professors Todd Bryant and Kat Sullivan for the endless help and assistance during the late nights and early mornings. Next Aaron Moreno, and Katie Brady whose contributions were extremely valuable and lending their time to bring this piece to life. Lastly to Team Ono with Nancy Chiu, Louise Lessel and Chester Ma who made sure everyone was putting their best foot forward and motivating each other to make the best motion capture collaboration. Those all nighters and encouragements made it all worth it in the end.
