Informatics Seminar Series
Fall Quarter 2019

Abstract:
In this talk I'm going to discuss two research projects that are currently underway in my group: ShadowCast and VirDAW. ShadowCast is a platform for virtual reality musical theater karaoke. The name ShadowCast invokes other experiences of playing along with a popular media property, the most well-known of which is screenings of The Rocky Horror Picture Show where a shadow cast performs along with the cult film and an audience joins in via a shared set of inside jokes, interactive elements, and audience participation cues. We take inspiration from these experiences for our system, which we have built to support a mixed reality audience participation element.

ShadowCast combines live performance with augmented and virtual reality to create an immersive experience of stepping onto a virtual stage. Virtual performance, as mediated through an avatar, comes with a host of complex interactional challenges in order to give the performer the ability to easily express and communicate emotion, without distracting them from the moment of the performance. ShadowCast explores the pleasures of enacting and performing within the confines of a pre-existing script, in this case a karaoke song. It allows performers immense creative freedom to express themselves through their performance by providing them with an enactive scaffolding. One of our artistic and conceptual goals with this work is to move the locus of interaction away from deciding what to do and towards being in the moment. ShadowCast also lays the groundwork for a more ambitious project, because the software infrastructures that we have created for it are the first steps towards building a fully featured platform for virtual reality distributed theater performance. We are now in the preliminary stages of expanding our design to accommodate VR performance education, which will allow us to develop curriculums for performing arts education in virtual reality. A digital infrastructure for theater performance and spectating has the potential to expand access to the arts to rural and socioeconomically disadvantaged communities.

The second project I'll discuss, VirDAW,is a collaboration with Vinnie Olivieri in the Drama Department at UCI who specializes in sound design. Software and hardware for sound design and engineering currently operates with a set of skeuomorphic interface conventions and practices that have remained unchanged for over 60 years. Knobs, faders, buttons, and other small-scale widgets abound on contemporary sound control boards and digital audio workstations (DAWs). Consequentially, the creative practices of sound designers have evolved to favor the techniques and workflows that these platforms afford. In any artistic practice, the artist enters into a material dialogue with the medium of creation: an artist working with crayons operates under different constraints than an artist producing visuals in a piece of 3D rendering software. Thus, by altering the artistic platform we might reveal new aesthetic possibilities for an artform. VirDAW is an exploratory design project that reimagines the functions of traditional sound design software environments within virtual reality. Still in the early prototyping stage, VirDAW remediates widget-based interactions into an embodied spatial context, transforming how sound is visualized and manipulated. Our hope is that this new interactional context will expand the design space of tools and interfaces for creative storytelling through sound. VirDAW supports immersive and embodied versions of common audio engineering practices. These include loop sequencing, multi-track mixing, equalization, compression, and reverb. VirDAW also creates opportunities to easily solve sound spatialization challenges and surround mixing tasks, which are features that are poorly supported within current state-of-the-art DAWs. Finally, VirDAW supports playful new ways of interacting with sound through an interface we describe as “kinetic composition” that combines simulated physics with reactive virtual architectures that allow composers to put audio files into motion and then experience cascades of unexpected new effects over time. Taken together, these modules provide insight into a new paradigm of sound creation and manipulation.

These two projects are still in preliminary phases, but they help tell the story of the work happening in the Transformative Play Lab that cuts across disciplinary and methodological boundaries.to explore new technologies for artistic creation, expression, and consumption.

Bio:
Dr. Theresa Jean Tanenbaum (“Tess”) is a game designer, artist, maker, and assistant professor in the Department of Informatics at the Donald Bren School of Information and Computer Sciences at the University of California-Irvine where she is a founding member of the Transformative Play Lab. She received her PhD from the School of Interactive Arts + Technology at Simon Fraser University. She is the club advisor for the UCI Video Game Development Club, and the VR @ UCI club. She also organizes the UCI Global Game Jam Site which has grown to be the 8th largest site in the world over the last five years.

Dr. Tanenbaum’s work is playful, provocative, and interdisciplinary, frequently straddling the line between art, design, and research. Her doctoral research examined identity transformation and empathy in digital narratives and games, drawing on theories and methodologies from the performing arts and human-computer-interaction. Her ongoing work on “Transformative Play” draws on techniques from theater practice to create and explore playful experiences that communicate different perspectives on the world, encouraging players to viscerally inhabit new identities and experiences. Her first book, edited in collaboration with Magy Seif El-Nasr and Michael Nixon, entitled Nonverbal Communication in Virtual Worlds: Understanding and Designing Expressive Characters was released in early 2014 by Carnegie Mellon Universities ETC Press. She also served as a consulting researcher at the Nokia Chief Technology Office’s Advanced Engineering group where she advised on matters of storytelling and wearable technology for the Internet of Things.

An experienced game designer, Tess’s work incorporates physical objects, wearable technology, and interactive tabletops to explore embodied interactions with digital games and stories. In 2010 she was the lead designer for Futura: the sustainable futures game, which was showcased at the 2010 Winter Olympics in Vancouver, and later incorporated in the City of Vancouver’s “Greenest Cities Conversations” program. Collaborating with Karen Tanenbaum, she created The Reading Glove: a tangible, wearable, work of electronic literature where a reader explores a collection of evocative physical objects to piece together a historical narrative. She has developed new gaming technologies that push the boundaries of personal fabrication, using 3D printers and laser cutters as platforms for hybrid digital/physical games. Her recent game, Magia Transformo: The Dance of Transformation, was an official selection of IndieCade: the largest festival of independent games in the world. It uses costumes and movement to help players adopt the personas of witches and warlocks to uncover the secret magical history of the world. Tess is also a “Steampunk” artist, and maker, whose work on DiY culture appears in the book Vintage Tomorrows and the documentary film of the same name.

Her current work explores the intersection of live performance and MR/AR/VR technologies. Collaborating with Tim Kashani of Apples and Oranges Arts, she has mentored students in experimental designs for VR theater experiences, including a student team that took second place in the 2018 Butterworth Product development competition. She is currently developing Virtual Reality systems that transform how we think about creative practice in the performing arts. These include ShadowCast, a VR networked theatrical performance platform (created in collaboration with Tim Kashani), and VirDAW, a VR digital audio workstation (created in collaboration with UCI Drama Professor Vincent Olivieri).