• Microsoft Hololens 2
• Research Article
• Research In Progress
• Project Website
• Prof. Gillian Smith

Augmented reality (AR) allows the rendering of digital content on top of the physical space, which is a promising medium for tangible interaction. Marker-based AR is widely used thanks to its low costand ease of integration, but the gameful aspect of manipulating the physical AR markers remains understudied. In this paper, we explore the stacking mechanics of transparent AR markers and create an AR music game called Stackable Music. Stackable Music can be developed, assembled, and set up at the home or office with a printer using several sheets of transparent film and a PC or mobile device with a camera.

• Microsoft Hololens 2
• MRTK 2.4.7
• Research In Progress
• Project Website
• Claire Li, Noelle Rakotondravony, Hilson Shrestha, Karen Royer, Laurie Mazza, Shano Liang, Varun Bhat, Prof. Lane Harrison, Prof. Robert Dempski

Worked with Professor Robert Dempski in summer 2020, we developed a HoloLens AR application for training students to assemble components in order. HoloLens AR enables the user to watch the tutorial while working on the real-life components at the same time. We worked as a team of 6, spending 2 months to make this application.

Using mixed reality in chemistry laboratory settings is a perfect fit because you can get access to both virtual content and real-life at the same time. Head-mounted AR serves as an additional information source to the researcher, which doesn’t affect his/her ability in observing real-life events. This is crucial especially for chemistry laboratories because if the researcher’s attention is fully attracted by an electronic device, it might cause safety issues.

• Microsoft Hololens 2
• MRTK 2.8.2
• GitHub
• Side Project In Progress

An AR remake of Pskar Stalbergs Brick Block. The original brick block can be found Here.

• iOS, Android, Microsoft Hololens 2
• MRTK 2.8.2, Vuforia v10
• Research In Progress
• Project Website
• Rachel Foye, Dinah Agyemang, Ava Stockton

The overall objective of this research is to develop an Augmented Reality (AR) assisted learning tool with educators and students involved in the design. Balancing between informative and engaging, the AR tool is aiming to become both engaging and helpful in terms of enhancing high cognitive level learning outcomes. Applying AR in education in an interdisciplinary research area including pedagogy, learning science, interactive media, and psychology. This study follows a participatory research approach and includes educational technology engineers, educators, and students to co-create an AR tool iteratively – engineers come up with a prototype, educators give critical feedback, and students evaluate the tool from both engagement and learning outcomes aspects. The study will bring insights into how to utilize participatory design in AR educational tools and how AR has an impact on students learning outcomes, as well as effective ways of prototyping AR experiences.

• Microsoft Hololens 2
• MRTK 2.8.2, Vuforia v10
• Major Qualifying Project Completed
• Shruti Mahajan, Amanda Jones, Megan Letendre, Elise Nerden, Prof. Gillian Smith, Prof. Erin Solovey

Personal memories are associated with much more than just a picture or a video, for example, clothes, T-shirts, or other fabrics. Such tangible/tactile objects are important for memory and nostalgia. Quilts are often made with a loved one's old shirts or for special occasions. These quilts have special cultural importance and interaction that triggers emotional responses by touch. We envision a future of AR where people will be able to augment personal items with visuals and audio. Family or friends co-creating a memory quilt that is a collection of special clothing items pieced together to celebrate important milestones, special events, or the life of a loved one augmented with AR visualizations and audio.

• Vive Pro 3 (Android)
• Contract
• Laurie Mazza, Michael Schubert, Mengliu Lu, Prof. Jennifer deWinter, Monty Sharma, Delsys Team

This project is the development of VR games targeted toward children with muscular dystrophy in the 8 to 12-year-old age range. They have a range of movement capabilities but all gameplay is to be seated and have periods of rest built-in. The goal is to encourage the children to participate in clinical recovery training through gameplay interactions instead of direct instructions.

We worked as a team of 4 and spent 4 months on the development. This project is a collaboration among Delsys, Worcester Polytechnic Institute, and Clark University.

• PC, Oculus Quest 2
• GitHub
• Research Article
• Claire Li, Yihong Xu, Alex Sirois, Risa Kiani, Prof. Yunus Telliel, Prof. Yuko Oda, Prof. Erika Lewis, Prof. Kelilah Wolkowicz

WheelUp is a collaborative research project between Worcester Polytechnic Institute (WPI) and University of Massachusetts Lowell. The overall objective of the study is to co-design a high-fidelity VR wheelchair simulator for diverse body. It uses participatory design to engage local communities with alternate-abilities for the development of a wheelchair training simulator with multimodal input interfaces for wheelchair guidance and control. Simulated and virtual reality (VR) training allows users to practice difficult or stressful tasks in a safe environment. The virtual environment will integrate joystick, muscle signals (EMG), and eye gaze (EOG). Aesthetic attributes of the simulator such as color, texture, and layout will be considered to enhance the overall visual experience, making the user experience more enjoyable and appealing. The simulated environment includes standardized hospital or home; variable environments with changing obstacles such as shopping centers or workplaces. (PI: Prof. kelilah wolkowicz)

• Browser
• In Progress
• Project Website
• Shano Liang

We create 360 lab tours for virtual walkthroughs and training sessions.

• Experimental, NIRx's fNIRS System
• Thesis
• Prof. Erin Solovey, Prof. Gillian Smith

we explore the use of brain-computer interface (BCI)-adapted visual effects to support atmosphere in a walking simulator, and investigate its impact on player-reported immersive experience. While players were using a keyboard or joystick controller to control the basic movement of a character, their mental state was accessed by a non-invasive BCI technique called functional near-infrared spectroscopy (fNIRS) to implicitly adjust the visual effects. Specifically, when less brain activity is detected, the players' in-game vision becomes blurry and distorted, recreating the impression of losing focus. fNIRS measures blood oxygenation levels, are related to brain activation. When a particular area of the brain is activated, it receives an increase in blood flow. With this biological indication, we designed a BCI-controlled game, in which the vision becomes blurry and distorted when less brain activity is detected, recreating the impression of losing focus.To analyze the player’s experience, we conducted a within-subjects study where participants played both a BCI-controlled and non-BCI-controlled game and completed a questionnaire after each session. We then conducted a semi-structured interview to investigate player perceptions of the impact the BCI had on their experiences. The results showed that players had slightly improved immersion in the BCI-adaptive game, with the temporal dissociation score significantly different. Players also reported the BCI-adaptive visual effects are realistic and natural, and they enjoyed using BCI as a supplemental control.

• TidalCycles, Processing, Muse S
• Course Project
• Project Website
• Samin Shahriar Tokey, Prof. Charlie Roberts, Prof. Gillian Smith

We did a live coding performance using visuals and audio integrated Brain-Computer Interface input. We asked for a volunteer (audience or co-performer) to wear the BCI headset while we perform. With the BCI signal from the audience, we will generate audience-adapted visuals for our performance.

We followed the approach of performance-led research in-the-wild, which falls under the broad umbrella of Research through Design. This means research findings are emerging from reflection on the making of specific interactive artifacts. Being performance-led means the practice is led by artists, who follow an artistic process and deliver a professional artistic product, in our case, a live coding performance. Being in-the-wild means the artistic product is experienced by public audiences, in our case, being under critical judgment by the live coding cohort. The approach involves documenting both the artist’s reflection on producing the work and the audiences’ experience of it.

• Escape Room
• Project Website
• IMGD 5100 Class, Prof. Dean O'Donnell

The Escape Room

Holiday Escape is one of our best WPI Annual Escape Room series. Our escape room is a 30-minute experience for groups of people where the goal is to leave or escape the locked room by solving a series of puzzles.

This project is a class project of IMGD 5100 Interactive Media. We worked as a team of 16 and spent 2 months to design, build, and run the escape room.

The Story

You are boarding the train for your holiday tour of Santa’s toy shop, but something is afoot… the conductor can’t seem to get the doors to open. Upon successfully boarding the train, are greeted by a message from The Elves, who have rebelled and hijacked the train after tired of years of unsatisfactory working conditions making toys for the shop. If the train reaches its destination, you will be turned into elves and forced to take their place in the workshops for all eternity!

If players fail, they are “turned into elves” and wear elf hats for final photo. If players succeed, the elves unionize and improve their working conditions without capturing the players

My Role as the Lead Designer ("Puzzlemaster")

I led puzzle design and iteration, built and tested puzzle prototypes collected and analyzed playtest feedback.

Lessons Learned

The narrative in escape room needs to be explicit and straightforward. However, not all players care about the story. Embedding puzzles to storyline is the creative process that matters the overall experience.

• Theme Park
• Shanghai, China 2020
• Jiadi Zuo, Ekings Multimedia

The Show

IKALI Dreamland is the first comprehensive children's amusement park in China under the theme of scenario-based costume play. It integrates the novel concept of role play and edutainment for children, which is put into practice in the form of a nationwide brand park chain.

In December 2020, Ekings Multimedia and IKALI jointly operated Dreamland Shanghai. Daily admission flow reaches 300 at its peak.

My Role

During Jun 2020 to December 2020, I worked as a programmer at Ekings Multimedia, where I worked with clients on designing and developing interactive media applications and games with various needs.

I designed and developed a mini game Air Balloon Pilots, in which player controls their air balloon by balancing its mass.

I assisted in setting up and testing the infrared camera multi-touch screen system. It uses IR camera to track the user's hand gesture and projector to cast the interaction feedbacks, therefore it can turn any flat surface into a touch screen.

Lessons Learned

Immersive experience pulls a viewer into another real or imagined world, enabling them to manipulate and interact with their environment. It has a long history, from Panoroma landscape painting to Cinerama widescreen projection process, to Cave Automatic Virtual Environment where projectors are directed to walls of a room-sized cube. While the comtemporary screen media erode, if not erase, viewer's perceptual differentiations between the actual reality and the virual reality, the gap between real and virtual is where designers perform their the magic touch.

The market of projection mapping, IR camera touch screens, and sensor-driven interactions is open - the entry cost is relatively low comparing to VR/AR hardware research. The future lies in designing a meaningful and aesthetic experience. Level 99 Entertainment combines escape rooms, small "challenges", and arena games with microcontrollers and sensors, making the traditional plays into self-reset "smart" games. TeamLab brings artists, programmers, CG animators, mathematicians and architects together to navigate the confluence of art, science, technology, and the natural world. The list of examples can go on.

• Craft + Experimental Storytelling
• Presented at PAX East 2022
• Shano Liang, Tony Dai

The Show

Collection is an experiment combining non-linear storytelling and traditional origami. It appears as an opened package, with a letter and a treasure box inside. The treasure box contains old pieces of newspaper clips, lab notes, drawings, and segments of a diary, forming a bizarre tragedy about the Werners.

It is up to you to decide how to explore the Werners’ story. There are 16 pockets on the top, but more are hidden underneath them. It is a tangible physical book that you can even tear apart. Search through the pieces in the Collection to discover the truth.

Lessons Learned

This treasure box format is similar to the take-home escape room adventures. Instead of having multiple physical objects, you have only one complex object. However, it also limits the format of story segments to be paper-based.

Origami and tangible interface provide the enjoyment and engagement while exploring the puzzle.

• PC
• WPI News

Fighting Against Coronavirus game is a single-player strategy game. The player takes the role of a government official in “City W” starting from January 1, 2020. The player’s mission is to guide local disease prevention and control work, as well as take care of the people at the epicenter of the ongoing outbreak with minimal loss in human life and economic growth. We worked as a team of 3, spending 4 months to make this game.

I designed the in-game epidemic model by modifying the SEIR model (Susceptible, Exposed, Infectious, and Recovered), which makes me think of the close connection between mathematical modeling, system dynamics and game. Later I learned Machinations framework from the book Game Mechanics Advanced Game Design, and it is similar to the stock and flow diagram, which confirms the close connection between in-game economic system and system dynamic.

• Browser
• Ready to Play
• Hibernate

Cirbuit@Dungeon is a web application for learning basic electronics. This project is supported by WPI Academic Research Computing Center.