Overcome public speaking anxiety with training in VR

Have you ever felt discomfort when it is your turn to speak in a group discussion or when you have to give a speech in school elocution competition? Before giving a speech in front of a large audience, few people practice in-front of a mirror or find a friend to get some feedback in a practice session. Training in virtual reality is likely to be the solution to public speaking anxiety. Studies were done on virtual reality exposure therapy and companies are using VR applications for employee training on public speaking.

Cross reality (XR) which includes virtual reality (VR), augmented reality (AR) and mixed reality (MR) provide immersive digital experiences. VR actively uses human sensory capabilities (like sight and sound) to provide understanding and general relations of an experience. This can be used to improve public speaking using immersive and realistic simulations. A study published in Cognitive Behaviour Therapy suggests that VR can be used as a therapeutic tool for public speaking anxiety. [1]

There are multiple VR applications that help with overcoming the fear of public speaking. VIRTUALSPEECH [2] is a company that specialises in professional development training with courses for mastering public speaking. The training contains a wide range of self-paced VR scenarios.

TEDx theatre as VirtualSpeech training scenario

VirtualSpeech application is available on VR headsets like Oculus Quest, VIVE Focus 3 and Pico Neo3. Besides the variety of VR scenarios, the features from VirtualSpeech that stand aside are Real-time feedback, display notes in the room on an autocue, and audio & visual distractions from the avatars to simulate real world experience. The Real-time feedback is given while the speech is being delivered; feedback is provided on eye contact, pace of the speech and volume of delivery.

One feature allows speaker to record and upload questions in advance. These questions are then asked by the virtual audience during the speech. Conversational AI can be used to further enhance this feature to allow real time communication between the speaker and the audience.

Real-time feedback in VirtualSpeech

Ovation [3] is another application in VR that helps overcome public speaking anxiety. It also provides real-time training tools and feedback as one speaks to a realistic, simulated audience. Training is provided for Gaze, Voice and Hands. The category, Gaze refers to where the speaker is looking while delivering a speech. This is detected by the movement of the VR headset. If the headset includes eye tracking, it can determine the exact location in the virtual scenario where the speaker is looking. VR motion controllers can be used for training on mic distance. If the mic is too far from speaker’s mouth, a red pulse is displayed and speaker will experience vibration. The motion controllers can also be used to determine the movement of speaker’s hands.

Ovation Public Speaking training in VR

The recommended VR headsets from Ovation are HP Reverb G2 Omnicept and HTC Vive Pro Eye. Sensors in Omnicept can detect cognitive load in real time, it captures the brain power needed to remember and properly deliver the speech. Both Vive Pro Eye and Omnicept detect the exact location where the speaker’s eyes are looking, this is used by Ovation to provide more accurate analytics and better insights.

The study published in Cognitive Behaviour Therapy [1] concluded that one-session virtual reality therapy can be an effective treatment of public speaking anxiety. In future, we can expect to see more VR applications in public speaking training with enhanced features like automatically generated questions and emotional responses from virtual audience based on sentiment of speaker’s speech.

References:
[1] Philip Lindner, Jesper Dagöö, William Hamilton, Alexander Miloff, Gerhard Andersson, Andreas Schill, and Per Carlbring (Sep 2020). Virtual Reality exposure therapy for public speaking anxiety in routine care: a single-subject effectiveness trial. Cognitive Behaviour Therapy.
[2] VirtualSpeech.com
[3] OvationVR.com

VR and Fantasy: A Match Made in Heaven

Skyrim VR

When I laid hands on the first fantasy game that I had ever played, The Elder Scrolls V: Skyrim, I was immediately hooked. The open-world game was rich with lore and diverse regions to explore. Coupled with great soundtracks and well-designed ambience, Skyrim was able to draw players in with its deep immersion – only limited by its technological interfaces at that time. After all, despite its best efforts, players could only interact with the game through a mouse and keyboard, and see the expansive world only through a 2D screen. However, all this changes with the introduction of virtual reality.

For the first time, players were able to truly walk around and explore this world with new eyes. They could stand in the middle of a wide open plain and “look around” at the scenic views the game had become well known for.

Combat in Skyrim VR

Combat was enhanced in particular through the use of VR. Gone were the days of button mashing and remembering the keyboard shortcuts for combat. Now, players get to hold weapons and shields in their hands, or actually wield magic should they choose to. They now come face to face with their enemies and feel as though they are truly in battle – presented with a level of immersion impossible without VR.

However, intense immersion may come with a few drawbacks in such an immersive combat fantasy game. Experiences that were once far removed, may now actually feel “lived” by the player.

Boo!

Draugr depicted in the image above, are zombie like creatures in the game. Imagine the scare players may experience when they feel like they are actually placed in the world. They go about minding their business and find themselves stumbling right into such a creature. Such experiences may feel much more vivid in VR, and game designers will have to take this into account.

Blade and Sorcery VR

Another VR game that I really like is Blade and Sorcery. Unlike the story and immersion centred Skyrim, Blade and Sorcery is a multi-player combat game that allows players to duke it out in an arena. While realistic combat is difficult to come by in games, Blade and Sorcery is at the top of its class in providing it. Combat and magic are realistic, and allows players to relive their childhood fantasies with its wide array of mods. Players can battle with lightsabers, or in medieval gear and with magic. The game also allows players to perform “trickshots” in combat, and have become an increasingly popular sub-goal in playing the game.

Despite its ability to bring realistic combat to the player, its visuals do seem clunky, and movement may not always be very natural. Players have complained of weird perspectives in game regarding their in-game limbs, which do break immersion for some players. Overall, it is an enjoyable game to play and sets the foundation for VR games of the future.

SkyView – Locates Stars, Planets, and Nebulae

About the APP

SkyView is a lightweight stargazing app available on iOS and Android devices. The augmented reality technology allows users explore different constellations, planets, star clusters, stars and other celestial bodies in the night sky through their smartphone’s camera. Simply pointing the smartphone to the sky, then SkyView will identify the stars and other objects in the view.

SkyView® Lite - Apps on Google Play

Why do I like it & Why it is engaging?

Have you ever imagined a date with your boyfriend or girlfriend, you guys take a walk along the beach, and then she or he points to the sky and tells you what constellation that is? Romantic, isn’t it?

I never imagined that until I met a boy who loves boys, we walked on the west coast park and identified all the constellations we could see. Still romantic. Hence, I find I can have such romantic experience without a boyfriend or girlfriend. I can do this by myself, with SkyView.

I downloaded the App months ago, and I found it is easy to set up and use. It brings a fascinating view of night sky to users with the stylish UI and the well designed graphics. The main function and interaction of the application are quite simple, so that many parents also use this as the stargazing education for their kids.

Orion constellation shown on SkyView

What features are well done & What can be improved?

The main function of the app, the AR stargazing camera, has simply interactions. A short description of the star or other objects will be shown on the side of the screen when you click on the object. This helps users connect the star on the screen and the star they see in the sky quickly.

Users are also allowed to increase and decrease the magnitude visibility of planets and stars, this make it easier to see the stars and the brightness of them for users who are in a region of high light pollution.

Night mode

However, the night mode is not clear as the usual mode. The filter is not very user friendly and covers some views in the camera. The same planet and star size settings lead to the less brightness of stars. The color of the filter can be changed to green or other light colors instead of red. The size setting of objects could be larger so that it will be clearer in the dark.

Overall, SkyView is a fascinating and romantic AR stargazing application which is easy to start and use.

Google Earth VR

There are so many amazing places that one would like to visit during one’s lifetime. With a bit of luck and resources, one may be able to travel to a few of them. However, to travel to all of them would be impossible… or would it?

With Google Earth VR this could be possible.

The industry of Virtual Reality (VR) is growing at a fast pace. Its many different applications are not only recreational but may also be educational and be more than just a distraction. The Google Earth VR application allows the user to strap on a VR headset and start exploring all the wonders of the world from the comfort of one’s room.

Why they are engaging?

During the rough times of COVID-19, the possibilities of travelling have been severely limited. With the use of VR technology, the world is opened up again without the risk of transmitting the virus. Google Earth VR uses the 3D model of Google Earth, which has mapped 98% of the World. However, it is not just Google Earth ported to VR, it is a completely new experience. All the interactions have been redesigned and the VR application allows flight as the mode of travel, both at different locations but also in-between them.

Flying over New York in Google Earth VR.

This allows a very engaging experience for the user. The application is fast-paced and allows for many different amazing sights in just a couple of minutes. But it is not only that the user gets to tick off locations from their bucket lists at a pace not possible without this technology. While travelling in VR, the user also gets to experience the location in a brand new way, from many different perspectives which would not be possible if physically present.

With the application, one can fly over any city and experience them in impressive VR, stand at the highest mountain peaks of the world and appreciate the view in all directions, walk along streets in remote destinations you never thought you would get to visit, and much more.

What features are well done?

The Google Earth VR application is impressive in many ways. The sheer scale of the project is unimaginable, as Google Earth covers almost all parts of the world on a real-life scale. The controls and the straightforwardness of the application allow the user to effortlessly explore the world in an immersive way as if one was there in person.

What features can be improved and how?

However, the features of Google Earth VR are not all perfect. Due to the scale of the project, Google has been required to do a lot of optimization for the experience to be enjoyable. The 3D models are procured from combining enormous amounts of 2D images and layering everything onto 3D space. As a consequence of this, the detailing is not very accurate and sometimes certain graphics appear as blocky smudges of colours.

The Eiffel Tower in Paris with lacking details.

Improving this while still maintaining the scale of the project is hard. For the user, going to ground level (which activates Google Street View’s POV) results in much more details than the approximations in the bird’s view. As for what Google could do, this seems to be a project that does not get that much care anymore. However, with the use of Artificial Intelligence, Google could aim to do some photogrammetry repairs of textures, for example by combining the detailed Street Views with the textures in the Google Earth view.

To summarize, Google Earth VR allows the user to travel to incredible places across the world. At its current state, it may not be detailed enough in order to replace real traveling. It does however offer an immersive sandbox experience with new sights to explore anytime.

SketchAR: A new way to learn drawing

SketchAR is a mobile app developed in 2017 which revolutionised the way we learn how to draw. It projects AR stencils on the user’s drawing paper, and all the user has to do is to follow the stencils and draw accordingly. I like this app because it is very beginner-friendly. Users with no background in drawing can pick up a pencil and paper and draw right away. It removes the technicality of drawing and allows beginners to quickly immerse themselves into drawing. While this app is targeted to novices, experienced artists can also use the app and try a new method of drawing with AR technology.

Step-by-step instructions with AR stencils

The main feature of this app are the drawing lessons, which are broken down into smaller steps – each consisting of an AR stencil. Upon completing the stencil for the current step, the user presses ‘Next’ and the stencil changes into the one in the next step. There are many lessons offered on the app, such as animals, food, vehicles, anime, and architecture, to suit the different interests of its users. Users are also given the option to disable the AR stencils and follow the steps from their phone instead. Here is a video review of the app by an artist:

Video review by Jazza, an artist on YouTube

However, a major drawback of using the app is that users are recommended to attach their phones to a tripod while drawing. Otherwise, it will be tiring for the user to hold up his phone with one hand and draw with the other. Perhaps the use of goggles could be used to solve this problem. The user will wear goggles to view the stencils and free both of his hands, which could help him to concentrate on his drawing better.

Users are recommended to use a tripod when drawing

Furthermore, the AR stencils restricts the area on the paper which the user’s hand can occupy. If the user places too much of his hand on the paper, the AR stencils will become distorted. A possible solution is for the app to continue projecting parts of the stencil that is covered by the user’s hands. In that way, the user can continue to follow the stencil without having to worry about touching the paper.

Before distortion
After distortion

Another concern expressed by artists with regards to using AR stencils to learn drawing is that it hinders one’s creativity. Budding artists may become used to strictly following instructions such that they do not engage their own imagination. They will also lack drawing fundamentals which is crucial in developing and refining their drawing skills.

However, for the average user who wishes to draw for their own entertainment, this app is great for helping them to dive right in. When users sign up for an account, they can choose categories which they are interested in, and their drawing lessons will be personalised according to their interests.

Besides the AR stencils, this app boasts other cool secondary features as well, such as sketching from stencils designed by other users, sketching photographs from the phone’s gallery, and even a colour-matching game that trains users to recognise colours of different shades.

Colour-matching game

All in all, I would say that SketchAR is a great app which allows anyone to start drawing almost immediately, without any prior experience in drawing. Although it does have some setbacks, I believe that they can be rectified quickly since the technology is already there (goggles etc). SketchAR utilises AR technology in a fun and innovative manner that can transform anyone into an artist!

VR Game “Bogo”

One of my favourite VR applications is called “Bogo”, because it seems like a very active game where users can interact with a virtual pet, which is a very expressive character with many different emotions. It is very engaging because users are free to pet or touch the virtual character anytime, and the character will respond accordingly with clear expressions on its face.

There are features which were well done. Besides the character itself, there are many other things occuring in the background, which includes preparing food to feed the character and throwing sticks or balls to play games with the pet character.

  • Despite the fact that the little alien pet character was evidently artificial, as it could feed on the food even if the user chooses not to give it directly to the pet but throws it on the ground instead, it seemed to have a mind of its own, which builds onto the level of subjective reality in the game.
  • The character has many ways of communicating with the user. For example, feeding the character also causes the character to change its own colour and sometimes, new architectures will evolve in the environment, which is fascinating and evokes positive emotions in the user, and elevates a sense of immersion and presence.
  • A good point is that the pet character is always close to the user and responds immediately to the user’s actions accordingly, which is a key and shining feature of the game as it has the potential to make users feel appreciated.
  • This feature where the pet character changes its colour also allows the user to see traits of the character that are unique to it, and helps to build a sense of personal bond between the user and the character.
  • Another element which I found was good, was that the character comes to the user, instead of the user having to walk over to the character, which can be an important feature in the game as it has the potential to be built on with some other traits of the character, to provide emotional support to the user and make the user feel much closer to the game character.

“Bogo” is an application that uses Oculus Quest Headset, along with Oculus Rift. It is developed by Oculus.

There were also some features which could be further improved to make the experience to the user much more pleasurable by further elevating the sense of presence of the user.

  • I thought it could be improved by elevating the aspect of social communication. Since it is an application involving virtual pets, it could be more fun to include more opportunities for users to communicate with other humans in the VR game who could be taking their virtual pets on a camp or a walk in the environment.
  • Alternatively, there could instead be a storyline on the virtual pet embarking on a quest to fulfil some missions, to help users immerse in the nature of the virtual pet in its natural environment and be able to see more facets of the traits of the virtual character, in order to build a much more immersive experience.

Personal Omnidirectional Treadmill Simulator

The concept of Omnidirectional Treadmill Simulator (for home use) first come into my sight in 2013 CES, when Virtuix showcase their prototype which was originated from a Kickstarter campaign.

The Omni-directional treadmill replaces the traditional use of joystick and allows players to walk and run in 360 degrees inside videos games using body movement, providing a higher level of immersion while gaming.

The technology was immature when it was first released 9 years ago as the devices only allow simply walking and running. The technology has matured over the year, the latest product allows the user to perform more complex movements such as crouch, kneel and jump. The following video shows the latest version of the treadmill.

Other companies have followed the trend. The following video shows a similar product by another company that is currently available for purchase.

Gaming is often associated with unhealthy behaviour as gamers spend most of their time sitting down in front of the screen. I like how the device will change this idea to “Some Gaming device is healthy as the player will have to move their body while playing the game”. providing a platform for the gamer to play healthy while moving.

When playing a VR game, the players will often minimize their lower body movement. Otherwise, they might end up being one of the people in the video.

The treadmill improves the safety of the player by fixing the player in one place. It also reduces the space required for the VR setup, as a larger space is usually required to avoid collision with furniture while playing VR games. It also allows the player to make intensive movements such as running, which is usually impossible for a home VR setup due to limited space. The devices also improve the immersion of the VR experience as the player will not need to consistently worry about their location, avoiding the fate of ending up as one of the people in the video.

The Omni-direction treadmill is a heavy source of noise pollution. The noise produced by the device may be unbearable to the player family/housemate, especially when the player is making intensive movements such as running. The device may not be suitable for player who often plays the game at midnight and might disturb their family/housemate when using the devices. One of the possible improvements is to make use of noise damping material to reduce the sound produced when interacting with the devices. It is also possible to provide a soundproof dome (which the player can buy as an add-on) that cover the entire devices to minimize noise pollution when using the devices.

The notion of “walking” is replicated by using specialized shoe or shoe covers and a surface that reduces friction. It gives a feeling of walking on ice, which isn’t that realistic and immersive. The devices may be improved by using an actual Omnidirection treadmill.

The current technology is current immature (and bulky). I will be looking forward to a future(decades?) where a true Omni-direction treadmill is available to home consumers.

Minecraft VR Experience

With the recent trend of virtual reality, many games have tried to create a VR version of their world, and Minecraft is one of the more popular game that has succeeded in converting its world into virtual reality. Minecraft is a sandbox video game where the environment and its inhabitants are made up of blocks.

Being a classic game many people have played before, introducing the VR version brings a new sense of experience in exploring the virtual world. The activities from the original version could be done inside the virtual world, with a more feeling of closeness. One can feels more close to the world as compared to seeing it through the screen as the player movement is followed closely which bring a higher sense of control inside the world. Mining and killing enemies is also more exciting as hand movement is required to execute the order which make it more interactive as compared to clicking a button. Hence the VR version of Minecraft helps to create a more interactive experience for the players to enjoy inside the world.

However, some people might not be familiar with the experience that comes along with VR. Some might feels that they are too close to the world and might feels excessive emotion such as being scared when a explosive creeper appear out of nowhere inside the virtual world as compared to just appearing on the screen. Some might get VR motion sickness as their movement inside the world might be too excessive like when falling of a cliff. Therefore, not everyone would enjoy Minecraft VR as different people have different expectation of the virtual world and some experience are hard to be compromised for everybody. Nevertheless, some of the experience could be toned down as not everyone is well versed with virtual reality and some of them might be newcomers to virtual reality.

In conclusion, I feels that it is good that many games such as Minecraft are trying out VR version of their worlds to bring out a new gaming experience. However, not all games can be converted into virtual reality and some games should remain as it is as the VR version might be a disappointing experience in the end. Overall, Minecraft itself is a great game and the VR version attract people that want to experience more of its world.

Analysis of SUPERHOT VR (on Occulus Rift)

“Time moves only when you move” 

The caption above is a pretty succinct explanation of the core mechanics of the game. The gameplay is simple: time moves only when you move, use this ability to take down numerous bad guys with a variety of weapons lying all around the map. Originally it started as an entry from 2013 7 day FPS challenge, it further expanded to a full game with funding from Kickstarter. This is one of the most engaging FPS titles that I’ve played and it adds a new dimension to FPS genre in which the genre has gotten pretty stale over the years (my honest opinion, don’t stone me if you enjoy FPS games).

“Dodge this.” If you know where this iconic one-liner came from, you’re awesome.

Despite dying repeatedly in this game, I could not help but to keep retrying over and over again. Because time only moves when you move, everything around you slows down, allowing you to evade bullets and take your time to aim with utmost precision. It feels incredibly badass to be able to dodge incoming bullets similar to the famous bullet dodging scene in the Matrix. It is thrilling to see bullets whizz past you knowing that if one of these hits you, it’s game over.

Being able to dodge bullets in slow motion is a powerful selling point, but it’s not the only element that keeps me engaged in the application. The reason why I keep going back to the game is due to the high level of freedom, creativity and immersion offered. In VR, you can individually control each hand. This allows for more immersive gameplay as you feel directly involved in the middle of the gunfight itself. Instead of playing through WASD + mouse on PC, your hands are frantically searching for anything useful to fight back.

you are not controlling the character, you ARE the character!

To make things more interesting, each hand can carry a different weapon. Other than firing guns, you can actually throw your weapons to kill, hit enemies in close range or even block bullets like in a Japanese anime fight. This allows for many creative approaches to the game. You can dual wield pistols like Lara Croft, throw bottles like in a bar fight or shank opponents up-close. Unlike most VR games that has clunky controls and being largely restrictive, freedom in Superhot VR is largely bounded by your wits, luck and the amount of space in your room.

The first feature that immediately caught my eye is the minimalist, abstract art style. I really liked the simple color palette of the game: white means environmental backdrop, red means bad guys, navy blue means weapons, only 3 crucial colours reminiscent of the popular “3 Colour Rule” in fashion. It is stunning to see the bad guys shatter like glass when they get killed. Other than simply being a unique treat for the eyes, this minimalist graphics serves an important function: allowing the players to see important elements clearly. In a game where you have to dodge bullets to survive, you have to be able to see the bullet trails, where the weapons are, where the bad guys are located. With all these things that the player have to look out for, I can’t see this game having a realistic-looking graphics replicating the Matrix as it would be too visually noisy to be immersed in evading bullets. Overall, this abstract art style is an excellent choice, both aesthetically-stylish and utilitarian.

The next nice feature that I would like to highlight is the game balance & direction. In this game, ammunition is scare. You run dry after firing 2-5 rounds. Made worse by the fact that you can’t reload, forcing you to scavenge weapons from fallen enemies. The lack of a hit-point bar makes it very clear that once you’re hit, you are dead. The combination of the inability to reload, death on hit and the slow motion gameplay makes the direction of the game very clear: Be careful, do not waste bullets and kill with well-timed precision and finesse. Not only does this make the game engaging in terms of difficulty, it reinforces the image that you don’t need overwhelming firepower to win. All you need are just your wits and using your foes’ weapons against them. This makes Superhot VR stand in the interesting middle ground between being an action and a strategy game.

However despite the game having nailed well in numerous aspects, there are still some shortcomings to be addressed.

The first shortcoming is insufficient levels and the lack of difficulty chokepoints (aka mini-bosses). The campaign (story mode) eventually becomes boring, and I ended up only playing endless mode, where I try my best to survive for as long as possible. The lack of bottlenecks and levels might make some users lose interest in the game experience gradually. Taking this game to a multiplayer realm is extremely daunting. Hence, my suggestion to improve replay-ability and engagement would be to make the game more rogue-like. This heavily ramps up the game’s unpredictability, making the experience more interesting for veterans.

The next is the ‘lobby’ of the game which I think is not very intuitive and have perspective scaling issues. To start the game, you have to insert a floppy disk (which some players might not even know what a floppy disk is!) representing a game mode, and then putting on a VR headset from above (yes, putting on a VR headset in-game while wearing a VR headset). I have to stand up physically to reach for the top to try to put on the virtual VR headset and it is difficult to grab. Not to mention the floppy disk representing different game modes are not the most easy to understand. This act of inserting a disk and then putting a VR headset looks cool, but to a first-timer with no prior experience of the game, this can be a bummer as they probably do not know what to do to start the game.

What’s the difference between “Endless” and “Don’t Die” game mode? They sound the same to me.

The last shortcoming is the limited freedom of movement. While the VR arm movement are well-executed, I felt that locomotion (movement from one place to another) still leaves a lot to be desired. You can move in the map to a limited extent as long as you have sufficient space in your room, but gameplay is still largely static. The developer addressed this by having the option to teleport to another spot in the room with a grab of a pyramid after slaying all/enough enemies.

Grabbing the pyramid allows you to progress the level in Campaign mode, or change to another location in Endless mode

I understand that this implementation of locomotion is easy for the user, but I really wished I could have the option of moving in all 4 directions in the room like your typical FPS shooter. Decimating an enemy stronghold on the offence is much more fun than defending myself from enemies.

Overall, this is still a fun and interesting VR application that is easy to learn and fun to play. I had so much fun doing numerous goofy antics, like lying down on the floor killing minions or waving my hands to speed up the game. All the while, my friends having a great laugh watching me play. I highly recommend this to anyone who wants a VR application that is easy to pickup and play while still wanting to feel like a badass.

Reunite with your Loved Ones in VR

With the advancements in VR, the applications now include helping people cope with grief at the loss of a loved one.

In 2020, a 2-part Korean television documentary, called Meeting You (or I Met You depending on translation) was released, that told the story of using VR to reunite a mother with her deceased daughter. Jang Ji-sung, mother of four, lost her seven-year-old daughter Nayeon due to a blood disease. After 3 years, Ji-sung keeps pictures of Nayeon and all her favourite toys around the house, promising to remember her forever. But as time passed, she feared she would start to forget her daughter as her memories become more distant. Hence, she hoped to see her daughter for one last time. 

To fulfil her wish, the production team worked to produce this meeting between mother and daughter in the virtual world. They recreated Nayeon to be as accurate as possible, from the face down to the voice and motions. With a set of VR HMD and gloves, Ji-sung was able to meet Nayeon once again and had an emotional reunion. Nayeon’s father and siblings watched tearfully as her mother finally gotten the closure she was seeking from this final meeting.

To make this demo possible, an HTC Vive Pro was used, along with Noitom’s VR gloves that allowed Ji-sung to interact with her daughter. The virtual Nayeon was created by Korean startup Vive Studios over the course of 8 months using various methods. High resolution photos of Nayeon’s younger sister were taken in a process called photogrammetry to serve as the basis for the 3D model, with further refinement done using old photos to better resemble the 7-year-old Nayeon.  In addition, numerous child actors were brought in to recreate Nayeon’s voice according to old recordings, and motion capture was used to record an adult actor emulating the motions and facial experience based on old videos and photos.

The demo itself was run in Unreal Engine 4, with an NPC using the animations and model of the recreated Nayeon. The model included different levels of detail, allowing the pores on Nayeon’s face to be seen when Ji-sung moved nearer, along with interactable hair and clothes. The short demo also included a basic AI, allowing Ji-sung to experience events with her daughter such as her birthday and putting her down on her bed for a nap. While these events were ultimately scripted, it still elicited a very emotional response from Ji-sung.

Although I have had personal experience with many VR applications such as Superhot and Beat Saber, as well as having developed a VR demo in UE4, it was this recreation of a passed loved one that truly struck a chord with me. It demonstrated the ability of VR to create immersive experiences that resonate on a truly visceral and emotional level.

However, there is the elephant in the room with regards to the ethics of such an application. Many see this as impeding the natural healing process of grief. But such accusations were rebutted by Ji-sung, who maintained that it was a positive expirence. Others criticised the experience as ‘fake’. While it is true, given the virtual Nayeon is not a truly robust AI and all the encounters within the demo are scripted, the emotional response from Ji-sung cannot be denied. And neither can the efforts of the developers in creating a virtual Nayeon that is sufficiently realistic while still avoiding the uncanny valley. For performance, capturing the right motions for Nayeon to move like a convincing 7-year-old and the voice to sound like her are also props I would give to the developers.

As to the question of how such a use case can be improved, I would say that only time will tell. If AI progresses to the point that we can accurately recreate personalities of the deceased and generate interactions in real time, we could truly remake a late loved one. And the latest improvements in creating realistic faces can already be seen in the trailer for The Matrix Awakens made in Unreal Engine 5 which features a highly realistic Keanu Reeves.

The viral video of Ji-sung’s final meeting with Nayeon has touched many hearts in its time on Youtube.