What is it?

The MediaScape XR experience offers users a way to visit a virtual museum and interact with 3D models of cultural artifacts. The users also get to interact with each other and enjoy the experience together. The demo was created by the partnership between Centrum Wiskunde & Informatica (CWI) and the Netherlands Institute for Sound & Vision.

What features make it engaging?

Highly realistic representation of yourself

The users will be captured in 3D and a photorealistic hologram of themselves will be projected into the virtual world [3]. This allows the users to see themselves in the museum, as well as other users nearby. The representation opens up ways to make the experience realistic as the visitors see how the artifacts can interact with their presence and also what others are doing.

Multi-user setup to allow social interactions

The highlight of the experience is the opportunity to learn more from the 3D recreation of a cultural artifact: a costume once worn by Jereney Kaagman, a member of the Dutch rock band Earth and Fire, for the performance of their song “Weekend” in the 1979 television program [4]. The user gets to observe the costume from different perspectives, and even relive the experience of wearing the costume and performing on stage with their friends. This is amazing because it goes beyond just visual appreciation of the cultural heritage. The users get to experience the historical setup and have their rendition/creative output in the same scene.

What features can be improved and how?

Virtualizing museums is one of the most natural applications of XR. We can take on museum tours without leaving our homes and the experience is often claimed to be more immersive. While I think that MediaScape XR has done a good job of showcasing what’s possible in the future, there is still a lot of room for improvement.

Some potential improvements that I can think of are:

• Transcending the physical limitations for the museum tour: while I know that XR applications tend to want to reproduce how people visit museums, I wonder if they can do so solving problems that can’t be eliminated in the real world. For example, as a rather lazy person, I would love to be able to navigate the museum without having to walk around. Seated, if possible. Other potential improvements include having the ability to interact with the exhibits with touch. The current MediaScape XR setup requires a significant room space and it is not possible to have the experience at home. Support for more than two users is also difficult due to the equipment needed. I would hope that in the future, the XR experience can truly be a way to explore without ever leaving one’s home.

• Ways to tune the crowd/environment at museums: just as how MediaScape XR refers to their application as social XR, museum tours can be a very social activity. This means it would be nice to see what others are looking at, have guided tours by museum staff, and exchange comments with other visitors.

• More immersive experience: museums as a way to store artifacts is a somewhat old concept that originated from an era without photos or films. I think that XR applications can do a better job if they don’t just think from the standpoint of how to make artifacts more alive, but also how to make the experience immersive in time and space. For example, allowing visitors to hold conversations with historical figures.

I look forward to further developments of museum tours by MediaScape XR and hope that such tours will become available for museums in Singapore.

References

[1] “DIS wins best demo award at ACM Multimedia 2022,” CWI, [Online]. Available: https://www.cwi.nl/en/news/dis-wins-best-demo-award-at-acm-multimedia-2022/

[2] “Earth & Fire – Weekend (Video),” YouTube, [Online]. Available: https://www.youtube.com/watch?v=4iK5e76auJA

[3] “Sneak peek into future of cultural heritage,” CWI and Netherlands Institute for Sound and Vision, [Online]. Available: https://www.cwi.nl/en/news/cwi-and-the-netherlands-institute-for-sound-vision-gave-a-sneek-peek-into-the-future-of-cultural-heritage/

[4] “With MediaScape XR, Cultural Heritage Gets The Virtual Reality Treatment,” Jingculturecrypto, [Online]. Available: https://jingculturecrypto.com/cwi-netherlands-sound-and-vision-mediascape-xr/

If you’re a gamer, you’ve probably heard about this VR game. I think it’s one of the more innovative works for VR, so please keep reading and (hopefully) nobody explodes.

Introduction

Keep Talking and Nobody Explodes (subsequently shortened as Keep Talking) is a 2015 multiplayer puzzle game initially developed for Virtual Reality (VR) systems. The game is played with at least one “Defuser”, who is given a virtual bomb to defuse, and one or more players as “Experts”, who has to communicate instructions to the Defuser for them to successfully defuse the bomb.

How this works with VR: the Defuser wears the head-mounted display (HMD), and using the controls, they manipulate the bomb within the game. Examples of interactions include cutting wires, pressing buttons, and flipping switches. Meanwhile, the Experts read off a manual to the Defuser, giving instructions on how to defuse the bomb. Of course, the bomb is different (procedurally generated) each time, which means the players need to communicate to each other, as neither can see each other’s bomb/manual.

Why is it engaging?

Firstly, the Defuser is given a lot of pressure in trying to Defuse the bomb within a given amount of time. This goal-oriented behaviour coupled with time pressure and a wealth of interaction types, gives the player strong feelings of satisfaction (if things go well), control, and pressure (if things don’t go well) that increase engagement and hopefully, presence, because the virtual elements command the player’s full attention. Otherwise, they explode (and if done right, the fear of explosion feels very real).

Fidelity

The game is relatively low-fidelity; the visuals are kept simple and distinct for easier visual recognition of patterns, and the controls are not 100% high-fidelity either. You won’t have to actually exert “pressure” (push) to press a button, nor make any cutting motion to cut a wire. However, if you’re using VR controllers, the bomb’s representation in 3D space is quite accurate, such that it feels like you’re handling a real object, and moving/rotating the bomb following the movement of your hands.

This strikes a fine balance between control and frustration. The user is kept in control with regard to rotating the bomb and visually inspecting it, as that is a required interaction to gather information, but there are no unnecessary interactions that create only frustration, that detract from the main point of the game.

Innovative take on VR

Yet, the point of the game is strangely to not be fully present within the virtual environment.

Why? Because while the game is projecting stimuli onto the player, they still have to remain aware of external stimuli — in this case, audio stimuli outside of the VR environment, which are the defusal instructions given by the other players.

This game’s best features are in what it lacks. The creators of the game were inspired by the fact that users had to queue to use VR HMD for demos. Wanting to involve those not using the HMD, they created this game, which creatively turned the lack of HMD into a feature.

Moreover, the use of HMD not only transports the user into the virtual environment, it also removes the user from the actual environment. The creators realised that the HMD not only create stimuli, but also remove stimuli from the actual world, in this case, visual stimuli.

Removing Stimuli

If you’re a gamer from an earlier generation, you may be familiar with split-screen gameplay. One challenge throughout this era of gameplay was in keeping information from others. If you were competing with other players, it was a common tactic to look at their portion on the screen for a tactical advantage, like knowing their location.

Using VR technology, Keep Talking used the lack of real-world visual stimuli as a feature. The lack of information from the external world naturally provides immersion for the Defuser, but also perfectly serves as a game mechanism. Perhaps the use of VR HMD may provide a new angle on having multiple screens and differing streams of information in gameplay.

Possible Improvements

As it stands, the game is very modifiable. You may find several custom-made modules (puzzles on the bomb) or modifications to make the game more challenging, such as having the player drive a bus while defusing the bomb. I imagine if there’s a change that could be made, someone could have already made the modification.

The player base seems to adore challenge, so although I mentioned the game does not have high-fidelity interactions above, perhaps it would be a good optional challenge for the game.

In any case, it has been nearly 10 years since the publishing of this game, and VR technology has definitely become more accessible for the general public due to increase in support, production, and popular usage (such as with the advent of the Metaverse).

The original problem that the game sought to address (Lack of VR HMD sets) can now be set aside to pave the road for further innovations, involving more HMDs. Perhaps, it could now offer two-way gameplay, such as two players both performing the role of Defuser AND Expert, each given incomplete information and needing to help each other progress to defuse their own bomb. A similar concept has been done in this puzzle game, We Were Here, in which clues to solve to one player’s puzzle is found in another player’s world, and vice versa.

---

Altogether, Keep Talking creates a novel experience for co-operative work by encompassing VR’s features (and lack thereof). Apart from recreational activities, it’s been used in work interviews by this company for a quick gauge of how potential hires work with each other, showcasing the use of VR demos beyond pure entertainment. Do give the game a try if you haven’t already, and try not to explode!