Unit 10.1: Virtual Reality
Synopsis
Virtual Reality (VR) has been a useful tool for many applications as well as designing experiences: from training simulations to immersive gaming. Although VR has been around since the late 60s and technological advances have made it more and more popular these days, VR has yet to become a feature of the average household. We have some way to go before we can become the natives of this new medium. So what can we do with it in the meantime? What are the benefits, constraints and challenges of this new medium, and how can we use it to create new artistic works?
Introduction
The term “Virtual reality” (VR) has been used throughout the years in various ways depending on the context. In the 14th century the word “virtual” was used to describe the state of being something in essence or effect but not in presence or in a physical way. In the late 1930’s Antonin Artaud used the term to describe the characters and objects of a theatrical screenplay in his collection of essays “Le Théâtre et son double” (1938). It is Jaron Lanier, however, who is credited with popularizing the term as well as founding the field of virtual reality.
Nowadays, four main definitions co-exist to describe possible relations between the real and virtual worlds: mixed reality (MR), extended reality (XR), augmented reality (AR) and virtual reality (VR). These terms can cause some confusion and are not easy to differentiate, but we can begin by considering them as two poles of a continuum, rather than two separate entities (see illustration below).
Extended Reality (XR) is a broad term that refers to immersive technologies that allow human-machine interaction between computer-generated virtual environments and the real world.
Mixed Reality (MR) describes the hybridity between reality and virtuality.
Augmented Reality (AR) takes place in the physical world. Through AR, this physical world is enriched by computer generated, mostly perceptual, information. That means that an object that exists in the physical world can be “augmented” to something different and new in real-time. For example, a tree pops out of a ball—the ball is there physically in the real world and the tree is added virtually into the real world. On the contrary, virtual Reality (VR) takes place completely in the virtual world without any reference to the real world.
For AR experiences, specific technologies are used, such as the Microsoft HoloLens or the Magic Leap. Similarly, VR experiences rely on specialized technologies such as the HTC Vive or the Oculus Rift, both of which are head-mounted displays (HMD)
Some History and the Current State of VR Technologies Now
The vision of VR already existed in the 1960s. The cinematographer Morton Heilig developed the “Sensorama” (1962), an apparatus for a holistic cinematic experience. Through a boxed-like display the person seated on a chair could see images, which were displayed on every side of the box, simulating a 3D visual effect. Additionally, the person could smell different aromas, hear stereophonic sounds and have their body tilted since the chair was able to rotate in both axes. The device was patented in the USA and is considered to be the ancestor of 4D cinema.
Some years later, Ivan Sutherland published an essay called “The Ultimate Display” (1965) where he describes a display that is connected to a digital computer that “gives us a chance to gain familiarity with concepts not realizable in the physical world. It is a looking glass into a mathematical wonderland”. Sutherland, who is considered to be the pioneer of computer graphics, was able to construct the first head-mounted display (HMD) which was extremely heavy and therefore was hanging from the ceiling. The idea was that the one would look through the display and by rotating one’s head to the left or to the right the person could move and navigate through the virtual graphic environment. On this specific HMD, which is the ancestor of current HMDs, Sutherland published the paper “A head-mounted three dimensional display” (1968).
It is impossible to sum up in just a few lines all of the technological advances that took place between the 1970s and today, however it is worth pointing out NASA’s developments for the Virtual Interface Environment Workstations (1988). An HMD as well as a data glove were designed and developed in order to simulate operational tasks related to space aviation such as telerobotic or telepresence control of remotely operated devices.
The biggest milestone in the development of virtual reality products was the company Virtual Programming Languages Research founded by Jaron Lanier in 1984. The most well-known products were the DataGlove and the EyePhone. In 1994 the Forte Technologies company brought to market the VFX1 Headgear, the first affordable HMD for the end user. In 2011 Sony produced the HMZ-T1 headset, the first 3D capable HMD with a high definition OLED display. Nowadays, the most popular VR toolkits are the Oculus Rift and the Vive Toolkit by the company HTC.
VR technologies are used in various fields for different purposes from training simulations up to post-traumatic therapies and entertainment. There is a big variety of VR headsets and controllers (among others: Oculus Rift touch controllers, Vive controllers, Valve Index controllers or PSVR controllers) that allow all kinds of interaction and creative possibilities. However, the cost of these technologies is still high for the end user, which means that VR isn’t freely accessible to everyone yet. Many of us don’t possess a VR headset at home and maybe some of us have not even experienced VR yet. This indicates that we are lacking practice with this new medium and at this moment we cannot be considered native users of it. We have a lot to learn from a medium that is continuously developing. What are the benefits, the constraints and the challenges that this new medium provides?
Benefits of VR Technologies
Being able to create new imaginary worlds that don’t and can’t exist in the real world, is one of the biggest challenges and excitements of virtual reality. But do these worlds have to obey the laws of physics? A question to which we will return later.
The medium is suitable to transform video games into a highly immersive experience. The user can dive in the video-game world and be part of it. VR has been used also as a technology for the realization of escape games. In the multiplayer VR Escape Room “Huxley”, the user is immersed in the year 3007AD where mankind has been replaced by machines and has 45min to solve a puzzle and save the world. In the VR application “Dream of Dali” the user has the possibility to explore the surrealistic fictional world of Salvador Dali’s painting “Archaeological Reminiscence of Millet’s Angelus”, where over-sized creatures pass over the head of the user. Another function that VR technologies enable is the re-creation of worlds that either don’t exist or aren’t accessible anymore. For example, we can restore the Colosseum to its original state and visit it virtually or take a walk through the streets of San Francisco just by connecting a VR headset with the Google Earth application.
VR has become a permanent feature for immersive learning experiences providing, in many fields, the possibility of recreating real life settings and simulations of work challenges. This has been particularly beneficial in areas that deal with purpose-created scenarios and crisis management, hands-on experiences, skills analysis and decision making, and remote training.
VR technologies can be also very useful for art mediation and education. In the project “Moving Sound Pictures” paintings are transformed into 3D spaces in which the user has the opportunity to touch, move or enlarge the objects in the painting and to create music through these actions. But it’s not just about interaction and music. In addition to these playful actions, the user gets information about the artwork and its history. While the user interacts with the VR environment, the transfer of knowledge takes place on this virtual stage at the same time. So far, two parts of the project have been completed. In the first part called “The Abstract Painters” four paintings of the Abstract Era between 1916 - 1924 by Kasimir Malevich, El Lissitzky, Piet Mondrian and Wassily Kandinsky were transferred into a VR environment and dressed up musically. The user enters a gallery designed in the style of the 1920s, with a wallpaper in wine red colour and a shiny brown wooden floor. The paintings hang on the walls like in an exhibition and act as gateways into the walk-in virtual rooms: in the VR version of Malevich’s painting “Suprematism” you can make the objects sound with two mallets, in the VR version of Kandinsky’s “Merry Structure” you can touch the objects with your hands and thus create music.
In the second part called “Dali’s Persistence of Memory” the painting of the same name by Salvador Dali was transferred into a virtual interactive landscape outdoors. In a desert landscape surrounded by the sea, the sound of ticking melted clocks is combined with the sound of the sea. The user can move freely and discover this “musical” world through teleportation.
These environments don’t strictly follow the laws of physics. For example, yellow circles and multi-coloured triangles are gliding and move smoothly on the air without falling down to the ground as the case would be in the physical world. For the most part, we expect that things in a VR environment behave exactly as they would behave in the physical world. Whenever something breaks these “unwritten” rules, it makes us feel uncomfortable or weird because it does not fulfill our expectations. For example, in Kandinsky’s VR environment, the big yellow circle stands up near the ceiling. Normally the object, if it’s not hanging from somewhere, should fall down because of gravity, but in a virtual world gravity isn’t a “must”. Of course, we can simulate it but we do not have to. While VR makes it possible to create worlds that don’t obey the laws of physics, perhaps we must ask ourselves whether we “should” create such worlds. On the other hand, we have so much freedom to create environments that would seem fantastic to our reality, and that is one of the biggest advantages but also the biggest challenge in VR creation.
Constraints and Challenges
HMDs are a window to a virtual world. When “wearing” an HMD we are suddenly cut off from the physical world and we have the feeling that we are physically present in a non-physical world. As such, total immersion is reached through VR technologies. But what happens with our senses and especially our body? Our body is somehow trapped between two worlds: the physical and the virtual. Our brain and body are going to simulate whatever happens in the virtual world. That means that if you are on a roller coaster in VR but you are standing in the physical world, you will still feel nauseous and possibly even get sick. This sensory discrepancy is called “virtual reality sickness”, which is similar to motion sickness. Walking or wandering around in VR, the so-called locomotion, can be enabled by using handheld controllers. Here, though the user can only walk within a limited physical space in the real world without necessarily having the same sized space in the virtual world. Teleportation (selecting the spot where the user wants to teleport to in the space) is one solution to solve the discrepancy between virtual and physical space. If an obstacle is in your way in the virtual world, you will feel the need to jump over it in the physical world although the object doesn’t physically exist.
Inspired by the problem of how our perception works in virtual worlds, in 2016 Rebecca Allen created the VR art installation INSIDE (https://vimeo.com/420957109). In a narrative VR environment, the user travels from a desert to a prehistoric cave. The cave functions as a representation of the human brain where the real world mixes with the “unreal”. Images, sounds, and virtual objects are mixed in this cave. The artist to what extent our sensory perception can be altered when we are immersed in a VR environment and the role of the human body in VR. How can we enjoy such an environment, and do we find this pleasant?
Another question that arises is that of the feeling of presence or absence when emerging in VR environments. Do you have the feeling of really being there or it is just an illusion? And what is that it makes the experience real or illusive? High-end graphics? Or a 100% representation of the real world?
In our current moment, none of these questions are easy to answer. The puzzle of finding the right balance between actions in the virtual world and the reaction of the body in the physical world, as well as what this combination can make us feel, is already mind breaking.