VOL.50
March 2019
Special ____ ICT to See, Hear, and Touch!
What Will Be
the Future of ICT,
Advancing While
Engaging
Our Five Senses?
With the Fourth Industrial Revolution unfolding in earnest, our world is being flooded with a series of new technologies. More and more new ICT solutions are emerging every day, such as AI, big data, the cloud, the IoT, and computing technology, just to name a few. Feel with your five senses how rapidly today’s technologies and services are evolving. Here, let us guide you through these new ICT solutions that you will face first-hand in the near future.
AR widely used in various industrial fields, such as design product assembly, drawing review, and entertainment
Virtual Reality and Augmented Reality Evolving into Five Senses-based Hands-on Content
Virtual Reality (VR) and Augmented Reality (AR), once frequently featured in science fiction movies, are now drawing significant attention as key technologies that will drive the Fourth Industrial Revolution. In response, major global companies are striving to make large-scale investments in both VR and AR technologies.
First, VR expands and shares five-sense data obtained in a virtual environment created by computers, allowing us to experience things that are difficult to experience on our own in the real world. This VR technology serves as a magnet to attract clients in various industrial fields. With the upcoming expansion of relevant markets, VR-related content services will be widely available in our daily lives, converging the virtual and physical worlds while transcending the barriers of time and place. To make this VR content even more realistic and absorbing, relevant interface technologies are also expected to evolve, which will help stimulate all our senses, i.e., even the senses of touch, smell, and taste. With this trend, rather than simple viewing, multimedia content of today is going into the realm of five senses-based hands-on experience. For example, in the future, users physically distant from each other might be able to cooperate in real-time in a virtual environment as if they were together in the same physical space (e.g., in remote virtual meetings and remote orchestra performances).
In the meantime, AR creates augmented objects in real, physical space to provide users with more extensive hands-on services and information. AR applications primarily apply to marketing, media, entertainment, and education fields. So far, the focus of AR has been on improving the recognition and tracking of target objects in terms of precision, so that the technology can be widely used in various industrial fields, such as medicine and manufacturing, extending its application scope to a practical level. However, the release of the PICO projector, which is compact, light, and capable of projecting images using an embedded battery without an external power supply, is expected to accelerate the advance of projection-based wearable AR technologies.
PICO Projector
A compact beam projector that, while linked to mobile devices such as smartphones, displays various content on a large beam projector screen
Glove-shaped controller allowing users to feel as if they are actually touching an object in the virtual space and thereby allowing them to estimate its size and shape © Dexmo / AR medical simulation that provides additional information by projecting virtual images onto real-life backgrounds
Dexmo, a glove-shaped input device intended for VR systems, developed by Dexter Robotics, a Chinese startup © Dexmo
Haptic Reproduction Technology Enabling a Touch to Be Sensed without Touching
In early 2016, for the first time in the world, Oculus in the US commercialized a VR device that, when worn like goggles, enables the user to view a virtual world that covers the entire field of sight. Three years later, VR devices are now becoming typical components of various game consoles and applications. Also, going one step further, the technology is poised to evolve into Mixed Reality (MR), combined with the sense of touch. MR is a term that conceptualizes the Reality-Virtuality Continuum. Simply put, the technology creates a smart environment where the real and virtual worlds are naturally connected, thereby providing users with enriched hands-on experiences. For example, users can have fun with virtual pets built on the floor while playing with and touching them, or they may create a virtual gaming environment within the real, physical space. Also, they can feel the abrasive texture of a virtual brick when touching it in games or feel the weight of a water bottle when lifting it virtually.
A case in point is EXOS, a glove-shaped haptic reproduction device, released by exii (Japan) in January of 2017. EXOS reproduces the tension of muscle fibers when touching objects and is characterized by its ability to minimize unfamiliarity through the delicate distribution of power. EXOS has drawn significant attention from many game companies, and some automakers are preparing to introduce EXOS as training equipment for unskilled workers. Meanwhile, Dexter Robotics, a Chinese company, released a prototype of its glove-shaped haptic model named Dexmo. When a user wearing the device attempts to hold a wood stick in the VR space, he or she cannot clench the fist tighter than the thickness of the stick. When holding a rubber ball in the hand, the user may feel as if his or her fingers were clenching around the ball. In the US, a prototype Force Jacket developed by Disney Research was released in April of 2018. Force Jacket is designed to transmit high frequencies and power to the upper half of the body through the contraction and expansion of its airbags and high-frequency vibrations. Force Jacket allows the wearer to feel first-hand the sense of hitting and to sense not only light touches but also delicate feelings of hugging, collisions, or being coiled up by a snake when playing VR games.
ETRI is currently developing laser-based haptic reproduction technology that does not require the user to wear a separate device. This technology uses heat generated by laser irradiation to induce mechanical deformation in a film. The deformed film may have an embossed surface or cause the feeling of trembling, thereby allowing the user to experience the sense of touch reproduced accordingly.
Today, this haptic reproduction technology is being actively combined with the VR, AR, and MR fields. Going forward, the technology is expected to be applied to online shopping, remote operation, and the implementation of holograms. Also, it will go one step further beyond existing temporal and spatial barriers, even making it possible for users to experience things that are difficult to experience in real-life settings. Researchers at home and abroad are eager to bring the technology to the next level through multifaceted approaches.
Mixed Reality (MR)
An environment in which physical objects and virtual objects can interact with one another by combining the physical world with VR
VR entertainment advancing toward implementation of senses of smell and taste, aside from senses of vision and hearing
Alarm clock using sense of smell developed by Sensorwake © 2018 terry johns All rights reserved.
Electrically Stimulated Senses of Smell and Taste
Aside from senses of vision, hearing, and touch, there has been growing interest in extending the application scope of ICT solutions to senses of smell and taste. These technologies make people feel as if they are tasting or smelling specific foods while delicately stimulating their sensory organs, such as tongues and noses, by applying weak electrical signals. These technologies, when commercialized, will make it possible to transmit human senses wherever the users are.
Researchers from the Imagineering Institute in Malaysia recently developed digital olfactory technology that stimulates users’ sense of smell to make them feel as if they are smelling something specific. The researchers inserted electrodes and cables equipped with cameras into the human nose, where they electrically stimulated the epithelial cells of the nose while watching camera images. The human nose can smell when its olfactory cells come in contact with chemical substances that generate smells. The researchers attempted to replace this process with electrostimulation.
Yoshiaki Miyashita, a professor at Japan’s Meiji University, developed technology that creates tastes, including an electric gum that uses the piezoelectric effect to create a flavor that does not diminish over time. The piezoelectric effect is a phenomenon in which a change in the shape of an object generates electricity and, when electricity is applied, the shape of the object changes. The research team developed a thin plastic film made of piezoelectric material. This film, when chewed like gum, generates and releases electricity little by little, thereby stimulating the taste cells of the tongue. Meanwhile, in 2016, Rekimoto Lab, a Japanese startup, developed jointly with the University of Tokyo an electric fork that stimulates salty flavors. When the button in its handle is pressed, a weak electric current is generated at the tip of the fork. This weak electrostimulation causes the user’s tongue to taste salty flavors; the intensity of the flavor can be adjusted in five steps depending on the intensity of the applied current. Reportedly, they are currently developing a form of electric candy that will allow diabetics to fully enjoy sweetness without worrying about glycemic control.
ETRI’s Affective Interaction Research Group is now developing an image recognition-based smell generating device capable of automatically releasing specific smells that fit the context of given VR images. This technology controls the release of smells, synchronizing them with the set time frames of given VR images displayed in Head Mounted Display (HMD) equipment. Based on deep learning technology, this system collects and categorizes video images related to smell generation. Later on, when a specific image is inputted, it is recognized and converted into smell-generation content, while the system triggers the preset smell cartridges to automatically generate smells accordingly. At the current level, the technology can automatically release various smells, such as those of coffee, oranges, and roses. Still, there is a long way to go before we can help people enjoy what they want to smell and taste at the very right moment and in a precise manner. It is still very difficult to implement the delicate process in which a previously-provided scent must disappear before the release of a new one. Also, technology that synchronizes the delivery of smells with audiovisual content is required. There are many barriers to overcome, but researchers will continue to make their way toward the objective of providing virtual experiences that allow users to enjoy their five senses.
Head Mounted Display (HMD)
An image display device that the user wears like a pair of glasses to enjoy VR and AR applications