The Importance of Multi Sense: Focus on Tangibility

Statement

The purpose of this project is to expand the cognitive scope through multiple senses and to stimulate the user’s emotions by actually acquiring the sense of space and material. According to Eagleman, we’re not seeing all the waves out there. Our experience of reality is constrained by our biology. Brains are sampling just a little bit of the world. What the brain sees are electrochemical signals that come in along different data cables, and this is all it has to work with, and nothing more. The brain is really good at taking in these signals and extracting patterns and assigning meaning, so that it takes this inner cosmos and puts together a story of this, our subjective world. If we embrace the senses we have never experienced before, the creative ideas that come from them will be more than we imagine. Also, I think that it is not a real way of communication between human-machine interaction with the machine-like device. Aesthetic appearance, of course, also occupies a large part. However, most important thing is that how its system process. When the system itself is designed to stimulate the human senses to the utmost, the feeling of immersion and the flow at the moment can be a true form of interaction. Chapman described the relationship between human and objects is from embracing. This epoch-making Societal transition has cast us within an abstract version of reality in which empathy and meaning are sought from toasters, mobile phones and other fabricated experiences. Today, empathy is consumed not so much from each other, but through fleeting embraces with designed objects. Empathy is one of the most important virtues that human beings must have. But modern people, especially those who were grown in the high-tech era, are now accustomed to a screen-based flattened world. We can not ignore the advantages of the world on the screen, but according to Roe and Mujis, screen-based technologies have a tendency to pull them away from their physical environment and make them physically and socially isolated. I personally believe that technology must benefit human in a very human way. So my project is located in the largest area of compassionate and empathetic according to my beliefs on technology. At the same time, I started to identify this project in the domains of augmented reality and poetic storytelling. AR is responsible for the technical part, and poetic storytelling is for concept of contents that will be placed in AR. Degrees of immersion are prevalent in most forms of subject-object engagement, and on some level, even the most banal and meaningless tools are capable of inducing some measure of immersive experience, brought about quite simply by enabling users to calmly engage in the process of using the object until conscious influence partially recedes - enabling the conceptual unification of tool and hand. Similarly, when the experience of using and interacting with the product becomes greater than the physical product itself, conscious recognition of the product drops back to make way for full cognition of the experience.

Precedents

Yannier et al. did experiment with three-dimensional physical objects in mixed-reality environments to produce better learning and enjoyment than flat-screen two-dimensional interaction. Following the result of their research, the mixed-reality version of experiment helped children learn more than otherwise equivalent flat-screen versions (laptop or tablet) of the same game. Children interacting with the mixed-reality game reported greater enjoyment and used more meaningful gestures to explain their predictions. These results indicate that the observation of physical phenomena in an interactive game can foster effective learning and inferences. They suggest the similar problem what I am concerned about in this century such as the side effects that life on the screen base can bring. Although they more focused on the field of education, it is a study that draws sympathetic situations by using multi-senses and creates a more efficient educational effect by adding physical objects to their system. Osmo, a platform for multiple physical-digital combination games, is a STEM toy of Tangible Play. They use the combination of an iPad app and physical objects to bring kids back into a social environment with real-world play. This is a platform for multiple physical-digital combination games. This creative toy meets most of my needs that does not alienate people. The domains of my project are empathy, poetic storytelling and AR. Those two precedents show how AR or MR can be work with empathy. I will fill out this form with poetic story and that may be about perception of sense.

Figure 1 Yannier’s MR environment

Figure 2 Osmo, the STEM toy with tangible experience

Description of project:

I propose an augmented reality kit using sense of touch. This workbox allows the audiences to experience extended experience through the sensory cooperation of tactile information and visual information from the materiality of physical objects. The components and user scenarios of this project are as follows. Components: Screened box (cannot look inside) with camera, multiple object packages.

1. The user selects an object package and binds it to the box.
2. When the package is attached to the box, a virtual space is formed on the screen.
3. The screen expresses sensory information reflecting the outline and materiality of the objects in the package in a virtual world view. (For example, if you put a piece of wood inside a dark box, you will see firewood burning on the screen.)
4. The user can experience augmented feeling by modifying the position or shape of the object with his / her hand in the box.

Figure 3 Sketch of final proposal

Bibliography

Chapman, Jonathan. Emotionally durable design: objects, experiences and empathy. London: Routledge, Taylor & Francis Group, 2015. Eagleman, David. David Eagleman: Can we create new senses for humans? | TED Talk | TED.com. March 2015. https://www.ted.com/talks/david_eagleman_can_we_create_new_senses_for_humans?utm_source=tedcomshare&utm_medium=referral&utm_campaign=tedspread. Yannier, Nesra, Scott E. Hudson, Eliane Stampfer Wiese, and Kenneth R. Koedinger. “Adding Physical Objects to an Interactive Game Improves Learning and Enjoyment.” ACM Transactions on Computer-Human Interaction 23, no. 4 (September 2016): 1-31. doi:10.1145/2934668. TAKAHASHI, DEAN. “Osmo uses physical objects to turn iPad games from a solitary experience into a social one (review).” VentureBeat. July 10, 2014. http://venturebeat.com/2014/07/10/osmo-uses-physical-objects-to-turn-ipad-games-from-a-solitary-experience-into-a-social-one-review/. Keith Roe and Daniel Mujis. 1998. Children and computer games – A profile of the heavy user. Eur. J. Commun. 2, 13 (1998), 181–200.