Multimodal Interaction System for Home Appliances Control

Multimodal Interaction System for Home Appliances Control

multimodal interaction

speech recognition

gesture recognition

smartphone application

smart home.

This paper proposes a way to control home appliances using a multimodal interaction system such as speech, gestures, and smartphone applications. The sensor to capture speech, in the Indonesian language, and gestures from users are Kinect v2. Speech recognition process with the Google Cloud Speech, gesture recognition process with the K-Means clustering, and dialogue system process with the finite state machine. Users can also use the smartphone application to remotely control home appliances through mobile devices such as tablets or smartphones that are connected directly to the real-time database. There are two output responses from this system, namely the audio response generator to provide feedback to the user through the sound of the computer speaker and also provide an action to control home appliances use Esp8266. The average level of accuracy testing of interaction using dialogue systems and gesture are 92.5% and 79,25%. Interaction using dialogue systems is better than gesture. Smartphone applications can control home appliances properly.

Fakhrurroja, Hanif

Machbub, Carmadi

Prihatmanto, Ary Setijadi

Purwarianti, Ayu

International Journal of Interactive Mobile Technologies (iJIM); Vol. 14 No. 15 (2020); pp. 44-67

1865-7923

International Association of Online Engineering (IAOE), Vienna, Austria

2020-09-11

Copyright (c) 2020 Hanif Fakhrurroja, Carmadi Machbub, Ary Setijadi Prihatmanto, Ayu Purwarianti

application/pdf

eng

info:eu-repo/semantics/article

info:eu-repo/semantics/publishedVersion

Peer-reviewed Article