GENUIN: Generic multimodal User Interface

; (). GENUIN: Generic multimodal User Interface: Internal Report 05/01. In: InIT/ZHW (Hg.). . (57).

The main result of the project GENUIN is a concept for generating a multimodal user interface (MMUI) for a target application from an abstract description of the tasks and concepts in this application: The generation process starts with a description of the tasks and concepts of the application in an OOA-Model augmented with additional information for the different modalities (GUI, speech input/output). In a first step this description is transformed into an abstract description of the whole MMUI comprising different screens. Each screen is build from a hierarchy of so-called multimodal objects (MMO). An MMO is a new concept of a multimodal component, we propose, which is described further down. In a second step the actual MMUI for a specific device is generated from the abstract description of the MMUI. In this step, the MMUI-designer can fine tune the whole MMUI interaction. The result of this step is either a running MMUI, which finally must be connected with the application logic. Another possibility is to generate a standardized XML-description of the actual MMUI for later integration.
MMOs are a new kind of multimodal components we propose in order to simplify the construction of MMUIs. An MMO is in the simplest case a basic interaction element like a single button or textfield. These MMOs are built according to the model-view-controller pattern. The model of the MMO is responsible for the logic of the MMO. The view of the MMO has different aspects, one for each modality. One aspect of the view is the graphical view well known from corresponding GUI-components. Another aspect of the view of an MMO is the acoustic aspect, which gives an acoustic representation of the MMO and is responsible for the spoken dialogue with the MMO. The controller of the MMO is responsible for the event fusion, i.e. to coordinate the different event coming from the different aspects of the view and compiling them into a semantic event of the MMO. This semantic event is then transmitted to the model of the MMO which normally results in a state change of the model.
In order to demonstrate the feasability and usefulness of these concepts a proof-of-concept demonstrator was implemented based on MMOs.