Multimodal Architecture and Interfaces

Multimodal Architecture and Interfaces

Developed by	MMI Working Group (World Wide Web Consortium)
Initial release	April 22, 2005
Latest release	Recommendation (October 25, 2012)
Type of format	Recommendation
Standard	Last Version
Website	http://www.w3.org/2002/mmi/

Multimodal Architecture and Interfaces is an open standard developed by the World Wide Web Consortium since 2005. It was published as a Recommendation of the W3C on October 25, 2012. The document is a technical report specifying a multimodal system architecture and its generic interfaces to facilitate integration and multimodal interaction management in a computer system. It has been developed by the W3C's Multimodal Interaction Working Group.

Description

The Multimodal Architecture and Interfaces recommendation introduces a generic structure and a communication protocol to allow the modules in a multimodal system to communicate with each other.

This specification proposes an event-driven architecture as a general frame of reference focused in the control flow data exchange. It can be used to determine the basic infrastructures needed to command the application's multimodal services.

The architecture is also proposed to facilitate the task of implementing several types of multimodal services providers on multiple devices: mobile devices and cell phones, home appliances, Internet of Things objects, television and home networks, enterprise applications, web applications, "smart" cars or on medical devices and applications.

Logical structure

The MMI Architecture in the MMI Runtime Framework

Multimodal Architecture and Interfaces is the specified description of a larger services infrastructure called The Runtime Framework which provides the main functions that a multimodal system can need. This framework is at a higher level of abstraction than the MMI architecture. The MMI Runtime Framework is the runtime support and communication modules of the multimodal system while MMI Architecture is the description and the specification of its main modules, its interfaces and its communication modes.

The MVC design pattern in the MMI Architecture

The Multimodal Architecture and Interfaces specification is based on the MVC design pattern, that proposes to organize the user interface structure in three parts: the Model, the View and the Controller. This design pattern is also shown by the Data-Flow-Presentation architecture from the Voice Browser Working Group.

Recursion in the MMI Architecture

A particularity of this architecture is that although the presentation layer represented by theView has traditionally been associated with graphical interfaces; this recommendation abstraction generalizes the View to the broader context of the multimodal interaction, where the user can use a combination of visual, auditory, biometric and / or tactile modalities.

The MMI Architecture recommendation distinguishes three types of components: the Interaction Manager – IM, the Data Component – DC and the Modality Components – MC. This distinction is similar to the separation between the Controller, the Model and the presentation documents of the View in the MVC pattern.

Another characteristic is recursion. The modules are black boxes and it is possible to encapsulate several components in a more complex component, which communicate with an Interaction Manager at a higher level. In this way, the architecture follows the nested dolls principle .

The specification also covers the issues of a distributed implementation on multiple material resources in a network or a centralized implementation, with all the modules installed in a single material support. Information sharing between modules is loose coupled. This promotes low dependence between modules, reducing the impact of changes in one module on other modules, and facilitating the modules' reuse. In this way, the modules have little or no knowledge of the functioning of any other modules and the communication between modules is done through the exchange of messages following a precise communication protocol provided by the architecture's API.

The MMI architecture modules

MMI Architecture Modules

The interaction manager

The interaction manager is a logical component, responsible for all message exchanges between the components of the system and the multimodal Runtime Framework. It is a communication bus and also an event handler.

Each application can configure at least one Interaction Manager to define the required interaction logic. This controller is the core of the multimodal interaction:

• It manages the specific behaviors triggered by the events exchanged between the various input and output components.
• It manages the communication between the modules and the client application.
• It ensures consistency between multiple inputs and outputs and provides a general perception of the application's current status.
• It is responsible for data synchronization.
• It is responsible for focus management.
• It manages communication with any other entity outside the system.

The modality components

The modality components are responsible for specific tasks, including handling inputs and outputs in various ways, such as speech, writing, video, etc..

These are logical entities that handles the input and output of different hardware devices (microphone, graphic tablet, keyboard) and software services (motion detection, biometric changes) associated with the multimodal system. For example, (see figure below), a modality component A can be charged at the same time of the speech recognition and the audio input management. Another modality component B can manage the complementary command inputs on two different devices: a graphics tablet and a microphone. Two modality components C, can manage separately two complementary inputs given by a single device: a camcorder. And finally, a modality component D, can use an external recognition web service and only be responsible for the control of communication exchanges needed for the recognition task.

Input abstraction in the modality components

In all four cases the system has a generic modality component for the detection of a voice command input, despite the differences in implementation. Any modality component can potentially wrap multiple features provided by multiple physical devices but also more than one modality component could be included in a single device. To this extent the modality component is an abstraction of the same kind of input handled and implemented differently in each case.

For this reason, the W3C recommendation currently does not describe in detail the structure or implementation of the modality components. It focuses only on the need of a communication interface with the Interaction Manager and the need of an implementation that follows a specific communication protocol: the Life-Cycle Events.

Public and private data component

The data component

The data component's primary role is to save the public data of the application that may be required by one or several modality components or by other modules (e.g., the Framework's session module).

The data component can be an internal module A or an external module B to the Interaction Manager (see Figure). This depends on the implementation chosen by each application. However, the Interaction Manager is the only module that has direct access to the Data Component and only the Interaction Manager can view and edit the data and communicate with external servers if necessary. As a result, the Modality Components must use the Interaction Manager as an intermediary to access the public data of the multimodal application.

However, for the storage of private data, each Modality Component can implement its own Data Component. This private Data Component can also access external servers B (see Figure) and keep the data that the Modality Component may require, for example, in the speech recognition task. This can be the case of an implementation following the principle of nested dolls given by the MMI architecture recommendation.

Communication protocol between different modules

In the MMI architecture, the communication protocol is asynchronous, bi-directional and based on the exchange of event notifications that are raised by the system following a user action or some internal activity.

This protocol defines the exchange mode and how to establish and end the communication between modules. In the case of this specification, it is reflected in the Life-Cycle Events. These are six standard control events that are proposed for controlling devices and material services (such as a video player or a sound reproduction device) and two notifications proposed for monitoring the current status of the multimodal system.

The MMI Life-Cycle Events

The Standard Life-Cycle Events

The specification recommends eight standard Life-Cycle Events specified as pairs of Request > Response exchanges:

• NewContext ( NewContextRequest / NewContextResponse )

Indicates the creation of a cycle of interaction (context) between zero, one or more users with one or multiple modality components. The context is the longest period of interaction in which the modules must keep the information available.

The context is associated with the semantics of the user's and system's activity during the interaction cycle. This allows to the implementation decide whether to keeping the information still makes sense for the execution of the current activity.

Usually the context is created from user input. The event is normally sent by one or more modality components to the interaction manager that must answer the query.

For example, a modality component responsible for managing the inputs associated with the finger gesture ( touchmove ) in a web page viewed on a touch screen. At the beginning of the physical interaction, the modality component will send the query:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:newContextRequest requestID="myReq1" source="myPointerMC.php" target="myIM.php" data="myMCStatus.xml" />
</mmi:mmi>

At this request the Interaction Manager will respond:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:newContextResponse requestID="myReq1" source="myIM.php" target="myPointerMC.php" context="myContextID1" status="success" />
</mmi:mmi>

It is also possible that the Interaction Manager is at the origin of the request. In this case the syntax remains the same but the value of source and target properties must change.

• ClearContext ( ClearContextRequest / ClearContextResponse )

Sent by the interaction manager, the event marks the end of the cycle of interaction (context) and request the freed of the resources that were allocated to the current interaction context. The ClearContext event may be associated with the end of user or system activity. For example:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:clearContextRequest requestID="myReq2" source="myPointerMC.php" target="myIM.php" context="myContextID1" />
</mmi:mmi>

Upon receipt the request, the modality component may give as an answer:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:clearContextResponse requestID="myReq2" source="myPointerMC.php" target="myIM.php" context="myContextID1" status="success" />
</mmi:mmi>

• Prepare ( PrepareRequest / PrepareResponse )

Sent by the interaction manager, this event control signals to the modality component that it must be prepared to start its task and that it can load data that will be required to perform its work. For example:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0" xmlns:svg="http://www.w3.org/2000/svg">
<mmi:prepareRequest requestID="myReq3" source="myIM.php" target="myDisplayMC.php" context="myContextID2">
<mmi:content>
<svg:svg width="100%" height="100%" version="1.1">
<rect width="300" height="100" style="fill:rgb(0,0,255);stroke-width:1; stroke:rgb(0,0,0)"/>
</svg:svg>
</mmi:content>
</mmi:prepareRequest>
</mmi:mmi>

If there are multiple documents or data to be loaded during the preparation phase, the Interaction Manager could trigger the PrepareRequest event several times without starting the task after each request. Nevertheless, each request call must be answered. In our example, if the content has been properly preloaded the answer is:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:prepareResponse requestID="myReq3" source="myDisplayMC.php" target="myIM.php" status="success" context="myContextID2">
</mmi:prepareResponse>
</mmi:mmi>

• Start ( StartRequest / StartResponse )

Sent by the Interaction Manager this control event signals to the Modality Component that it may start its task. For example:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:startRequest requestID="myReq4" source="myIM.php" target="myPlayerMC.php" context="myContextID3" data="myPlayerParams.xml">
<mmi:contentURL href="myAnimation.swf"/>
</mmi:startRequest>
</mmi:mmi>

If during the execution of its work the Modality Component receives a new StartRequest event it may either start the new task, or report the failure. In the case of a successful execution the answer must be:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:startResponse requestID="myReq4" source="myPlayerMC.php" target="myIM.php" status="success" context="myContextID3">
</mmi:startResponse>
</mmi:mmi>

In the case of a failure, the response may be for example:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:startResponse requestID="myReq4" source="myPlayerMC.php" target="myIM.php" status="failure" context="myContextID3">
<mmi:statusInfo>
NoContent
</mmi:statusInfo>
</mmi:startResponse>
</mmi:mmi>

• Cancel ( CancelRequest / CancelResponse )

Sent by the Interaction Manager, this control event signals to the Modality Component that it should stop its current task. For example:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:cancelRequest requestID="myReq5" source="myIM.php" target="mySpokerMC.php" context="myContextID4" Immediate="true"/>
</mmi:mmi>

The Modality Component response may be:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:cancelResponse requestID="myReq5" source="mySpokerMC.php" target="myIM.php" context="myContextID4" status="success" data="myMCStatus.xml"/>
</mmi:mmi>

• Pause ( PauseRequest / PauseResponse )

Sent by the Interaction Manager, this control event signals to the Modality Component that it must pause its current task. For example, the request for a Modality Component that manages the input of a graphics tablet may be:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:pauseRequest requestID="myReq6" source="myIM.php" target="myWriterMC.php" context="myContextID5" immediate="false"/>
</mmi:mmi>

And the response of the Modality Component may be:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:pauseResponse requestID="myReq6" source="myWriterMC.php" target="myIM.php" context="myContextID5" status="success" data="myMCStatus.xml"/>
</mmi:mmi>

• Resume ( ResumeRequest / ResumeResponse )

Sent by the Interaction Manager, this control event signals to the Modality Component that the task previously paused should resume:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:resumeRequest requestID="myReq7" source="myIM.php" target="myWriterMC.php" context="myContextID5"/>
</mmi:mmi>

The Modality Component's response may be:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:resumeResponse requestID="myReq7" source="myWriterMC.php" target="myIM.php" context="myContextID5" status="success"/>
</mmi:mmi>

• Status ( StatusRequest / StatusResponse )

It is sent either by the Interaction Manager or by the Modality Components. This event indicates whether the cycle of interaction is still running, that is, if the context is "alive". For example, a Modality Component may need to monitor the status of the system to stop a process in case of fault or standby. In this case, it may send to the Interaction Manager :

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:statusRequest requestID="myReq8" source="myRecorderMC.php" target="myIM.php" requestAutomaticUpdate="true" />
</mmi:mmi>

Given that the request don't have the context parameter, the controller should return the system or the host server status and not the status of the current interaction context. To this request, the Interaction Manager may respond:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:statusResponse requestID="myReq8" source="myIM.php" target="myRecorderMC.php" status="alive" AutomaticUpdate="true" />
</mmi:mmi>

The Notifications

The specification also recommends two types of notifications, which one, the Extension Notification, can contain control or command data to handle devices or material services. For this reason this notification is regarded like an exception, a standard control event that is not described as a pair Request > Response (in a previous version it was called the Data Event). The two notifications are:

• Extension ( ExtensionNotification )

This event is used to communicate application-specific control data or any other needed data. It can be generated either by the Interaction Manager or by a Modality Component. It ensures the extensibility of the architecture by giving a generic API dedicated to the application-specific needs. For example, a Modality Component that handles a DVD player may indicate to the system an interaction with the DVD's main menu:

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
     <mmi:extensionNotification requestID="myReq9" source="myPlayerMC.php" target="myIM.php" context="myContextID6" name="playerNavigation">
     <applicationdata>
         <data id="menu" selected="true" />
     </applicationdata>
     </mmi:extensionNotification>
</mmi:mmi>

• Done ( DoneNotification )

This notification is sent by the Modality Component to the Interaction Manager when it has finished its task. For example, when a recognizer Modality Component has finished the image recognition task (finding a face):

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
     <mmi:doneNotification requestID="myReq10" source="myDetectorMC.php" target="myIM.php" context="myContextID7" status="success">
         <mmi:data>
           <data id="detectionList">
               <users>
                   <user id="u58" confidence=".85" />
                   <user id="u32" confidence=".75" />
                   <user id="u87" confidence=".60" />
               </users>
           </data>
         </mmi:data>
     </mmi:doneNotification>
</mmi:mmi>

External links

Standards and notes of W3C

• MMIF: Multimodal Interaction Framework by James A. Larson, T.V. Raman and Dave Raggett, Ed., W3C, 2003. This W3C Note proposes a framework that describes the main generic components of a multimodal system. This framework is a basis for developing multimodal applications, it suggest which language to use, which scripts, which styles and other useful resources.
• SCXML: State Chart XML. A State Machine Notation for Control Abstraction by Jim Barnett et al. Ed. W3C, 2006. SCXML is a state machine language for a general use and based on events. It can be used: as a dialog control language invoking different types of recognition; as a metalanguage for voice applications that can also control access to databases and business logic modules, such as a multimodal control language combining VoiceXML dialogues with other modalities, including keyboard, mouse, ink, vision, haptic devices, etc.; as a management language for extended call centers and finally as a control language for general processes in other contexts not involving speech processing.
• CCXML: Voice Browser Call Control Version 1.0 by R.J. Auburn Ed., W3C, 2005. CCXML allows the control of voice calls in dialog systems. It can handle various types of media for example in systems using VoiceXML.
• EMMA: Extensible Multimodal Annotation markup language by Michael Johnson and al. Ed., W3C, 2005. EMMA is an XML format to mark up the interpretation of user input with application-specific information such as the level of trust, the timestamp, the input modality and relevant options to the recognition of the input data.
• MMIUse: Multimodal Interaction Use Cases by Emily Candell and Dave Raggett, Ed., W3C, 2002. This W3C's Note describes several use cases for multimodal interaction and presents them in terms of the multiple device capacities and required events for each use case. It helps to assemble the various components of a multimodal application in which a user can interact with multiple modalities, for example, using speech, writing, shortcuts, voice commands to have an audio or visual output result.
• SMIL: Synchronized Multimedia Integration Language Version 2.1 by Dick Bulterman and al. Ed. W3C, 2005. SMIL is a language that allows you to write interactive multimedia presentations describing the temporal behavior of the presentation, by combining links to the media describing the layout of the presentation on the screen. It also provides timing and synchronization to other languages that can need it.
• VoiceXML: Voice Extensible Markup Language Version 2.0 by Scott McGlashan and al. Ed., W3C, 2004. It allows to create pages with which a user can interact with his voice or any sound. It allows, also the creation of a dialog between the user and a web page with a synthesized speech, digitized audio signal, a voice input or a DTMF tone.
• HTML: HyperText Markup Language Version 4.01 by Raggett and al. Ed., W3C, 1999. It is used to describe the presentation and the text of web pages and include in the pages of links to additional texts, media resources (images, video, sound) and interactive media resources ( forms, animations, interactive universe 3D ) . This description is made with tags written in a structured code and can be completed by information about a given graphic style ( CSS ) or a programmed interaction by a script ( ECMAScript ).
• SVG: Scalable Vector Graphics 1.1 by Jon Ferraiolo and al. Ed., W3C, 1995. SVG is a file format for describing adaptable vector graphics.
• XMLSig: XML-SignatureSyntax and Processing by Eastlake and al. Ed., W3C, 2001. It is used to authenticate the author of the web page and ensure its integrity through digital signatures on XML documents.

Other standards

• RFC 7230: HTTP/1.1 Message Syntax and Routing. IETF, 2014.
• RFC 2119: Key words for use in RFCs to Indicate Requirement Levels in IETF, 1997.
• RFC 2396: Uniform Resource Identifiers in IETF, 1995.

Scholar references

• Multimodal Interaction in Distributed and Ubiquitous Computing by Pous, M. et Ceccaroni, L, Internet and Web Applications and Services (ICIW), 2010 Fifth International Conference on, Barcelona, Spain, 2010.
• A Multimodal Interaction Manager for Device Independent Mobile Applications by Wegscheider, F. et al., Internet and Web Applications and Services (ICIW), 2010 Fifth International Conference on, Barcelona, Spain, 2010.
• STANCIULESCU, Adrian (2008). A Methodology for Developing Multimodal User Interfaces of Information Systems. Presses universitaires de Louvain. ISBN 978-2-87463-114-6. 
• A Model for Adaptive Multimodal Mobile Notification by William Brander, Nelson Mandela Metropolitan University, Port Elizabeth, South Africa, 2007.
• Providing multimodal context-sensitive services to mobile users by Ardit, C et al., National Event on Virtual Mobile Guides, Turin, Italy, 2006.
• Papers presented to W3C's Multimodal Architecture and Interfaces Workshop, 19–20 July 2004. Sophia Antipolis, France
• Mobile Field Data Entry for Concrete Quality Control Information by Irina KONDRATOVA dans Proceedings of the European Conference on Products and Processes Modelling (ECPPM 2004).NRC Publication Number: NRC 47158.

Links

• W3C Drafts New Multimodal Standard by Leonard Klie, speechtechmag, Feb, 17, 2009.
• W3C's Multimodal Architecture and Interfaces Workshop, 16–17 Nov. 2007. Keio University, Japon.
• The W3C Multimodal Architecture, Part 2: The XML specification stack. Multimodal authoring with SCXML, XHTML, REX, and more by Gerald MCCOBB, IBM, 31 May 2007.
• Multimodal interaction and the mobile Web, Part 1: Multimodal auto-fill. Automatically enter personal information into forms by Gerald MCCOBB, IBM, 15 November 2005.
• Multimodal interaction and the mobile Web, Part 2: Simple searches with Find-It How to enable voice access to the Yahoo! local search engine by Gerald MCCOBB, IBM, 6 December 2005.
• Multimodal interaction and the mobile Web, Part 3: User authentication. Secure user authentication with voice and visual interaction by Gerald MCCOBB, IBM, 10 January 2006.
• Distributed Multimodal Synchronization Protocol (DMSP) by Chris Cross With Contribution from Gerald McCobb and Les Wilson, IBM Corporation, July 13, 2006.
• W3C leads a New European Project: Multimodal Web Ercim News, 2004.
• Multimodal interaction promises device integration, accessibility, and enhanced communication services Techrepublic, 2003.
• Interview to Dr. Deborah Dahl by Paolo Baggia. VoiceXML Italian User Group, July, 2003.

Publishers

• W3C's MMI Group
• Multimodal Interaction Activity
• Multimodal Interaction Activity in the W3C website