Multimodal speech interfaces to GIS

Multimodal speech interfaces to GIS

Ken Sam's project invloves leveraging existing commercial off the shelf (COTS) web-GIS component and open specification Speech Application Language Tags (SALT) as building blocks for creating a multimodal web-GIS application. In this paper, we will address how the different technology components were applied for creating a multimodal interfaces for the navigation, interaction and feedback for the web-based GIS application.

Speech driven GIS interface

In most computing and information technology environment, data is presented in either text or graphic format as a means of conveying information to the end users. This has been the traditional paradigm of data display and visualization in the computing world. Efforts have been made in the software industry to design better navigation interfaces for software products and improve on the overall user-friendliness of the products. With geospatial data, additional dimensions are introduced in the presentation and display of the data. Because of the added complexity of geospatial data, there are a number of researches that are still on-going in trying to improve on the interface, visualization and interpretation of geospatial data. One can normally expect geospatial data to be viewed or interpreted by a normal-vision user without much challenge. Yet, visualization and navigation of map is a huge challenge for people who are visually impaired. The design and usability of GIS applications has traditionally been tailored to keyboard and mouse interaction in an office environment. To help with the visualization of geospatial data and navigation of a GIS application, this project presents the result of a prototype application that incorporates voice as another mode of interacting with a web-GIS application. While voice is not a replacement for the mouse and keyboard interface, it can act as an enhancement or augmentation to improve the accessibility and usability of an application. The multimodal approach of combining voice with other user interface for navigation and data presentation is beneficial to the interpretation and visualization of geospatial data and make GIS easier to use for all users.

Publications
Jacobson, R.D., and Sam, K. (2006) Multimodal Web-GIS: AugmentingMap Navigation and Spatial Data Visualization with Voice Control, AutoCarto 2006, June 26-28, Electronic Proceedings.

Multimodal zooming in digital geographic information

As a basic research issue, how well can people integrate and reconcile spatial information from various modalities, and how useful is such integration?

As an applied issue, what is the potential for haptic and auditory navigation within geographic information systems? Can visual information be augmented by the presentation of information via other modalities, namely, haptics and audition, and if so, to what extent?

The research will investigate a particular form of navigation within geographic information systems, namely, zooming. The research aims to investigate non-visual methods of representing or augmenting a visual zoom through the auditory and haptic senses, creating a multimodal zooming mechanism.

Multimodal Web-GIS: Augmenting Map Navigation and Spatial Data Visualization with Voice Control

Jacobson, R.D., and Sam, K. (2006) Multimodal Web-GIS: Augmenting Map Navigation and Spatial Data Visualization with Voice Control, AutoCarto 2006, June 26-28, Electronic Proceedings.

Abstract

This paper describes the design and architecture of a prototype project that was implemented to augment the navigation and visualization of geospatial data for a web-GIS application.  This project leverages existing commercial off the shelf (COTS) web-GIS component and open specification Speech Application Language Tags (SALT)  as building blocks for creating a multimodal web-GIS application.  In this paper, we will address how the different technology components were applied for creating a multimodal interfaces for the navigation, interaction and feedback for the web-based GIS application.  The design, integration process and the architecture of the prototype application are covered as a part of this project report.

[VIEW PDF]

Representing Spatial Information Through Multimodal Interfaces: Overview and preliminary results in non-visual interfaces

Jacobson, R.D. (2002) Representing Spatial Information Through Multimodal Interfaces: Overview and preliminary results in non-visual interfaces.  6^th International Conference on Information Visualization: Symposium on Spatial/Geographic Data Visualization, IEEE Proceedings, London, 10-12 July, 2002, 730-734.

Abstract

The research discussed here is a component of a larger study to explore the accessibility and usability of spatial data presented through multiple sensory modalities including haptic, auditory, and visual interfaces.  Geographical Information Systems (GIS) and other computer-based tools for spatial display predominantly use vision to communicate information to the user, as sight is the spatial sense par excellence. Ongoing research is exploring the fundamental concepts and techniques necessary to navigate through multimodal interfaces, which are user, task, domain, and interface specific. This highlights the necessity for both a conceptual / theoretical schema, and the need for extensive usability studies.  Preliminary results presented here exploring feature recognition, and shape tracing in non-visual environments indicate multimodal interfaces have a great deal of potential for facilitating access to spatial data for blind and visually impaired persons. The research is undertaken with the wider goals of increasing information accessibility and promoting “universal access”.  

[VIEW PDF]

Multimodal virtual reality for presenting geographic information

Jacobson, R.D., Kitchin, R.M., and Golledge R.G. (2002) Multimodal virtual reality for presenting geographic information.  In: Fisher, P. and Unwin, D. (eds.) Virtual Reality in Geography. Taylor and Francis: London, pp. 382-400.

Abstract

Since the conception of virtual reality (VR) environments, interaction has been predominantly visual and haptic in nature.  Only recently have developers and scientists explored non-visual and multimodal VR environments.  In this paper we examine these recent developments and assess their viability as geographic tools for people with severe visual impairments.  Our own research and  the work of others suggests that multimodal VR, where visual interaction is either augmented by, or substituted for, other forms of data such as sound and touch, offers people with severe visual impairments access to geographic information that is in many cases otherwise inaccessible.  Such offerings open up opportunities to explore the spatial relations of geographic representations and real world environments, and could qualitatively improve their quality of life.

[VIEW PDF]

Exploratory user study of haptic and auditory display for multimodal information systems

Jeong, W. and Jacobson, R.D. (2002) Exploratory user study of haptic and auditory display for multimodal information systems. In: McLaughlin, M. L., Hespanha, J.P., and Sukhatme, G.S. (eds.) Touch in virtual Environments: Haptics and the design of interactive systems. IMSC Series in Multimedia, Prentice Hall: New York, pp. 194-204.

 Abstract

Since the inception of virtual reality (VR) environments, interaction has been predominantly visual, especially in conveying spatial information. However, in many situations vision is not enough or is not available. For example, for the visually impaired over-reliance on visual display denies them access to the information. Even for the general population, if there is no light or weak light, a visual display is not optimal for conveying information. Recently a number of researchers have tried to add other modalities, such as sound or haptics, to overcome the imitations of visual display.

[VIEW PDF]