Showing posts with label human computer interaction. Show all posts
Showing posts with label human computer interaction. Show all posts

Friday, 31 May 2013

Multimodal speech interfaces to GIS

Multimodal speech interfaces to GIS

Ken Sam's project invloves leveraging existing commercial off the shelf (COTS) web-GIS component and open specification Speech Application Language Tags (SALT) as building blocks for creating a multimodal web-GIS application. In this paper, we will address how the different technology components were applied for creating a multimodal interfaces for the navigation, interaction and feedback for the web-based GIS application.

Screen caputure of Voice-enabled multimodal WebGIS application interface
Speech driven GIS interface
In most computing and information technology environment, data is presented in either text or graphic format as a means of conveying information to the end users. This has been the traditional paradigm of data display and visualization in the computing world. Efforts have been made in the software industry to design better navigation interfaces for software products and improve on the overall user-friendliness of the products. With geospatial data, additional dimensions are introduced in the presentation and display of the data. Because of the added complexity of geospatial data, there are a number of researches that are still on-going in trying to improve on the interface, visualization and interpretation of geospatial data. One can normally expect geospatial data to be viewed or interpreted by a normal-vision user without much challenge. Yet, visualization and navigation of map is a huge challenge for people who are visually impaired. The design and usability of GIS applications has traditionally been tailored to keyboard and mouse interaction in an office environment. To help with the visualization of geospatial data and navigation of a GIS application, this project presents the result of a prototype application that incorporates voice as another mode of interacting with a web-GIS application. While voice is not a replacement for the mouse and keyboard interface, it can act as an enhancement or augmentation to improve the accessibility and usability of an application. The multimodal approach of combining voice with other user interface for navigation and data presentation is beneficial to the interpretation and visualization of geospatial data and make GIS easier to use for all users.

Publications
Jacobson, R.D., and Sam, K. (2006) Multimodal Web-GIS: AugmentingMap Navigation and Spatial Data Visualization with Voice Control, AutoCarto 2006, June 26-28, Electronic Proceedings.

Multimodal zooming in digital geographic information

As a basic research issue, how well can people integrate and reconcile spatial information from various modalities, and how useful is such integration?

As an applied issue, what is the potential for haptic and auditory navigation within geographic information systems? Can visual information be augmented by the presentation of information via other modalities, namely, haptics and audition, and if so, to what extent?

The research will investigate a particular form of navigation within geographic information systems, namely, zooming. The research aims to investigate non-visual methods of representing or augmenting a visual zoom through the auditory and haptic senses, creating a multimodal zooming mechanism.

Transcending the Digital Divide

The purpose of this research is to develop, evaluate, and disseminate a non-visual interface for accessing digital information. The aim is to investigate the perceptual and cognitive problems that blind people face when trying to interpret information provided in a multimodal manner. The project also plans to provide touch sensitive and sound based network interface and navigation devices that incorporate cognitive wayfinding heuristics. Haptic (force feedback) interfaces will be provided for exploring web pages that consist of map, graphic, iconic or image products. Sound identifiers for on-screen windowed, map, and image information will also be provided. These tasks will contribute to transcending the Digital Divide that increasingly separates blind or vision impaired people from the growing information-based workplace. Recent research at UCSB has begun to explore how individuals identify features presented through sound and touch. Other research (e.g. O'Modhrrain and Gillespie, 1998; McKinley and Scott, 1998) have used haptics to explore screen objects such as windows, pulldown menus, buttons, and sliders; but map, graphic and other cartographic representations have not been explored. In particular, the potential of auditory maps of on-screen phenomena (e.g. as would be important in GIS applications) has barely been examined and few examples exist of combining audio and touch principles to build an interface. While imaginative efforts to build non-visual interfaces have been proceeding. there is a yet little empirical evidence that people without sight can use them effectively (i.e. develop a true representation of the experienced phenomena). Experiments will be undertaken to test the ability of vision impaired and sighted people from different age groups to use these new interface or features such as: (i) the haptic mouse or a touch window tied to auditory communication displays; (ii) digitized real sounds to indicate environmental features at their mapped locations; (iii) "sound painting" of maps, images, or charts to indicate gradients of phenomena like temperature, precipitation, pressure, population density and altitude. Tests will be developed to evaluate (i) the minimum resolvable area for the haptic interpretation of scenes; (ii) the development of skills for shape tracing in the sound or the force-feedback haptic domain, (iii) the possibility of using continuous or discreet sound symbols associated with touch sensitive pads to learn hierarchically nested screen information (e.g. locations of cities within regions within states within nations); (iv) to evaluate how dynamic activities such as scrolling, zooming, and searching can be conducted in the haptic or auditory domain, (v) to evaluate people's comprehension and ability to explore, comprehend, and make inferences about various non-visual interpretations of complex visual displays (e.g. maps and diagrams), and (vi) to explore the effectiveness of using a haptic mouse with a 2" square motion domain to search a 14" screen (i.e. scale effects).