Immerse Virtual Environment for Individualized Progressive Telerehalibitation

Virtual Reality (VR) environments, which exploit the sensory-immersion of the user to facilitate development of interactive and intuitive human-computer interfaces have applications in diverse fields. In our work, we propose the development and evaluation of a virtual driving simulator with haptic feedback (providing the sense of touch, as well as kinesthetic information) to aid in rehabilitation of lost motor coordination skills in (stroke) patients. While, many past studies of virtual simulator environments for rehabilitation have examined visual and auditory cues, few have examined haptic cues. In particular, our focus in the development of a haptic feedback driving simulator is on: (a) Aiding the controlled performance of rehabilitation exercises (of a stroke patient), and (b) Simultaneous assessment of the patient's improvement/progress. The development of such systems offers many challenges for the overall system design, modeling, real-time computation, safety, reliability, etc. We wish to examine the trade-offs involved and quantitatively assessing the benefits of our proposed approach.

It is estimated that in the U.S. alone, each year, there are over 700,000 people who become victims of Upper Limb (UL) dysfunctionalities because of brain injuries (especially, stroke). With practice, however, they often improve their ability to reach, grasp, and manipulate objects. Rehabilitation over a period of time is an essential part of their therapy. The duration of the rehabilitation therapy is important, as is timeliness of the treatment. Assessment and therapy have to occur early on, or else the same therapy duration will have diminished results. Significant improvements are possible even several years after the brain injury. Unfortunately, while the number of patients affected by brain injuries have increase, the resources available for them have reduced. The problem is compounded for those people living in rural or remote locations, where regular visits by a therapist over a period of time becomes infeasible.

With the advent of low-cost, force-feedback devices, faster and powerful computers and the networking power of the internet, it is now possible to create complete home-based rehabilitation system. Commercially available force feedback devices, primarily used for gaming applications, can not only sense a person¡¯s movement, but can also apply forces during movement. Similar to existing robotic therapy devices, such devices could be used to simulate the sense of touch and movement, and could apply therapeutic patterns of forces to the hand and arm as the user attempts to move. Unlike larger robotic devices, however, force feedback devices could become truly accessible personal movement trainers because they are already in mass production and can be purchased at a low cost.

By networking them to rehabilitation centers through the Internet, such devices could provide a means for an individual with a brain injury to access a personalized program of therapeutic exercises, customized by a rehabilitation expert. Also, networking could provide a means for the rehabilitation expert to track the user's sensory motor performance while the user stays at home. Force feedback movement therapy could thus become a viable aspect of telerehabilitation services.

Our proposed study focuses on the development of a low-cost haptically-enabled virtual driving environment and a series of exercises/protocols to serve as an integrated low-cost diagnostic and therapeutic tool for both assessment of UL dysfunction and UL motor rehabilitation. The VE driving paradigm explored over here, offers a promising and cost-effective method for objective/quantitative assessment of UL performance while performing both unilateral and bimanual sensorimotor tasks in the context of one higher activities of daily living (AsDL). Our ultimate goal is to develop a system to permit ease of evaluation of various modalities of driver augmentation and driver assistive technology (visual, auditory and haptic) for various driving condition.


 Students Involved:

- Chin Pei Tang, PhD Candidate, University at Buffalo
- Leng-Feng Lee, PhD Candidate, University at Buffalo

- Anand Naik  M.S. Candidate, University at Buffalo [Graduated]

- Glenn D. White, M.S., University at Buffalo [Graduated]

- Chetan G. Jadhav, M.S., University at Buffalo [Graduated]

- Pravin K. Nair, M.S., University at Buffalo [Graduated]

 Research Issue :

Quantitative (but Inexpensive) Data Acquisition

The suitable selection of the therapeutic equipment is critical - they need to serve as interfaces to stimulate the sense of touch and movement, as well as to create customizable patterns of active/passive motion and force assists to user motions. On one hand, highly sophisticated force feedback devices with multiple degrees of freedom are available. We believe that there is a class of problems where coupling low cost COTS therapy devices with rehabilitation therapy protocols and diagnosis, opens up the possibility of widespread deployment as home-based personal-movement trainers.


Model-Based Parametric Information Transmission/Playback

In recent years, leveraging the power of internet, real-time transmission of video has come to supplement audio and data transmission for telemedicine applications in general and telerehabilitation, in particular. Quantitative assessment of patient's performance is difficult with standard videoconferencing infrastructure and thus such systems are unable to leverage the available computational infrastructure to assist the diagnosis. In our telerehabilitation system, we stream selected sets of parametric information (joint angles, steering wheel angle) across the internet instead of full motion video thereby significantly reducing bandwidth requirements. In our framework, the data acquired from the patients' arm movements are replicated by digital Jack model on the therapists' computer. The benefit of such a model-based approach is that the therapist can interact (viewed from different viewpoints, playback captured motions and proposed exercises) with the digital patient model in ways not possible simply with video conferencing approaches.


Parametric Biomechanical Identification

We also propose to use the ongoing and continuous streaming measurements to facilitate: (a) parametric estimation of the patients' biomechanical parameters (arm-lengths, ranges of joint motion, etc.); and (b) invariant and quantitative performance measures to quantify the changes in functional performance (improvement/degeneration). Data reduction of the collected sensor information will be critical for good performance in limited-bandwidth networks. We focus on data reduction techniques that will also have a ready physiological interpretation. Specifically, we adapt methods from online parameter estimation to identify adequate model parameters from the patient's musculoskeletal system and use these to develop a custom biomechanical model, representative of the virtual patient.


 Movies :

1. Model-Based Parametric Information Transmission/Playback

- The data acquired from the patients' arm movements are remotely transferred over the internet connection and replicated by digital Jack model on the therapists' computer.

- File Size: 19.9MB [Download]

- Larger Screen: view it on YouTube:


2. Steering Interface with MATLAB

- Force Feedback Toolbox implementation interfaced with a Microsoft Sidewinder Steering in MATLAB environment using DirectX functions.

- File Size: 1.27MB [Download]

- Larger Screen: view it on YouTube:


3. Virtual Driving Environment (VDE)

- A VRML-based Virtual Driving Environment (VDE) developed using full car dynamics.

- File Size: 753KB [Download]

- Larger Screen: view it on YouTube:



4. Preliminary Framework Implementation of Patient Interface

- Movie of a car model being driven in a virtual environment (VRML) using a force feedback driving wheel and pedals

- Demo #01: File Size: 2.34MB [Download] - Larger Screen: view it on YouTube:

- Demo #02: File Size: 1.89MB [Download] - Larger Screen: view it on YouTube:

- Demo #03: File Size: 1.36MB [Download] - Larger Screen: view it on YouTube:


5. Driving a Mobile Robot in a Virtual Environment

- A model of a virtual mobile robot being driven (manipulated) in a virtual environment (VRML) using a force feedback driving wheel and pedals

- File Size: 976KB [Download]

- Larger Screen: view it on YouTube:
































 Related Publications - Journal Articles:
[01] C. Jadhav, P. Nair, and V. Krovi, “Individualized Interactive Home-based Haptic Telerehabilitation,” IEEE Multimedia Systems Magazine: Haptic User Interfaces in Multimedia Systems, Vol. 13, No. 3, pp. 2-9, July 2006. [PDF]
 Related Publications - Conference Proceedings:

Jadhav, C. G., and Krovi, V., "A Low-Cost Framework for Individualized Interactive Telerehabilitation," 26th Annual International Conference in IEEE Engineering in Medicine and Biology Society (EMBS), San Francisco, California, September 1-5, 2004.


Jadhav, C., and Krovi, V., "In-Vivo Estimation of Unknown Upper-Limb Kinematic Parameters", 11th National Conference on Machines and Mechanisms, (NaCoMM-2003), Delhi, India, December 18-19, 2003.


Nair, P., Jadhav, C., and Krovi, V., "Development and Testing of a Low-Cost Diagnostic Tool for Upper Limb Dysfunction," IEEE/RSJ IROS 2003, Proceedings of  2003 IEEE/RSJ International Conference on Intelligent Robotics and Systems Las Vegas, NV, USA, October 27-31, 2003

 Related Publications - Theses:
[03] Naik, A. P., Role of Vehicle Dynamic Modeling Fidelity with Haptic Collaboration in Steer-By-Wire Systems, M.S. Thesis, Dept. of Mechanical & Aerospace Engineering, SUNY at Buffalo, Sep. 2007.  
[02] Jadhav, C. G., A Low-Cost Framework for Individualized Interactive Telerehabilitation, M.S. Thesis, Dept. of Mechanical & Aerospace Engineering, SUNY at Buffalo, Sep. 2004. [PDF] [PPT]
[01] Pravin Nair, Development of Quantitative Measures for Characterization of Upper Limb Dysfunction, M.S. Thesis, Department of Mechanical & Aerospace Engineering, SUNY at Buffalo, Feb 2004. [PDF] [PPT]

Sponsor: This project was funded by Research and Creative Activities Proposal, VP Research, University at Buffalo and NYSCEDII.

Last Updated: October 27, 2008