While Autonomous Vehicles (AVs) are currently being tested on carefully plotted out test areas, they cannot quite mimic the way real people feel when they drive. The challenge is to infuse cars with the intelligence they need to understand humans and take decisions that align with human preferences. Current autonomous vehicles deliver the task of driving through shared-autonomy, which is the result of a collaboration between the human driver and the vehicle. Effective shared autonomy requires a clear understanding of driver’s behavior, which is driven by multiple psychophysiological and environmental variables.  Naturalistic Driving Studies (NDS) have shown to be an effective approach to understanding the driver’s state and behavior in real-world scenarios. However, due to the lack of technological and computing capabilities, former NDS only focused on vision-based approaches, ignoring important psychophysiological factors such as cognition and emotion.



In this project, we introduce HARMONY a novel human-centered multimodal driving study framework. Through HARMONY, we collect, analyze and understand drivers states and behaviors in real-world naturalistic settings through multiple physical and virtual sensors. The main goal of this project is to utilize both human and machine advantages to humanize autonomy by instilling the beneficial nuances of human behavior, emotions, and trust with the technological and safety benefits of AVs. Behavior-guided AVs will bring human factors such as emotions, behaviors, and trust into the autonomous loop, where AVs can enhance the passenger experience, safety, and comfort. In this project, we are building models to predict driver behavior, and emotional changes in response to different environmental conditions, and to automatically infer their preferences. Through using HARMONY, we can better contextualize driving scenarios by bringing together multiple measures of driver’s emotions, cognition, and attention, together with environmental attributes. HARMONY framework consists of physical and virtual devices including cameras, smart wearables, and APIs that each provide a piece of driving context.

Harmony is the basis of multiple projects at BRAIn Lab

Driver State and Behavior Detection Through Smart Wearables

Empathetic Routing

How does environment affect driver’s state?

Reason-Aware Annotation for Contextualizing Driving Scenarios

ORIEL: an Open-Source Multi-modal Driver’s Activity Recognition Classifier in the Wild

HARMONY 1.0 Sensors and Devices

HARMONY 1.0 is the current version of our driving sensing platform. This platform collects data from 3 major sources: (1) Camera, (2) Smartwatch, (3) Music API. Our platform uses a BlackVue DR750S-2CH, which is a commercially available dual dash camera. It records both inside and outside of the cabin simultaneously. BlackVue DR750-2CH supports up to 256 GB of SD card memory, which makes it suitable for long term data collection. This camera has a built-in GPS that connects to GPS as the car turns on. This will sync the timing of the device with the global GPS timing so that the camera always has the current timing. In addition, BlackVue records the speed of the car from the output of the built-in GPS. The speed is overlayed on the recorded video. The camera records videos with 30 fps, Full HD resolution in 3-minute segments. Each 3 minutes segment of driving is saved as a joint video of inside and outside the environment.

Our platform uses an android smartwatch that is equipped with “SWEAR”, an in-house designed app for collecting long-term data from smartwatches (A). This app is designed by Professor Boukhechba, who is also one of the LinkLab members . SWEAR smooths the data collection by adding the ability of changing each sensor’s data collection frequency to the desired frequency. SWEAR records heartrate , hand acceleration, audio amplitude (noise level), light intensity, location, and gyroscope. Participants are required to start/stop the data collection for every session of driving (B). The smartwatch saves every segment of driving data on the watch and shows the participant’s ID, and number of saved files on the status tab (C). Every participant is required to sync the watch with their phone. The watches are all registered on the university wireless network. Using both the smartphone of the participant and the wifi, the time of the watch will be synced to the current time at any moment. In this way the watch always shows the current timing. Every two weeks, the participant is required to transfer their data to our system by clicking on the upload icon on the settings tab (D). Then using participant’s PID the data will be recorded in our system.

Where have our participants driven so far?

The data has been mostly collected from eastern and northeastern regions of the United States including states of Virginia, Pennsylvania, Delaware, West Virginia, Indiana, Illinois, Ohio, Vermont, New Hampshire, Maine, and New York. This data has been collected over the period of June 2019 to present. The most recent map of the collected data to date is as follows. This map is gradually updated as the data collection is an on-going process.

In the below video we summarize HARMONY’s design and goals, and we showcase an example of the application of HARMONY.

In this example, a driver is feeling blocked by a lead vehicle. This feeling is accompanied by an increase in their heart rate which is visible on the lower left part of the video.

The driver then passes the lead vehicle, and the heart rate only decreases when the vehicle is completely passed.

Additionally, parts of the HARMONY data will be released gradually which is stored on OSF under the name of HARMONY. As we analyze more sections of the data, the storage gets updated.


Tavakoli, A., Heydarian, A., & Balali, V. (2019), How Do Environmental Factors Affect Driver’s Gaze Direction and Head Movements? A Long-Term Naturalistic Driving Study, Transportation Research Board Annual Meeting 2020


Tavakoli, A., Kumar, S., Boukhechba, M., Heydarian, A., (2021). Driver State and Behavior Detection Through Smart Wearables, in IEEE Intelligent Vehicles Symposium 2021

Tavakoli, A., Boukhechba, M., Heydarian, A., (2021). Leveraging Ubiquitous Computing for Empathetic Routing, Computer-Human Interaction (CHI) 2021

Tavakoli, A., Kumar, S., Guo, X., Balali, V., Boukhechba, M., Heydarian, A., (2021). HARMONY: A Human-centered Multimodal Driving Study in the Wild, Journal of IEEE ACCESS

Tavakoli, A., Boukhechba, M., & Heydarian, A. (2020, July). Personalized Driver State Profiles: A Naturalistic Data-Driven Study. In International Conference on Applied Human Factors and Ergonomics (pp. 32-39). Springer, Cham.

Tavakoli, A., Balali, V., & Heydarian, A. (2019). “A Multimodal Approach for Monitoring Driving Behavior and Emotions” (No. 19-05204).


Team Members

Arsalan Heydarian

Assistant Professor

Principal Investigator

Arash Tavakoli

PhD Student

Graduate Research Assistant

Xiang Guo

PhD Student

Graduate Research Assistant