Heart-Beat Rate From Face Video Estimator

Table of Contents

Introduction

This estimator enables you to calculate the user’s heart rate from a short video of their face taken from the smartphone’s front camera. The estimator implements an algorithm described in  and yielded interesting results. Figure 1 compares the performance of this algorithm implemented in Android against the heart rate as provided by an FDA approved blood pressure monitor – Omron.

Figure 1: Performance of Android Heart Rate Estimator with respect to the Omron device
Figure 1: Performance of Android Heart Rate Estimator with respect to the Omron device

This short tutorial will guide you through the steps you need to take to incorporate this heart rate estimator. First, we need to check the prerequisites to make sure our smartphone is capable of performing the necessary computations.

Prerequisites

In order to estimate the user’s heartbeat from a face-video estimator, the device must support a front-facing camera, flash, and flash mode. Before starting the estimator, we check if these prerequisites are satisfied by calling the function reqSatisfied() . The function will throw an ExtractorPrerequisiteUnsatisfiedException instead of returning void  if at least one of these prerequisites aren’t met. Otherwise, the function will return null and we can start the estimator. We can accomplish this with a try-catch statement:

try {
    heartBeatFaceExt.reqSatisfed();
    heartBeatFaceExt.start();
} catch (ExtractorPrerequisiteUnsatisfiedException e) {
    // The prerequisites of the extractor were not satisfied
}

// Keep collecting until...
heartBeatFaceExt.terminate();

Making a Fragment

The camera used by the estimator takes the form of a Fragment  in an Android Activity. For setting it up in your app, make a Fragment  for the camera in a RelativeLayout  or LinearLayout  in your Activity as shown here.

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:paddingBottom="@dimen/activity_vertical_margin"
    android:paddingLeft="@dimen/activity_horizontal_margin"
    android:paddingRight="@dimen/activity_horizontal_margin"
    android:paddingTop="@dimen/activity_vertical_margin"
    tools:context="com.yourProject.FaceHRActivity">

    <FrameLayout
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_alignParentTop="true"
        android:layout_centerHorizontal="true"
        android:id="@+id/activity_face_hr_camera"
        android:layout_marginBottom="60dp"></FrameLayout>


</RelativeLayout>

Initializing the Estimator

The Activity must extend the NoSleepFragmentActivity and the estimator and it’s associated encapsulator can be initialized as follows:

private static BackableEncapsulator<TimeZonedTimestampedObject<HeartRateContainer>> faceHREncapsulator = null;
private HRFaceVideoEstimator mHeartRateCalculator;

Within the OnCreate function you need to tie the estimator to the camera fragment. To do this, add:

this.mHeartRateCalculator = new HRFaceVideoEstimator(faceHREncapsulator); 

FragmentManager manager = getSupportFragmentManager(); 
FragmentTransaction transaction = manager.beginTransaction(); 
Fragment cameraFragment = this.mHeartRateCalculator.getCameraFragment(); 
transaction.add(R.id.activity_face_hr_camera, cameraFragment, "faceHRCameraFragment"); 
transaction.commit();

Don’t forget to add the required permissions in the Android Manifest and ensure that your screen orientation is locked on either portrait or landscape mode for the duration of the activity.

Add a listener to the encapsulator if required.

Starting the Estimator

The last step to start the estimator. This can be done by overriding the OnResume  function. In order to complete the activity life cycle, we stop the estimator by overriding the OnPause  function.

@Override
protected void onPause() {
    this.mHeartRateCalculator.terminate();
    super.onPause();
}

@Override
protected void onResume() {
    super.onResume();
    this.mHeartRateCalculator.start();
}

Your heart-rate estimator is now ready to run!

Face Video HR Estimation
Figure 2: An example screenshot of the estimator in action.

Running Example

(TODO: complete)

References