New App to Enable Mid-Air Commands on Smartphones, Smartwatches

As wearable devices grow and screens shrink, a team of Indian-origin researchers has developed a new sonar technology that allows you to interact with smarwatches by writing or gesturing on any nearby surface – a tabletop, a sheet of paper or even in mid-air.

Called FingerIO, it tracks fine-grained finger movements by turning a smartphone or smartwatch into an active sonar system using the device’s own microphones and speakers.

Because sound waves travel through fabric and do not require a line of sight, users can even interact with a phone inside a front pocket or a smartwatch hidden under a sweater sleeve.

“You can’t type very easily onto a smartwatch display, so we wanted to transform a desk or any area around a device into an input surface,” said lead author Rajalakshmi Nandakumar, doctoral student in computer science and engineering from University of Washington.

“I don’t need to instrument my fingers with any other sensors. I just use my finger to write something on a desk or any other surface and the device can track it with high resolution,” Nandakumar added.

FingerIO can accurately track two-dimensional finger movements to within eight-mm, which is sufficiently accurate to interact with mobile devices.

Using FingerIO, one could use the flick of a finger to turn up the volume, press a button or scroll through menus on a smartphone without touching it, or even write a search command or text in the air rather than typing on a tiny screen.

“Using sound waves to track finger motion offers several advantages over cameras and other technologies like radar that require both custom sensor hardware and greater computing power,” explained Shyam Gollakota, senior author and assistant professor of computer science and engineering.

FingerIO turns a smartwatch or smartphone into a sonar system using the device’s own speaker to emit an inaudible sound wave.

That signal bounces off the finger, and those “echoes” are recorded by the device’s microphones and used to calculate the finger’s location in space.

“Acoustic signals are great because sound waves travel much slower than the radio waves used in radar, you don’t need as much processing bandwidth so everything is simpler,” Gollakota noted.

From a cost perspective, almost every device has a speaker and microphones so you can achieve this without any special hardware.

The researchers employed a type of signal typically used in wireless communication called Orthogonal Frequency Division Multiplexing.

They demonstrated that it can be used to achieve high-resolution finger tracking using sound.

Their algorithms leverage the properties of OFDM signals to track phase changes in the echoes and correct for any errors in the finger location to achieve sub-centimeter finger tracking.

To test their approach, the researchers created a FingerIO prototype app for Android devices and downloaded it to an off-the-shelf Samsung Galaxy S4 smartphone and a smartwatch customised with two microphones which are needed to track finger motion in two dimensions.

Today’s smartwatches typically only have one which can be used to track a finger in one dimension.

The average difference between the drawings and the FingerIO tracings was 0.8 centimeters for the smartphone and 1.2 centimeters for the smartwatch.

“Given that your finger is already a centimeter thick, that’s sufficient to accurately interact with the devices,” said co-author Vikram Iyer.

The team is now working on how FingerIO can be used to track multiple fingers moving at the same time and extending its tracking abilities into three dimensions by adding additional microphones to the devices.

The paper will be presented at the Association for Computing Machinery’s CHI 2016 conference in San Jose, California, in May.