In a recent client work, we had the wonderful opportunity to provide hardware prototyping development services, combining wearable sensors and drones to support a Swiss innovation project. As we enter the Fourth Industrial Revolution, our lives are being exposed to more and more computing devices and robotic systems. However, the question of how we can interact naturally with robotic systems is still evolving. We were tasked with developing the hardware and software to enable Natural User Interface (NUI) control of a robotic drone. This exciting project gave us the opportunity to integrate NUI with a Parrot drone in order to control a variety of robotic devices based on the Parrot SDK.
Smart Strap Wearable Sensor
At the heart of the drone control system was a wearable sensor based on a novel conductive composite developed at Empa in the Laboratory for High Performance Ceramics. The sensor fiber was integrated into a watchband design for integration with a smart watch platform as part of an innovation project with STBL Medical Research AG, currently being marketed under (www.carunda24.com). The sensor combines nano-textured carbon particles with a thermoplastic elastomer, resulting in a highly elastic piezoresistive fiber sensor. It has been investigated for various applications, including vital sign measurement (Piezoresistive Soft Condensed Matter Sensor for Body-Mounted Vital Function Applications).
Motion Capture Approach
Human Activity Recognition (HAR) is focused on extracting human activity from wearable sensors. This most often includes activity recognition based on Inertial Measurement Unit (IMU) sensors including accelerometers and gyroscopes. However, these sensors are not able to accurately capture the finger and hand motions such as opening and closing of the fist. The smart strap with integrated sensor allows the opening and closing motion of the hand to be easily determined by monitoring the surface strain of the skin in the region of the tendons. A gesture recognition approach was developed that would extract gestures from the smart strap sensor data and then map those gestures to drone flight commands.
Hardware / Software Development
The smart strap required analog digital conversion (ADC) to generate raw data and the choice of micro controller platform. We went with an Arduino platform based on the Adafruit Feather HUZZAH board with integrated ESP8266 chip. This allowed the integration of different ADC chips and passive components to optimize sensor sensitivity and signal strength. As part of a lean prototyping solution, speed is often essential in order to build / test and integrate quickly. In this project we utilized JavaScript and Node.js, which allowed easy integration with the Parrot SDK and associated Node libraries. This made it straight forward to integrate drone control with the prototyping hardware and motion control approach.
