Gesture-Controlled Robotic Arm Manipulator for Teleoperation

Senior Capstone | January - August 2025

Project Overview

As part of a two-person senior design team, I contributed to the development of a gesture-controlled robotic arm for real-time human-to-robot teleoperation. The system combines computer vision, a wearable 9-axis IMU, ultrasonic sensing, and embedded controllers to translate human arm and hand movements into coordinated robotic motion.

The 5-DOF robotic arm was designed in SolidWorks and fabricated using 3D printing, while the control system integrated Python, OpenCV, MediaPipe, Arduino microcontrollers, and multiple motor and sensor interfaces. Computer vision provided horizontal and vertical positional information, while the wearable IMU supplied wrist orientation. Ultrasonic sensing supplemented the vision system with depth information.

During testing, the vision based gesture recognition module achieved over 95% classification and spatial positioning accuracy in real time for discrete hand gestures. The complete system also achieved end-to-end latency below 600 ms.

Publication: This project was published through the American Society for Engineering Education (ASEE). [View the Full Paper Here]

Objectives

• Develop an intuitive gesture-based interface for real-time robotic teleoperation.

• Combine computer vision and wearable IMU data to track human motion.

• Design and fabricate a 5-DOF robotic arm capable of reproducing human movements.

• Develop motion-mapping and communication algorithms to translate human movement into coordinated robotic motion.

• Evaluate gesture recognition, responsiveness, and manipulation performance.

Constraints and Challenges 

The human arm has seven degrees of freedom, while most robotic manipulators use six. To reduce redundant actuation and simplify both the physical design and control program, I decided to use five DOF for the arm. The wrist yaw actuator was omitted in favor of a stronger shoulder yaw mechanism, which also reduced mass near the end effector and marginally simplified the control strategy.

The vision system also had a limitation in depth perception. The webcam provided reliable horizontal and vertical position information from the image, but it did not provide direct depth measurements. An HC-SR04 ultrasonic sensor was then consequently incorporated to provide distance information and supplement the camera-based tracking.

Another challenge was synchronizing the sensing systems, which operated at different update rates. Python threading allowed the inputs to be processed concurrently, while queues managed the incoming data. Before transmission to the Arduino Uno, outdated measurements were cleared so that the arm received the most recent data.

Design Methodology 

The system was developed by integrating embedded hardware, mechanical design, multimodal sensing, and motion-control algorithms into a unified robotic platform.

System Architecture

The project was divided into four primary subsystems: Wearable Sensing, Signal Processing and Sensor Fusion, Robotic Arm Control, and Power Distribution.

The Arduino Nano 33 BLE formed the wearable module and provided 9-axis IMU data through its onboard sensor. A TTP223 capacitive touch sensor was also incorporated to control the opening and closing of the end effector. The wearable module transmitted its data wirelessly using Bluetooth Low Energy (BLE).

On the processing side, a laptop running Python 3.12 handled the computer vision and data routing. OpenCV and MediaPipe were used to identify arm and hand landmarks, while the ultrasonic sensor provided depth information. An Arduino Nano 33 IoT acted as an intermediary device between the BLE wearable module, ultrasonic sensor, and laptop.

The processed information was then sent through a wired serial connection to an Arduino Uno, which controlled the robotic arm's motors.

Motion Mapping and Data Processing

The incoming measurements were mapped to the five joints of the manipulator according to the type of human movement being measured:

• Joint 1 - Horizontal arm movement from the computer vision.

• Joint 2 - Vertical arm movement from the computer vision.

• Joint 3 - Depth arm movement from the computer vision and ultrasonic sensor.

• Joint 4 - Wrist roll from the IMU.

• Joint 5 - Wrist pitch from the IMU.

• End-Effector - Touch input from the wearable module.

The IMU measurements were processed using a Madgwick filter to estimate the wearable's roll, pitch, and yaw while reducing the effects of sensor noise and motion-related disturbances. This provided a more stable orientation estimate for controlling the wrist joints than relying directly on the raw accelerometer and gyroscope measurements.

Because the different sensing processes operated at different update rates, Python threading allowed the vision and routing processes to run concurrently. Queues were used to manage incoming measurements, with outdated values cleared before new commands were sent to the Arduino Uno.

Mechanical Design and Fabrication

The robotic arm was designed in SolidWorks and fabricated using 3D-printed PETG components. The mechanical design included the individual joints, linkages, belt-driven mechanisms, and mounting points for the actuators. 

The three lower joints used NEMA 17 stepper motors, while a smaller NEMA 11 stepper motor was used for wrist rotation. Because the NEMA 17 motors were relatively heavy, they were positioned closer to the lower portion of the arm rather than directly at the joints farther from the base. Belt-driven mechanisms transferred the motor rotation to the corresponding joints, reducing the amount of motor weight carried farther along the arm.

PETG was selected for the printed components because it provided the strength, durability, and heat resistance needed for the mechanical structure while remaining practical for rapid prototyping.

Hardware and Electronics Integration

The electronic hardware was selected based on the torque, size, compatibility, and power requirements of the actuators. NEMA 17 stepper motors were used for the three lower joints to provide the required torque needed to support the further arm linkages, while a smaller NEMA 11 was used for wrist rotation. MG995 servos were selected for the wrist and end effector because of their lightweight design as well as their fast and accurate response.

DRV8825 drivers were selected for their Arduino compatibility, ability to supply the required motor current, and compact design. Each driver was powered by an external 24 V DC supply.

The Arduino Uno, Nano 33 BLE, and Nano 33 IoT were integrated through serial and Bluetooth communication, while the HC-SR04 provided distance measurements alongside the wearable sensor data. The end effector was also programmed to stop when resistance was detected, helping prevent excessive mechanical strain.

My Contributions

My primary contributions focused on the arm control, mechanical design, and embedded system integration. This included:

• Control & Motion Mapping: Developed the motion-mapping algorithm that converted incoming human-motion data into coordinated joint-space commands for the robotic arm, enabling the arm to reproduce the operator's movements.

• Mechanical Design: Designed the 5-DOF robotic arm in SolidWorks and fabricated the mechanical structure through 3D printing, including the joint and belt-driven mechanisms used to transfer motor motion.

• System Integration: Developed the system-level architecture connecting the sensing, processing, and actuation components, including the communication interfaces and power-distribution pathways between the wearable sensors, microcontrollers, motor drivers, and actuators.

• Embedded Hardware: Programmed and interfaced the Arduino Uno, Nano 33 BLE, and Nano 33 IoT with DRV8825 drivers, NEMA 17/11 stepper motors, MG995 servos, and ultrasonic sensors. Component selection and integration were also accomplished while balancing performance, electrical requirements, and the project's $200 budget.

Results and Evaluation 

The vision-based gesture recognition system achieved over 95% classification and spatial positioning accuracy in real time. The complete system achieved an end-to-end response latency of less than 600 ms, allowing user movements to be translated into responsive arm motion.

The wearable IMU provided orientation data for wrist control, contributing to smooth and continuous motion of the corresponding joints. Combined with the vision-based measurements, the multiple sensing inputs allowed the manipulator to respond to different types of user movement and maintain control during operation.

The robotic arm was evaluated through pick-and-place tasks and trajectory-following exercises. These tests demonstrated that the integrated sensing, control, and actuation approach could translate user movements into coordinated robotic motion.

Conclusions and Future Work 

The completed system demonstrated the integration of computer vision, wearable sensing, ultrasonic measurements, embedded control, and a custom robotic arm into a gesture-based teleoperation platform.

Future improvements include adding a serial connection as an alternative to Bluetooth to reduce potential interference, using a second calibrated camera to improve depth perception, and incorporating adjustable belt tensioners to compensate for belt stretching over time.