Manipulator Control with Computer Vision

Description

Dependencies

For running the code, ROS is required to be set up. This project uses ROS Noetic on Ubuntu 20.02. This page contains necessary installation: https://emanual.robotis.com/docs/en/platform/openmanipulator_x/quick_start_guide/#setup.

If ROS is installed and catkin_ws was setup, following git repositories are required to be placed in the src directory, in order to simulate OpenManipulator-X.

git clone -b noetic-devel https://github.com/ROBOTIS-GIT/open_manipulator.git
git clone -b noetic-devel https://github.com/ROBOTIS-GIT/open_manipulator_msgs.git
git clone -b noetic-devel https://github.com/ROBOTIS-GIT/open_manipulator_simulations.git
git clone https://github.com/ROBOTIS-GIT/open_manipulator_dependencies.git

To run MediaPipe's models, the model has to be downloaded from their website.

Getting started

In order to run the project run the following commands:

cd catkin_ws
catkin_make
source devel/setup.bash
roscore

Roslaunch

The following roslaunch files can be run:

1. Arm Keypoint Capture + Control

roslaunch open_manipulator_cv_controller keypoint_collection_control.launch

2. Convert and Visualize Keypoints

roslaunch open_manipulator_cv_controller training_data.launch

3. Training and Evaluation

roslaunch open_manipulator_cv_controller train_evaluate.launch

Global variables for each can be viewed with --ros-args.