Control Your Mouse with Hand Gestures and Camera

I came across OpenCV modules that allow controlling your computer mouse pointer with your finger in space using open-source computer vision software and a webcam.

I slightly modified the source code to make it runnable. The python source code is attached at the bottom.

This can be useful if you want to attach a monitor to a wall without a mouse, and a person can interact with it through a camera.

Satisfying the Prerequisites

1) Python version 3.8.10
This is the latest version of Python that is compatible with Autopy module. Other older versions may also work, but I haven’t tested them. I did confirm that Python 3.9 will not work. Download link.

2) Rust
Download link.

3) OpenCV
I have tested it with opencv-python 4.5.5.62. More info.
Run in the terminal:

pip3.8 install opencv-python

4) Hand Tracking Module
I have tested it with HandTrackingModule 0.1. More info.
Run in the terminal:

pip3.8 install HandTrackingModule

5) Setuptools plugin for Rust extensions
I have tested it with setuptools-rust 1.1.2. More info.
Run in the terminal:

pip3.8 install setuptools-rust

6) Autopy
I have tested it with autopy 4.0.0. More info.
Run in the terminal:

rustup toolchain add nightly-2019-10-05 --profile minimal
rustup default nightly-2019-10-05
pip3.8 install autopy

If you find that any other python module is missing in the error message, install it.

If PIP Module Installation Fails

Try running the pip installation in the command prompt window with admin privileges (“Run as Administrator” for Windows, or with sudo for Linux and MacOS). Or,

Try running the command with user privileges:

pip3.8 install modulenameexample --user

Running the Program

1. Download the source files:
   AIVirtualMouseProject.py
   HandTrackingModule.py

2. Put these two files into a single folder.

3. Run the AIVirtualMouseProject.py through Python3.8 or the environment that you configured.

If you have multiple cameras, or you have a problem with your camera source, try changing the channel from 1 to 0. For example:

cap = cv2.VideoCapture(1) # for external USB camera, usually additional camera
or
cap = cv2.VideoCapture(0) # for primary camera, usually built-in laptop camera

1. To navigate your mouse, you stick only your second finger out and move your hand around.

2. To perform left-click, you stick the middle finger out in addition to your second finger and touch them together.

3. If you want to program different gestures or more functionality, you can modify the source code.

4. To exit the program, you can either press CTRL+C on Linux or MacOS if the program was started from the terminal, or you can terminate the program from the Task Manager.

#opencv
#python
#ai
#computervision
#opensource
#programming
#python3
#engineer
#engineering
#software
#developer 
#development