Gesture digital enlightenment learning machine

adoptive son

Today, I saw a neighbor's child learning to use gestures to represent numbers

He doesn't know if it's right , It was there that I compared , It looks very confused （ Children, do you have many question marks ~~）.

I saw him , Therefore, an idea came into being ： Make an initiator that can recognize gestures .

But think of just a gesture recognition , That's too monotonous . On the basis of gesture recognition , Added a program for calculation and verification

---

design idea

• The system gives the problem of addition and subtraction within ten （ Considering that gestures are commonly used 1-10 Between the expression of ）
• After the user calculates this problem , Show it by hand , The system will recognize and display the number indicated by the gesture
• The system will evaluate whether the number represented by the gesture is the answer to the question
• If it is the answer to the question , Will prompt ：“YOU ARE RIGHT”
• If it is not the answer to the question , Will prompt ：“YOU ARE ERROR”

---

Technical analysis

What technology is used in gesture recognition ？

• For gesture recognition , What I use here is the classification algorithm in deep learning , Previously, it was considered that the detection algorithm can be used for identification ,
• But there is no need to use detection , On the one hand, the size of pictures processed by our own machine is limited , On the other hand, it is necessary to leave only one display hand in the interface segment OK 了
• Because the camera is fixed , The background is white , So there is no need to detect the hand part of the human body
• To this end, the final decision was made ： Gesture recognition using classifier
• For classifiers , I used BackBone yes ResNet18,Loss The choice is CrossEntropy Loss To deal with
• The picture is （224, 224, 3） Size data , Every gesture picture （1-10） They correspond to each other 300-400 Zhang
• The data preprocessing part is also very simple , Subtract the mean from the picture and divide by the variance OK 了
• epoch It's settled 50 Rounds ,batchsize Set the 32, The gesture accuracy of the final test is 97% about

What is used to display the interface , And the layout of the interface ?

• At first, I thought of building a web, use flask Deployment , Then I thought it was unnecessary , use OpenCV Things that can be done （ How to kill a chicken ）
• To this end, the whole interface began to adopt OpenCV Design the project interface
• First, the prompt of whether the question is answered correctly is displayed in the upper left corner of the interface , The title of the question is shown below
• A window is designed on the right side of the interface to display the information collected by the camera , That is, the place to let go
• The final recognized answer is displayed after the formula equal sign on the left , Mark with other colors , Just OK La
• The effect of the design is as follows , For a young man whose artistic cells are negative , Don't ask for beauty ：
  

Whether the current hardware supports the development of the project ?

• Now that the model implementation is complete , The interface is also designed , It is necessary to consider whether the current hardware can support
• At present, my hardware here has , A home camera , A notebook （ core i7 series 5 generation , A built-in display GTX950）
• Do not run do not know , I'm scared . The processing speed can be called turtle speed （1 Minutes can handle 1 frame ）
• But to see a good graphics card on the Internet is thousands of . I finally bowed to money ...
• Bought a piece of Jeston Nano Development board , At least it's cheap ~~~
• Last run , The processing speed can reach... Per second 20 The frame looks like , Poverty limits my imagination ...

---

Effect display ：https://www.bilibili.com/video/BV1C44y1j76W#reply97610586848

Project address ：https://github.com/martinzhang710/SignLanguage.git

---

Later period and prospect

• I can't help roast about the interface , The subsequent beautification is certain , If necessary, use flask It's good to deploy
• The accuracy of the model still needs to be further improved
• But it's enough to play with my neighbor's children ~~