We have built an intelligent retail settlement platform

 Datawhale Share  

author ： Yan Xin ,Datawhale member

At present, in the actual operation of the retail industry , There will be huge labor costs , For example, shopping guide 、 Procter & Gamble 、 Settlement, etc , And one of , In particular, it needs to spend a lot of manpower and time costs in the process of identifying goods and settling their prices , And in the process , Customers also need to wait in line . In this way, the labor cost of the retail industry is large 、 Very inefficient , Secondly, it also reduces the shopping experience of customers .

With the development of computer vision technology , And unmanned 、 The concept of automated supermarket operation , The demand of using image recognition technology and target detection technology to realize automatic product recognition and automatic settlement is imminent , Automatic closing system （Automatic checkout, ACO）. The automatic checkout system based on computer vision can effectively reduce the operating costs of the retail industry , Improve customer checkout efficiency , So as to further improve users' sense of experience and happiness in the shopping process .

AI The core of settlement is image recognition . The accuracy of image recognition determines AI Feasibility of settlement . at present ,AI Settlement faces the following pain points ：

1. Similar commodity packaging ： Similar commodities have different tastes and prices , The outer packaging of different categories of goods is similar , Both have high requirements for image recognition accuracy ;
2. There are many interference factors ： Similar commodities are prone to deformation due to angle problems during identification 、 Fold 、 Interference such as occlusion , Impact on identification results ;
3. Category update is very fast ： Retail items are usually updated at an hourly rate , If the first mock exam is added to a new product, the training model is needed to rely on a single model. , Model training cost and time cost are extremely high ;
4. The system requires high performance ： It is necessary to solve the two tasks of detection and identification at the same time , When choosing the model and optimizing, we should weigh the accuracy and speed .

Sack slide hemp ColugoMum Committed to Small and medium-sized offline retail experience stores Provide Vision based Smart retail settlement scheme , And in Github Community and Qizhi community open source synchronously .

https://github.com/ColugoMumhttps://git.openi.org.cn/ColugoMum

Based on the above pain points ,ColugoMum The team uses oars PaddleClas[1] The team put forward PP-ShiTu[2] Image recognition system . be based on PP-ShiTu The realized product identification scheme is multi category products in the retail scene 、 Small sample 、 High similarity and frequent updates provide new ideas , It can not only accurately identify multiple categories of goods , It can also meet the ultimate pursuit of prediction efficiency . The most practical function is ： When actually used online , Encounter new product categories that need to be identified , No need to retrain the model , Only the image features of this category need to be added to the retrieval library , Can realize the identification of new products ！

PP-ShiTu It is a practical lightweight general image recognition system , Mainly detected by the main body 、 There are three modules: retrieval and feature vector learning . The system selects and adjusts from the backbone network 、 The choice of loss function 、 Data to enhance 、 Learning rate transformation strategy 、 Regularization parameter selection 、 Pre training model usage and model tailoring quantification 8 In terms of , Adopt a variety of strategies , Optimize the model of each module , And pass by 10w+ Category data for training , Finally get in CPU The last forecast time is only 0.2s Multi scene general image recognition system .

Simply speaking ,PP-ShiTu There are three steps to the use of ：

1. Detect the model through the main body , Recognize the objects in the picture one by one ;
2. Feature extraction for each candidate region ;
3. The vector after feature extraction is retrieved in the retrieval library , Complete the match , Return recognition results .

Considering the extreme pursuit of accuracy and prediction speed in the actual retail scenario ,ColugoMum The team selected PicoDet Model as agent detection algorithm , Selected lightweight PPLCNet_x2_5_ssld The model is used for feature extraction , Finally, use the vector search module Faiss Medium HNSW32 As a retrieval algorithm , Achieve the ultimate balance between speed and accuracy .

Based on this ,ColugoMum The team is based on RP2K The dataset has achieved the highest **96.91%** The prediction accuracy of .

RP2K Data sets [3] Included 50 ten thousand + Pictures of retail shelves , Commodity category exceeds 2,000 Kind of , It is currently the data set with the largest number of product categories in the retail data set . Unlike data sets that generally focus on new products ,RP2K Included more than 50 10000 pictures of retail goods shelves , Commodity category exceeds 2000 Kind of , This data set is the number of product categories in the current retail data set TOP1, At the same time, all the pictures are collected manually in the real scene , For each product , Pinlan provides a very detailed annotation .

Besides ,ColugoMum Also collected and sorted out the industry SKU Level product image dataset , And look forward to working with developers , Open source can be influential in the industry 、 Data sets that meet the application requirements of actual scenarios .

https://github.com/ColugoMum/Dataset

also ,ColugoMum The team opened source based on RP2K High precision training model and prediction model of data set . Developers can fine tune the training model provided based on their own data , You can also use the provided prediction model to directly predict the experience . meanwhile ,ColugoMum Also opened based on RP2K List making activities , Developers are welcome to participate .

https://github.com/ColugoMum/Exprements

In terms of deployment, the propeller service deployment framework is used Paddle Serving[4] Deployment , Meet user batch forecast 、 High data security 、 Delay low demand , stay CPU All you need to do is 0.2 Seconds to achieve the prediction effect , Truly achieve the ultimate balance between prediction speed and accuracy .

In order to facilitate developers to better understand PP-ShiTu、 Make better use of its advantages in the field of image recognition ,ColugoMum The team open source the smart retail product recognition tutorial based on image recognition , Developers can use on this basis PP-ShiTu Fast docking business .

https://github.com/ColugoMum/Goods_Recognition

Based on the above core technology , at present ColugoMum The team uses PP-ShiTu technology , Open source cloud edge integration 、 Product identification that meets the application requirements of the actual scenario Smart_container. It can accurately locate the goods purchased by customers , And intellectualize 、 Automated price settlement .

https://github.com/ColugoMum/Smart_container

When customers place their purchased goods in the designated area ,Smart_container Be able to accurately locate and identify each commodity , And it can return the complete shopping list and the total price of the actual goods that the customer should pay . When new products are added in the system , Just update the search library , No need to retrain the model .Smart_container Cover the hardware settlement desk 、 Applet management platform 、 Big data visualization platform , Realize multi terminal unification , Smart management .

Core development team

Yan Xin , A junior majoring in automation in East China University of science and Technology , The research direction is cooperative control and decision-making of multiple robots , The main point of interest is computer vision 、 Reinforcement learning 、 Reasoning deployment . Propeller developers, technologists 、Datawhale member 、 Huawei cloud sharing expert 、 The first host of the National Undergraduate Innovation and entrepreneurship project , Won the 13th “ Challenge Cup ” Shanghai bronze medal in the college students' Entrepreneurship Plan Competition 、 The 10th East China University of science and technology “ Endeavour Cup ” Gold medal in college students' Entrepreneurship Plan Competition 、 Have two software copyrights 、 An international conference paper .

Shen Chen , A junior majoring in Information Engineering at East China University of science and Technology , Have obtained CRAIC The second prize of China robot and artificial intelligence competition in Shanghai 、 The second prize of Shanghai University Students' computer application ability Design Competition , The 10th East China University of science and technology “ Endeavour Cup ” Gold medal in college students' Entrepreneurship Plan Competition , Participate in many large-scale excellent open source projects , Have two software copyrights , I was in IEEE An international conference published a paper , Another utility model patent is pending . He used to be the deputy director of the Organization Department of the Youth League Committee of the Information Institute , Won the excellent scholarship and the advanced title of excellent students .

Du Xudong , A junior majoring in Information Engineering at East China University of science and Technology , Have a good command of C/C++/Python/Jave/Matlab/verilog And many other programming languages , The second prize of Shanghai University Students' computer application ability Design Competition , Participate in many large-scale excellent open source projects , Have two software copyrights , A utility model patent is pending . He used to be the deputy director of the community management department of the school of information , Won the excellent scholarship of the University .

ad locum , We have participated in ColugoMum Thank you for your research and development ： Huang Xiaoyue 、 Wang Xin 、 Zhao Yian 、 Zhou Tianyi 、 Shenjiachuan et al , for ColugoMum Helpful Datawhale organization 、 The paddle community and Qizhi community Thank you ! We welcome more developers to participate in the retail product identification data set 、 Product identification and Smart Container Open source co construction activities , To promote together AI Open source and open ecosystem construction , Promote China's physical retail to digitalization 、 Intelligent direction Transformation Development .

future ,ColugoMum The team will continue to break product and technology boundaries , Relying on the open source community , Open source produces more and better 、 Open source projects that can truly empower physical retail , We should really promote the real retail in China to be intelligent 、 Digital transformation , Realization ColugoMum“ Authors efficiency 、 Enabling retail ” The mission of .

reference ：

1.https://github.com/PaddlePaddle/PaddleClas  

2.S. Wei et al., "PP-ShiTu: A Practical Lightweight Image Recognition System," arXiv preprint arXiv:2111.00775, 2021. 

3.J. Peng, C. Xiao, and Y. Li, "RP2K: A large-scale retail product dataset for fine-grained image classification," arXiv preprint arXiv:2006.12634, 2020. 

4.https://github.com/PaddlePaddle/Serving

Open source sharing , give the thumbs-up Three even ↓