A comparative study of backdoor attack and counter sample attack

Catalog

One 、Introduction

1. What is a backdoor attack ？

2. What is fighting against sample attacks ？

3. The difference between backdoor attack and sample attack

4. Supplementary data poisoning attack

Two 、 Mind mapping

3、 ... and 、 Application scenarios

1. Application scenarios against sample attacks

2. Application scenario of backdoor attack

3. Research on typical neural network backdoor attacks

Four 、 Summarize the experience

5、 ... and 、reference

---

One 、Introduction

1. What is a backdoor attack ？

Explanation of some papers ：

（1） Neural network backdoor attack [1]： For any neural network model M、 Its training process P And attack algorithm A, Model M By algorithm A Get the model after the attack M' Or training process P By algorithm A After training, get the model M', When M' Meet the conditions ① And conditions ② when , Call the algorithm A Backdoor attack algorithm for neural network .

① For normal input , Model M' And model M Perform the same .

② For trigger input , Model M' An error will occur , Produce the output desired by the attacker .

Neural network backdoor attack implanting backdoor is essentially achieved by modifying the model , The implementation of neural network backdoor attack will cause the model to change in the direction expected by the attacker . The model changes in order to leave “ Trapdoor ”, Make the modified model sensitive to trigger input , Make any trigger input produce an attacker's design error in the reasoning process of the model

（2） Back door attack [2] The purpose of is to make the model learn the content specified by the attacker , It still has a good test effect on normal samples , But for poisoned samples, the attacker's preset label will be output .

Backdoor attacks also have the following characteristics ：

complexity ： The model involves more steps in the training phase , Including data collection 、 Data preprocessing 、 model building 、 model training 、 Model preservation 、 Model deployment and so on

Concealment ： Backdoor attacks are not abnormal for normal samples , Exception occurs only when the sample has a backdoor trigger , Therefore, it is difficult for users to detect when using , Besides , Poisoning samples injected by backdoor attacks are usually very few , Only 0.5% about .

Practicality ： The backdoor attack guarantee model still has good effect on the normal test set , Therefore, there is a high probability that the model after backdoor attack will be deployed and put into use .

（3） I understand it ： The attacker is in the training phase of the model , Modify the training data or directly modify some parameters of the model , Make the model complete the training , Have correct output for normal test data set , and For poisoning samples, output the content specified by the attacker .（ Output results for some model inputs , In my hands , I can output whatever I want ）

2. What is fighting against sample attacks ？

        Antagonistic input attack [3-4] Also known as escape （evasion） attack , It is an antagonistic input through construction , So that the neural network model has the wrong attack mode . The attack modifies the input of the model imperceptibly , Make the normal neural network model behave abnormally . The existing antagonistic input attacks can be divided into non target attacks and targeted attacks . The former simply wants the model to go wrong , The latter wants the model to infer the modified input, that is, antagonistic input, as a specific goal .（ Although your model is very powerful , But facing my confrontation sample , Your model is also right and wrong ）

                                                               chart 1 Concept map of confrontation samples

3. The difference between backdoor attack and sample attack

（1） On the stage of attack ： Fighting against sample attacks is in the reasoning stage of the model , That is to attack the trained model . Backdoor attack is to attack in the stage of model training .

（2） In terms of the impact on the model ： Against the sample attack is the deception model , Let the model make wrong judgments on some confrontation samples ; Backdoor attack is to induce models , Let the model output some poisoned data as the desired result of the attacker .

4. Supplementary data poisoning attack

        Data poisoning attack [5-8] It mainly exists in the training stage of the model , By modifying the training data, that is, poisoning , Affect and destroy the model , Make the generalization performance of the model worse , That is, the effect on the test set becomes worse , Models cannot be effectively learned , It can't even converge .（ I don't do anything else , I just don't want your model training to succeed ）

surface 1 Comparison of different attack algorithms  

Two 、 Mind mapping

chart 2  Mind mapping

3、 ... and 、 Application scenarios

1. Application scenarios against sample attacks

（1） Face forgery ：Bose wait forsomeone [9] Attack the face recognition algorithm , By adding indistinguishable trace interference to the original image , The face cannot be located by the detection algorithm . chart 4(a) For the original image ﹐ The detection algorithm can accurately locate , chart 4(b) To counter the sample , Has successfully bypassed the face detection algorithm .

                                                                  chart 3 Against face detection  

        Song wait forsomeone [10] Further, the face recognition system is attacked against samples , And make it recognize the specified error category . As shown in the table 1 Shown , The similar counter samples generated from the original samples are recognized as the specified error category .

  surface 2  Against face recognition

 （2） Text attack ： The work of text confrontation is also increasing , And the cost of text confrontation is lower , Just add, delete and change a few words , It may lead to the error of the recognition result of the model .Gao wait forsomeone [11] Black box attack on text classification model , Put forward DeepWordBug Algorithm . adopt DeepWordBug Disturb the text , You can make :
       1) be based on Word-ISTM The recognition rate of the model decreases 68%

       2) be based on Char-CNN The recognition rate of the model decreases 48%.

        The specific attack methods are as follows :
       DeepWordBug The algorithm will first select the English words that are most likely to change the classification of the model , Then disturb the English word , So that the text classification model recognition error . For example, figure 4 Through will Place Change to Plcae,heart Change to herat after , The recognition model will recognize the positive evaluation as negative .

              chart 4 Text attack

（3） Figure against attack ：Zigner wait forsomeone [12] This paper proposes a method to combat attacks against graph depth learning model , It is the first research on anti attack on attribute graph . Their research shows that :
      1） By manipulating the graph structure and node characteristics , It can significantly reduce the accuracy of node classification ;
      2)  By maintaining important data characteristics , For example, degree distribution 、 Feature co-occurrence , It can ensure that the disturbance is not detected ;
      3)  The attack method is portable .

chart 5 Figure attack  

（4） Voice against attack : Wei [13] On the basis of introducing the relevant concepts of countermeasure samples , Select text content recognition in speech recognition 、 Voiceprint identification Two typical tasks , From white box attack to black box attack 、 From digital attack to physical attack 、 The sequence from specific carrier to general carrier , Take from easy to difficult 、 Step by step paste Close to the actual scene , This paper systematically combs the typical attack methods of voice countermeasure samples in recent years .

2. Application scenario of backdoor attack

（1） Outsourcing training ： Because neural networks, especially convolutional neural networks, need large Amount of training data and millions of weights , Training them also requires a lot of computing power . For most people and businesses , It is more reasonable to choose to outsource the training of the model to the service provider choice . In this case , The attacker has complete control over the training process of the model , The back door can be constructed without restriction . One of the earliest BadNets[14] The simple attack is aimed at this Jing proposed , But the most obvious problem in this scenario is that the attacker's ability is too large , The reality of attack is low

（2） The migration study ： Directly control or indirectly affect the retraining process of the pre training model , Use data poisoning and other means to implant the back door , Such as Trojaning Attack[15] and Clean-Label Attack[16] Two attack methods . stay Trojaning Attack in , The attacker has complete control over the retraining process , At the same time, you can access the pre training model , You can choose training samples freely to retrain the model .Clean-Label Attack The attacker cannot directly control the retraining process , But it can be uploaded “ clean ” Poisoning samples indirectly affect the retraining of the model . On the other hand , Attackers can construct poisoning pre training model by themselves , Make the victim retrain the poisoning model , Thus, the back door is implanted into the target model generated by the victim's self retraining .

（3） Strengthen deep learning ：Yang Wait in the literature [17] For deep reinforcement Back door attack of learning , Use the order dependence between inputs , The triggering of the back door can be completed in a very short time , Only one trigger can affect the performance of the model for a long time . and

（4） Federal learning ： Federated learning constructs a framework that enables thousands of participants to jointly train models while keeping their own data confidential . In training , The central server distributes a random subset of the training set to each participant , Each participant will train locally , Then submit the updated model to the server , The server updates the Federation model according to the average of the models submitted by the participants . meanwhile , To protect the confidentiality of participants' local models , Federated learning adopts a secure aggregation approach . for example , The literature [18] Taking advantage of the characteristics that participants can locally train and update the model in the federated learning scenario , The attacker trains the poison model submission server locally . At the same time, because the poisoning model is confidential under the security aggregation mechanism , This makes it impossible to detect the abnormality of the poisoning model , Make the poisoning model difficult to be found

3. Research on typical neural network backdoor attacks

                                                  surface 3  Research on typical neural network backdoor attacks

Four 、 Summarize the experience

          Today's paper reading is mainly conceptual , It's an attack on the back door 、 General understanding of data poisoning attacks and countering sample attacks , They make the model lose its original normal function . Understand three common AI model attack methods ： Against sample attacks , Basic concepts of data poisoning attack and backdoor attack , And the way they attack , And know the differences and connections between them . The second is to study the application scenarios against sample attacks and backdoor attacks . Because these three attacks involve more knowledge points , So I haven't seen much about technology and algorithm , Later, we should focus on , Look at the references mentioned in the article , Then expand this article . After that , Also need to consider , Whether these model attack techniques can be used for data privacy protection of artificial intelligence , And use rigorous experiments to demonstrate their idea reliable .

            In addition to the above , Today, I also learned how to write and read my experience in a more standardized way , Thank you for your guidance , Developing good writing habits will also be of great help to writing papers in the future .

5、 ... and 、reference

[1]Qingyin TAN, Yingming ZENG, Ye HAN, Yijing LIU, Zheli LIU. Survey on backdoor attacks targeted on neural network[J]. Chinese Journal of Network and Information Security, 2021, 7(3): 46-58.

[2]Qingyin TAN, Yingming ZENG, Ye HAN, Yijing LIU, Zheli LIU. Survey on backdoor attacks targeted on neural network[J]. Chinese Journal of Network and Information Security, 2021, 7(3): 46-58.

[3]C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus. Intriguing properties of neural networks. ArXiv: 1312.6199, 2013

[4] SZEGEDY C, ZAREMBA W, SUTSKEVER I, et al. Intriguing properties of neural networks[J]. arXiv preprint arXiv:1312.6199, 2013.

[5] Barreno M, Nelson B, Sears R, et al. Can Machine Learning Be Secure? [C]. The 2006 ACM Symposium on Information, computer and communications security, 2006: 16-25.

[6] Biggio B, Nelson B, Laskov P. Support Vector Machines under Adversarial Label Noise[C]. Asian conference on machine learning, 2011: 97-112.

[7] M. Kloft, P. Laskov. Online anomaly detection under adversarial impact[C]. The Thirteenth International Conference on Artificial Intelligence and Statistics, 2010: 405-412.

[8] A. Shafahi, R. Huang, M. Najibi, et al. Poison frogs! Targeted clean-label poisoning attacks on neural networks[C]. Advances in Neural Information Processing Systems, 2018: 6103-6113

[9]Bose A,Aarabi P. Adversarial attacks on face detectors using neural net based constrained optimization[C]//Proc of the 20th Int Workshop on Multimedia Signal Processing(MMSP). Piscataway,NJ:IEEE,2018

[10]Song Qing,Wu Yingqi,Yang Lu. Attacks on state-of-theart face recognition using attentional adversarial attack generative network[J]. arXiv preprint arXiv:181112026,2018

[11]Gao Ji,Lanchantin J ,Soffa M,et al.Black-box generation of adversarial text sequences to evade deep learning classifiers[C]//Proc of Security and Privacy Workshops(SPW).Piscataway,NJ:IEEE,2018:50-56

[12]Zugner D,Akbarnejad A,Günnemann S. Adversarialattacks on neural networks for graph data[C]//Proc of the24th Int Conf on Knowledge Discovery &. Data Mining(KDD). New York:ACM,2018

[13]WEI Chunyu,SUN Meng,ZOU Xia,ZHANG Xiongwei.Reviews on the Attack and Defense Methods of Voice Adversarial Examples[J].Journal of Cyber Security,2022,7(1):100-113 

[14] GUT, DOLAN-GAVITT B, GARG S. BadNets: ldentifying vulne-rabilities in the machine learning model supply chain[J]. arXiv pre-print arXiv:1708.06733,2017.

[15]LIU Y, MA S,AAFER Y, et al. Trojaning attack on neural net-works[R]. 2017.

[16]SHAFAHI A, HUANG W R,NAJIBI M, et al. Poison Frogs! tar-geted clean-label poisoning attacks on neural networks[C]l/lAd-vances in Neural Information Processing Systems. 2018: 6103-6113.

[17]YANG Z,IYER N, REIMANN J, et al. Design of intentional back-doors in sequential models[J]. arXiv preprint arXiv: 1902.09972,2019.

[18]BAGDASARYAN E,VEIT A,HUA Y, et al. How to backdoorfederated learning[J]. arXiv preprint arXiv: 1807.00459,2018.