Joern's code uses -devign

Recently, I have been doing experiments on graph neural network , But open source code is really rare , Speaking of me and joern It's been a long time since my love and hate and sorrow of separation

One . First try

Recently, a preliminary study of graph neural networks has begun , Combined with the previous vulnerability detection related directions , Combined with the trend, we decide to use graph neural network application vulnerability detection problem , By reading papers at home and abroad , Summarize the basic steps to solve this problem , And the different measures taken in the steps in different papers .

To make a long story short , There are three steps

• Code composition
• Vectorization of graphs
• Neural network design
  Comprehensive consideration , Combining innovative and achievable industries , Specify the following strategies for later implementation
• Code composition —joern
• Vectorization of graphs —word2vec+ onehot
• Neural network design
  From then on, I joern Bond

Two . joern

 about joern Installation , I refer to the following articles , Hereby thank 
[https://blog.csdn.net/jlu_wangqi/article/details/109412399](https://blog.csdn.net/jlu_wangqi/article/details/109412399)
[https://blog.csdn.net/water_likly/article/details/88415127](https://blog.csdn.net/water_likly/article/details/88415127)
[https://blog.csdn.net/u013648063/article/details/106811275](https://blog.csdn.net/u013648063/article/details/106811275)
[https://blog.csdn.net/water_likly/article/details/89378623?spm=1001.2101.3001.6650.4&utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7ERate-4.pc_relevant_default&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7ERate-4.pc_relevant_default&utm_relevant_index=](https://blog.csdn.net/water_likly/article/details/89378623?spm=1001.2101.3001.6650.4&utm_medium=distribute.pc_relevant.none-task-blog-2~default~CTRLIST~Rate-4.pc_relevant_default&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2~default~CTRLIST~Rate-4.pc_relevant_default&utm_relevant_index=5)
[https://blog.csdn.net/m0_37924639/article/details/78783592](https://blog.csdn.net/m0_37924639/article/details/78783592)
[https://blog.csdn.net/God_Jn/article/details/103238013](https://blog.csdn.net/God_Jn/article/details/103238013)
** Mainly through the first and second articles , Finally, choose the method in Chapter 3 **

• Finally, complete configuration is successful , But in the end, because Neo4j The version problem of caused the failure , The failure process can be explained in a separate document , It takes a total of two days , In the end, it's fruitless , But in addition to Neo4j I feel there is absolutely no problem with my configuration , If there is a suitable old version Neo4j Welcome to send me a private message .
• It is not recommended to try the old version joern install , A new version of joern-cli, Go straight into joern Official website Just check the official documents and install , For the specific installation process, see article

3、 ... and . joern-cli

About joern-cli I directly refer to the installation of joern Official installation documentation , I can't see if I have hands , But three problems limit me

• problem one ： The problem of network speed , There is no problem with the three sentences , But when it comes to the last item, start downloading files , My reading speed tells me that it will take several days , At that moment, I collapsed , It's really speechless ！！！！
  
  answer one ： It was late at night , It's really hard to sleep , I'm ready to trust my computer , I've been driving it to download , I think I'm so handsome. Once the Internet speed soars at night , I decided to sleep , Sure enough, I didn't download it the next morning , I decided to ask someone else to help me with the following, but what should I do , I opened it. joern-install.sh, Take out the web address inside ,https://github.com/ShiftLeftSecurity/joern/releases/latest/download/joern-cli.zip, Find my good friend github Up and down , At first glance joern-cli The package , I'm so happy , I feel hopeful , But it was later found that the official website gave all joern-cli Compressed packages of all versions and all files website
  

• problemr two ： I started using it inspired by the following article joern-cli, I thought to myself, don't you just unzip it , Then I can run it directly according to the method he said . However, it is not so smooth , As soon as I run under the unzipped folder /joern, Just report the mistake directly java There's a problem , A lot of main classes and lang What's wrong , I forgot to take a screenshot
  
  https://blog.csdn.net/u013648063/article/details/112311392?spm=1001.2014.3001.5502
  https://blog.csdn.net/qq_36281420/article/details/114526209
  https://blog.csdn.net/u013648063/article/details/106970131
  answer two ： stay ./joern Before to add sudo. In the end, I think it is jdk The problem of version changed to 11.04linux Relatively safe , Yes orcle Register and download on the official website

• problemr three ： Finally, under the guidance of the following article , Change to your own code as follows , Painstakingly Bala exported several pictures
  https://blog.csdn.net/u013648063/article/details/112311392?spm=1001.2014.3001.5502

sudo ./joern-parse / home/code/
sudo ./joern-export cpg.bin --repr pdg --out /home/joern-cli/pdg
dot -Tpng -o test1.png ./pdg/1-pdg.dot

AST

CPG

PDG

CFG

answer three ： It was new year's Eve when the map was exported , The new year has solved the problem of work , Hey, hey, hey

Four . Use problems in code

 Because I have seen joern The power of , Therefore, the reappearance of this paper has been delayed until the beginning of school , In the past two days, I have encountered various problems when reproducing the article 

（ One ） Make plans

1. Make sure to reproduce the article ： Source code vulnerability detection based on graph convolution network
2. Find the right open source article ：Devign Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks
3. Choose code ： The first link is a big guy's own recurrence , The code is easy to read , But the processing of edges is limited to AST. The second link is the code of the official author of the paper, which is only about the training process , The second link is about data processing , Set up a map , And the vectorization process code of the graph . However, due to the large amount of the second code , And the data is relatively large, and it is difficult to realize , At the same time, we don't want to reproduce the modified article, but mainly want to borrow its process of drawing , So choose open source code one
   https://github.com/epicosy/devign
   https://github.com/saikat107/Devign
   https://github.com/VulDetProject/ReVeal
   The code structure of code 1 is as follows
   

（ Two ） Run code

1. Configuration environment ： The environmental requirements are as follows , about joern Come on , Directly put the previous good joern-cli Compressed package put in joern Under folder , Use at the same time anaconda Build a virtual environment GNN, And then to Python The library can be used pip ininstall everything .
   
   note：
   **tip1：** except gensim The version of the library should be 3.8.1 outside , No requirements for other library versions ,pip Just go , Using a higher version of gensim It can also be that it is troublesome to modify the parameter names in some code ,
   **tip2：** stay windows I have encountered installation when configuring the environment torch_sparse and torch_geometic Error reporting , See article

2. Adjust the code ：
   problem one See article
   problem two because joern The version problem is devign-master\src\prepare\cpg_generator You need to modify the output folder parameters , And because of java And joern The version adaptation problem needs to be added sudo, Therefore, it is revised joern_parse Function is

def joern_parse(joern_path, input_path, output_path, file_name):
    out_file = file_name + ".bin"
    joern_parse_call = subprocess.run([sudo,"./" + joern_path + "joern-parse", input_path, "--out", output_path + out_file],
                                      stdout=subprocess.PIPE, text=True, check=True)
    print(str(joern_parse_call))
    return out_file

At this time, I didn't notice the danger , A meal of operation also makes the code run , I didn't expect to bury a great hidden danger

Four . Problem solving

（ One ） Problem generation

Get up early in the morning and hug the sun , Full of positive energy , I want to reproduce that article through my own ideas , I want to start with the extraction of nodes and edges after generating the graph , But I found a problem . That is when execution python main.py -c Although I successfully called joern, But there are some errors , At the same time, prompt the file not process.

I found that is graph-for-funcs.sc An error is reported when a file is generated

（ Two ） Finding solutions

• Search on major websites , But the findings are rarely attached
• Later, I thought that maybe because this open source code has not been reproduced by many people , Not as good as github Look for this project to see if any partners have questions , Directly in github Search on graph-for-funcs.sc choice issue Special view , Check and find the following solutions again and again

（ 3、 ... and ） Finding solutions

• java edition
  First of all, according to the joern stay github Requirements on official documents on jdk11, At the same time, according to the error report, I found that it was a library import error , At the same time, not every import There are problems , Considering that java Version upgrade problem , Therefore, the jdk8 Rose to jdk11, But the final upgrade will not help .
• graph-for-funcs.sc edition
  
  Through the author's tips, I obviously feel graph-for-funcs.sc Version and joern-cli The adaptation of version is very critical , Then look at what the author said issue3, In the midst of hidden problems , We see that the author has updated , So I try to use the new version and the old version , At the same time https://github.com/epicosy/devign/issues/9
  Someone also raised the issue of version , A total of three experiments , No fruit , It is worth saying that in the third version graph-for-funcs.sc Many people in the document proposed to solve the problem , All versions of files are retained
  

4. joern-cli edition
   By means of github Search for keywords on graph-for-funcs.sc Check out some answers ,https://github.com/epicosy/devign/issues/9
   Finally, the question is answered in the above page , After reading these discussions, I get the following conclusions

• java Version is not the root of the problem

• graph-for-funcs.sc and joern-cli The adaptation problem of is the root of the problem , At the same time, the two have strong adaptability

• joern-cli Update very fast , It may be updated every day

• The graph-for-funcs.sc Under the code ,joern-cli 1.0.170 More applicable , But now we use the official document method to download the latest version by default .
  

  Finally I go joern-cli Download the versions of joern-cli 1.0.170, Also check out devign Files in open source code graph-for-funcs.sc Update time

joern-cli 1.0.170

graph-for-funcs.sc Update time

Find out graph-for-funcs.sc The update time is joern-cli 1.0.170 Two days after the release , The author should be based on joern-cli 1.0.170 It's done graph-for-funcs.sc, So the download joern-cli 1.0.170 Decompress the compressed package and put it into the code joern Run under folder , solve the problem . And in the follow-up, I inadvertently found a more interesting thing. See article

If there are problems in your practice or you want to communicate with bloggers , Pay attention to Blogger official account.