DataX tutorial (10) - hot plug principle of dataX plug-in

01 introduction

Through the previous blog , We are right. DataX With a certain in-depth solution ：

• 《DataX course （01）- introduction 》
• 《DataX course （02）- IDEA function DataX Complete process （ Fill all the holes ）》
• 《DataX course （03）- Source code interpretation （ Super detailed Edition ）
• 《DataX course （04）- The configuration is complete 》
• 《DataX course （05）- DataX Web Project practice 》
• 《DataX course （06）- DataX tuning 》
• 《DataX course （07）- The illustration DataX Task assignment and execution process 》
• 《DataX course （08）- Monitoring and reporting 》
• 《DataX course （09）- DataX How to limit the speed ？》

This article mainly talks about DataX The principle of plug-in loading , In understanding DataX Before , We need to know about “ Parent delegate mechanism ”.

02 Parent delegate mechanism

I have written relevant articles before , You can refer to ：《 In depth understanding of JVM Series of tutorials （11） - Class loader 》

2.1 Class loader relationship

Java The compiled class Bytecode is loaded by the classloader , And in the JVM Inside , There are several classloaders that come with the system , Take a look at the following diagram of class loaders ：

About the above class loaders , The relationship between them ：

• Start class loader （BootStrap ClassLoader）： from C++ Realization , No parent . It will be responsible for <JAVA_HOME>/lib The core class library or -Xbootclasspath Under the path specified by the parameter jar The package is loaded into memory , Note that the virtual machine is loaded according to the file name jar Bag , Such as rt.jar;
• Extend the classloader (Extension ClassLoader)： from Java Language implementation , No parent loader . It's responsible for loading <JAVA_HOME>/lib/ext Directory or by system variables -Djava.ext.dir Specifies the class library in the bit path ;
• system class loader (Application ClassLoader)： from Java Language implementation , The parent loader is Extension ClassLoader. It is responsible for loading the system classpath java -classpath or -D java.class.path Specify the class library under the path , That is what we often use classpath route , Developers can use the system classloader directly , In general, this class loading is the default class loader in the program , adopt ClassLoader#getSystemClassLoader() Method to get the class loader .
• Custom class loaders （Custom ClassLoader）： The parent loader is Application ClassLoader.

I see the concept , What is the parental delegation mechanism ？

2.2 Parent delegation mechanism process

Process description ：

• If a class loader receives a class load request , It doesn't load itself first , Instead, the request is delegated to the loader of the parent class ;
• If the parent loader still has its parent loader , Then further entrust , Recursion in turn , The request will eventually reach the top-level boot loader ;
• If the parent loader can complete the class loading task , You're back , If the parent loader cannot complete this load task , Sub loader will try to load by itself , This is the parent delegation model ;

Simply put ： Every son is lazy , Every time I have a job, I leave it to my father , Until my father said I couldn't do it , My son tries to finish it by himself .

So why do you do this ？

• The advantage is Java Class has a hierarchical relationship with priority along with its classloader , Through this level, we can avoid the repeated loading of classes , When the father has loaded the class , There is no need ClassLoader Load again .
• Secondly, considering the safety factors ,java The core api The type defined in will not be replaced at will , Let's say you pass a message over the network called java.lang.Integer Class , Passed through the parent delegate mode to the boot class loader , And the boot loader is at the core Java API The class that found the name , Found that the class has been loaded , It doesn't reload the network java.lang.Integer, And directly return the loaded Integer.class, This prevents the core API The library was tampered with at will .

Disadvantages of the parental delegation mechanism ：

• Limited by load range , The parent loader could not load into the required file , With Driver Interface, for example , because Driver Interface defined in jdk In the middle of , And its implementation is provided by the service providers of various databases , such as mysql That's it MYSQL CONNECTOR, So here's the problem ,DriverManager（ Also by jdk Provide ） To load the various implementations Driver Implementation class of interface , And then manage , however DriverManager Loaded by boot class loader , Load only JAVA_HOME Of lib The file , And its implementation is provided by the service provider , Loaded by the system class loader .

It's time to break the parental delegation , Start the classloader to delegate the subclass loader to load Driver Realization , That's the famous one SPI(SERVICE PROVIDER INTERFACE) Mechanism .

2.3 be based on SPI The mechanism destroys parental delegation

principle ： be based on “ Interface programming ＋ The strategy pattern ＋ The configuration file ” Dynamic loading mechanism of composite implementation .

No, SPI when :

• You can now classpath Riga mysql-connector-java.jar
• Then write like this Class clz = Class.forName("com.mysql.jdbc.Driver"); Driver d = (Driver) clz.newInstance(); That's no problem ;
• Reuse Application Classloader To load the mysql-connector-java.jar Of com.mysql.jdbc.Driver.

problem ： The hard coded , Be sure to load "com.mysql.jdbc.Driver", Not very elegant , Can't achieve “ Programming with interfaces , Automatic instantiation really realizes “ This coding form of .

Use SPI after :

• The code goes something like this ：Connection connection = DriverManager.getConnection("jdbc:mysql://xxxxxx/xxx", "xxxx", "xxxxx");
• DriverManager According to "jdbc:mysql" This prompt goes to the specific implementation .

ok, At this point, we will return to the theme of this article , About DataX How to load plug-ins ？

I'm sorry to say ,DataX There is no use SPI To sabotage parental delegation , But in another way （ Hot plug-in principle ： Load class =》 Get the plug-in class name and path through the configuration file =》 Instantiate the plug-in UrlClassLoader => Switch the thread context loader to UrlClassLoader And save the original thread context loader =》 Load plug-in implementation class =》 Complete the operation based on the implementation class =》 Restore the original thread context loader ）, Let's talk about that .

03 DataX Plug in hot plug

stay JobContainer have a look reader How the plug-in is loaded , Let's take a look at loading reader The code method of the plug-in ：

private Reader.Job initJobReader(
        JobPluginCollector jobPluginCollector) {
    
    this.readerPluginName = this.configuration.getString(
            CoreConstant.DATAX_JOB_CONTENT_READER_NAME);
    classLoaderSwapper.setCurrentThreadClassLoader(LoadUtil.getJarLoader(
            PluginType.READER, this.readerPluginName));

    Reader.Job jobReader = (Reader.Job) LoadUtil.loadJobPlugin(
            PluginType.READER, this.readerPluginName);

    //  Set up reader Of jobConfig
    jobReader.setPluginJobConf(this.configuration.getConfiguration(
            CoreConstant.DATAX_JOB_CONTENT_READER_PARAMETER));

    //  Set up reader Of readerConfig
    jobReader.setPeerPluginJobConf(this.configuration.getConfiguration(
            CoreConstant.DATAX_JOB_CONTENT_WRITER_PARAMETER));

    jobReader.setJobPluginCollector(jobPluginCollector);
    jobReader.init();

    classLoaderSwapper.restoreCurrentThreadClassLoader();
    return jobReader;
}

In fact, its process is very simple ,

1. Read job.json The name of the profile plug-in ;
2. Use LoadUtil according to Plug in type + The plugin name Get custom class loader JarLoader（JarLoader Inherited from jdk Inside URLClassLoader）;
3. ClassLoaderSwapper Thread class loader switch class The generated in the previous step JarLoader The classloader is set into the current context classloader ( Be careful ： The original thread context loader will be saved before saving );
4. And then use LoadUtil Add plug-ins , Then the plug-in performs some initialization operations ;
5. Finally using ClassLoaderSwapper Restore the original thread context loader .

You can see that there are two core classes , Respectively ：

• ClassLoaderSwapper（ Thread class loader management class ）
• LoadUtil（ Plug in loading tool class ）

Continue with the two classes .

3.1 ClassLoaderSwapper Thread class loader Management

First look at the source code ：

public final class ClassLoaderSwapper {
    
   private ClassLoader storeClassLoader = null;

   private ClassLoaderSwapper() {
    
   }

   public static ClassLoaderSwapper newCurrentThreadClassLoaderSwapper() {
    
       return new ClassLoaderSwapper();
   }

   /** *  Save the current classLoader, And classLoader Set to given classLoader * * @param * @return */
   public ClassLoader setCurrentThreadClassLoader(ClassLoader classLoader) {
    
       this.storeClassLoader = Thread.currentThread().getContextClassLoader();
       Thread.currentThread().setContextClassLoader(classLoader);
       return this.storeClassLoader;
   }

   /** *  Set the class loader of the current thread to the saved class load  * @return */
   public ClassLoader restoreCurrentThreadClassLoader() {
    
       ClassLoader classLoader = Thread.currentThread()
               .getContextClassLoader();
       Thread.currentThread().setContextClassLoader(this.storeClassLoader);
       return classLoader;
   }
}

According to the source code , You can see that its function is mainly aimed at The classloader of the system and Custom class loader Of , Mainly done These two kinds of loaders are loaded in the current thread “ Switch ” And “ preservation ” The operation of .

3.2 LoadUtil Plug in loader

LoadUtil Plug in loaded , According to the type of plug-ins, they are divided into ：

• reader
• writer
• transformer（ Unrealized ）

LoadUtil Plug in loaded , It is divided into... According to the type of operation ：

• Job
• Task

3.2.1 Gets the class loader

Let's first look at the method of getting the class loader ：

public static synchronized JarLoader getJarLoader(PluginType pluginType,
                                                  String pluginName) {
    
    Configuration pluginConf = getPluginConf(pluginType, pluginName);

    JarLoader jarLoader = jarLoaderCenter.get(generatePluginKey(pluginType,
            pluginName));
    if (null == jarLoader) {
    
        String pluginPath = pluginConf.getString("path");
        if (StringUtils.isBlank(pluginPath)) {
    
            throw DataXException.asDataXException(
                    FrameworkErrorCode.RUNTIME_ERROR,
                    String.format(
                            "%s plug-in unit [%s] Illegal path !",
                            pluginType, pluginName));
        }
        jarLoader = new JarLoader(new String[]{
    pluginPath});
        jarLoaderCenter.put(generatePluginKey(pluginType, pluginName),
                jarLoader);
    }

    return jarLoader;
}

getJarLoader() The main method is to directly follow the path of the plug-in new One. JarLoader, Take a closer look JarLoader Method view of ：

JarLoader It's based on jdk Inside URLClassLoader Carry out secondary implementation .

3.2.2 Add plug-ins

Put on LoadUtil The code loaded by the plug-in ：

public static AbstractJobPlugin loadJobPlugin(PluginType pluginType,
                                              String pluginName) {
    
    Class<? extends AbstractPlugin> clazz = LoadUtil.loadPluginClass(
            pluginType, pluginName, ContainerType.Job);

    try {
    
        AbstractJobPlugin jobPlugin = (AbstractJobPlugin) clazz
                .newInstance();
        jobPlugin.setPluginConf(getPluginConf(pluginType, pluginName));
        return jobPlugin;
    } catch (Exception e) {
    
        throw DataXException.asDataXException(
                FrameworkErrorCode.RUNTIME_ERROR,
                String.format("DataX find plugin[%s] Of Job To configure .",
                        pluginName), e);
    }
}

The code is very clear , It is through clazz.newInstance() Method to generate an instance （ The strategist model ）.

Come here LoadUtil The code of is basically explained .

04 At the end of the article

This article is about DataX Principle analysis of plug-in loading , If you have any questions about children's shoes, please leave a message , Thank you for reading , The end of this paper ！