当前位置：网站首页>Ambari (V) ---ambari integrated Azkaban (valid for personal test)

Ambari (V) ---ambari integrated Azkaban (valid for personal test)

2022-06-28 07:45:00 【New objects first】

Ambari Custom service integration –Azkaban

Reference material

Ambari Official documentation of integration components :Stacks and Services

Ambari Introduction to the principle of custom service integration

Ambari— Integrate Azkaban

Document directory

One . preparation

Two . Ambari Integrate Azkaban Principle and configuration

3、 ... and . Start the installation according to the interface prompts Azkaban

Four . Ambari Integrate Azkaban Problems encountered in the process

One . preparation

1.1 install Ambari

Ambari For source code compilation and installation, please refer to my previous blog :

Ambari2.6.0 Source code compilation and installation and problem solving ( Super detailed documentation )

Ambari For offline installation, please refer to my previous blog :

Ambari Problems encountered during installation and solutions

1.2 Source compilation configuration Azkaban

1.2.1 Azkaban Source code compilation

Download the decompression package :

#  Download the specified version of the compressed package 
wget https://github.com/azkaban/azkaban/archive/3.90.0.tar.gz

#  decompression 
tar -zxvf 3.90.0.tar.gz

Source code compilation :

 #  Enter into azkaban The root directory after decompression , The following directory uses $AZKABAN_HOME replace , Use gradlew How to compile 
./gradlew build -x test

As shown in the figure below , Compile successfully :

Insert picture description here

$AZKABAN_HOME/azkaban-?/build/distributions/ The corresponding compressed package will be generated under the directory

Source configuration

  1. Create directory 
mkdir -p var/www/html/azkaban

  1. decompression azkaban-db/build/distributions/azkaban-db-0.1.0-SNAPSHOT.tar.gz Get documents create-all-sql-0.1.0-SNAPSHOT.sql

  2. Put the following two files into var/www/html/azkaban Under the table of contents

    a. $AZKABAN_HOME/azkaban-web-server/build/distributions In the catalog azkaban-web-server-0.1.0-SNAPSHOT.tar.gz file

    b. $AZKABAN_HOME/azkaban-exec-server/build/distributions In the catalog azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz file

The above documents can be passed through web ui Mode of access , Here's the picture

Insert picture description here

1.2.2 Azkaban Database configuration (mysql)

establish azkaban database 

 create database azkaban character set utf8;

establish azkaban user 

create user 'azkaban'@'%' identified by 'yourpassword';

Grant authority 

grant all privileges on azkaban.* to azkaban;
flush privileges;

Import azkaban The watch used 

#  Use the created azkaban The user login mysql
mysql -uazkaban -pyourpassword;

use azkaban;

#  Import azkaban The watch used , The command needs to be in the sql Execute under the path where the script is located 
source create-all-sql-0.1.0-SNAPSHOT.sql

show tables;

Insert picture description here

Two . Ambari Integrate Azkaban Principle and configuration

2.1 Ambari Custom service integration principles

For the unaccepted ambari Interface management services , such as Elasticsearch、Kylin、 Even a jar package , All available Technologies related to custom service integration take service Integrated into the ambari In the interface .

such , You can go through ambari Realize to Custom service Of install 、 To configure 、 start-up 、 Monitor start state 、 stop it 、 Indicator monitoring 、 The alarm 、 Quick links Wait for a lot of operation , Very convenient .

Custom service integration , It's through stack To carry out ,stack The version directory of is in /ambari-server/src/main/resources/stacks Next

When we initialize ambari-server after ,stack Can also be defined in /var/lib/ambari-server/resources/stacks In the directory

stacks The table of contents is as follows , When customizing service integration , Configure the following directory tree :

|_ stacks
   |_ <stack_name>
      |_ <stack_version>
         metainfo.xml
         |_ hooks
         |_ repos
            repoinfo.xml
         |_ services
            |_ <service_name>
               metainfo.xml
               metrics.json
               |_ configuration
                  {configuration files}
               |_ package
                  {files, scripts, templates}

2.2 Ambari Integrate Azkaban To configure

First, in the
/var/lib/ambari-server/resources/stacks/HDP/2.6/services Create a new directory AZKABAN

2.2.1 metainfo.xml File configuration

Entry directory Azkaban, establish metainfo.xml file , The contents of the document are as follows :

<?xml version="1.0"?>
<!-- # -*- coding: utf-8 -*- # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -->
<metainfo>
    <schemaVersion>2.0</schemaVersion>
    <services>
        <service>
            <name>AZKABAN</name>
            <displayName>Azkaban</displayName>
            <comment>Azkaban It's for running Hadoop job Batch workflow job scheduler ,Azkaban Focus on sorting through job dependencies , And provide easy to use web User interface to maintain and track workflow .
            </comment>
            <version>3.90.0</version>
            <quickLinksConfigurations>
                <quickLinksConfiguration>
                    <fileName>quicklinks.json</fileName>
                    <default>true</default>
                </quickLinksConfiguration>
            </quickLinksConfigurations>
            <components>
                <component>
                    <name>AZKABAN_WEB</name>
                    <displayName>Azkaban Web Server</displayName>
                    <category>MASTER</category>
                    <cardinality>1</cardinality>
                    <versionAdvertised>true</versionAdvertised>
                    <recovery_enabled>true</recovery_enabled>
                    <commandScript>
                        <script>scripts/azkaban_web.py</script>
                        <scriptType>PYTHON</scriptType> <timeout>300</timeout> </commandScript> </component> <component> <name>AZKABAN_EXECUTOR</name> <displayName>Azkaban Executor Server</displayName> <category>SLAVE</category> <cardinality>0+</cardinality> <versionAdvertised>true</versionAdvertised> <recovery_enabled>true</recovery_enabled> <commandScript> <script>scripts/azkaban_executor.py</script>
                        <scriptType>PYTHON</scriptType>
                    </commandScript>
                </component>

            </components>

            <configuration-dependencies>
                <config-type>azkaban-common</config-type>
                <config-type>azkaban-executor.properties</config-type>
                <config-type>azkaban-users</config-type>
                <config-type>azkaban-web.properties</config-type>
                <config-type>global.properties</config-type>
                <config-type>log4j.properties</config-type>
            </configuration-dependencies>

        </service>
    </services>
</metainfo>

2.2.2 configuration Directory configuration

2.2.2.1 azkaban-users.xml To configure 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- /** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ -->
<configuration>
    <property>
        <name>content</name>
        <display-name>azkaban-users.xml template</display-name>
        <description>Custom azkaban-users.xml</description>
        <value>
            <![CDATA[ <azkaban-users> <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/> <user password="metrics" roles="metrics" username="metrics"/> <role name="admin" permissions="ADMIN"/> <role name="metrics" permissions="METRICS"/> </azkaban-users> ]]>
        </value>
        <value-attributes>
            <type>content</type>
            <show-property-name>false</show-property-name>
        </value-attributes>
        <on-ambari-upgrade add="true"/>
    </property>

</configuration>
2.2.2.2 global.properties.xml To configure 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- /** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ -->
<configuration>

    <property>
        <name>content</name>
        <display-name>azkaban-users.xml template</display-name>
        <description>Custom azkaban-users.xml</description>
        <value>
            <![CDATA[ retries=10 retry.backoff=10000 failure.emails= ]]>
        </value>
        <value-attributes>
            <type>content</type>
            <show-property-name>false</show-property-name>
        </value-attributes>
        <on-ambari-upgrade add="true"/>
    </property>

</configuration>
2.2.2.3 azkaban-executor.properties.xml File configuration 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- /** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ -->
<configuration>
    <property>
        <name>executor.port</name>
        <value>12321</value>
    </property>
    <property>
        <name>azkaban.jobtype.plugin.dir</name>
        <value>plugins/jobtypes</value>
    </property>
    <property>
        <name>executor.maxThreads</name>
        <value>100</value>
    </property>
    <property>
        <name>executor.flow.threads</name>
        <value>100</value>
    </property>
    <property require-input="true">
        <name>azkaban.webserver.url</name>
        <value>http://localhost:10100</value>
        <description>
            example: http://localhost:10100
        </description>
    </property>
</configuration>
2.2.2.4 azkaban-web.properties.xml To configure 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- /** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ -->
<configuration>

    <property>
        <name>mail.sender</name>
        <value>[email protected]</value>
    </property>
    <property>
        <name>mail.host</name>
        <value>smtp.163.com</value>
    </property>
    <property>
        <name>mail.user</name>
        <value>[email protected]</value>
    </property>
    <property>
        <name>mail.password</name>
        <value>xxx</value>
    </property>


    <property>
        <name>azkaban.use.multiple.executors</name>
        <value>true</value>
    </property>
    <property>
        <name>azkaban.executorselector.filters</name>
        <value>StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus</value>
    </property>
    <property>
        <name>azkaban.executorselector.comparator.NumberOfAssignedFlowComparator</name>
        <value>1</value>
    </property>
    <property>
        <name>azkaban.executorselector.comparator.Memory</name>
        <value>1</value>
    </property>
    <property>
        <name>azkaban.executorselector.comparator.LastDispatched</name>
        <value>1</value>
    </property>
    <property>
        <name>azkaban.executorselector.comparator.CpuUsage</name>
        <value>1</value>
    </property>

</configuration>
2.2.2.5 log4j.properties.xml To configure 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- /** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ -->
<configuration>
    <property>
        <name>content</name>
        <display-name>log4j.properties template</display-name>
        <description>Custom log4j.properties</description>
        <value>
            log4j.rootLogger=INFO, Console
            log4j.logger.azkaban=INFO, server
            log4j.appender.server=org.apache.log4j.RollingFileAppender
            log4j.appender.server.layout=org.apache.log4j.PatternLayout
            log4j.appender.server.File=logs/azkaban.log
            log4j.appender.server.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %5p [%c{1}] [%t] [Azkaban] %m%n
            log4j.appender.server.MaxFileSize=1024MB
            log4j.appender.server.MaxBackupIndex=2
            log4j.appender.Console=org.apache.log4j.ConsoleAppender
            log4j.appender.Console.layout=org.apache.log4j.PatternLayout
            log4j.appender.Console.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %5p [%c{1}] [%t] [Azkaban]
            %m%n
        </value>
        <value-attributes>
            <type>content</type>
            <show-property-name>false</show-property-name>
        </value-attributes>
        <on-ambari-upgrade add="true"/>
    </property>

</configuration>
2.2.2.6 azkaban-common.xml To configure 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
     /**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
-->
<configuration>
    <property>
        <name>azkaban.name</name>
        <value>Prod</value>
    </property>
    <property>
        <name>azkaban.label</name>
        <value>Prod Azkaban</value>
    </property>
    <property>
        <name>azkaban.color</name>
        <value>#FF3601</value>
    </property>
    <property>
        <name>azkaban.default.servlet.path</name>
        <value>/index</value>
    </property>
    <property>
        <name>web.resource.dir</name>
        <value>web/</value>
    </property>
    <property>
        <name>default.timezone.id</name>
        <value>Asia/Shanghai</value>
    </property>

    <property>
        <name>user.manager.class</name>
        <value>azkaban.user.XmlUserManager</value>
    </property>
    <property>
        <name>user.manager.xml.file</name>
        <value>conf/azkaban-users.xml</value>
    </property>

    <property>
        <name>executor.global.properties</name>
        <value>conf/global.properties</value>
    </property>
    <property>
        <name>azkaban.project.dir</name>
        <value>projects</value>
    </property>

    <property>
        <name>velocity.dev.mode</name>
        <value>false</value>
    </property>

    <property>
        <name>jetty.use.ssl</name>
        <value>false</value>
    </property>
    <property>
        <name>jetty.maxThreads</name>
        <value>100</value>
    </property>
    <property>
        <name>jetty.port</name>
        <value>10200</value>
    </property>

    <property>
        <name>lockdown.create.projects</name>
        <value>false</value>
    </property>
    <property>
        <name>cache.directory</name>
        <value>cache</value>
    </property>

    <property>
        <name>jetty.connector.stats</name>
        <value>true</value>
    </property>
    <property>
        <name>executor.connector.stats</name>
        <value>true</value>
    </property>

    <property>
        <name>database.type</name>
        <value>mysql</value>
    </property>
    <property require-input="true">
        <name>mysql.host</name>
        <value></value>
    </property>
    <property require-input="true">
        <name>mysql.port</name>
        <value>3306</value>
    </property>
    <property require-input="true">
        <name>mysql.database</name>
        <value>azkaban</value>
    </property>
    <property require-input="true">
        <name>mysql.user</name>
        <value>azkaban</value>
    </property>
    <property require-input="true">
        <name>mysql.password</name>
        <value></value>
    </property>
    <property>
        <name>mysql.numconnections</name>
        <value>200</value>
    </property>

</configuration>

2.2.3 package/scripts To configure

2.2.3.1 azkaban_executor.py To configure 
# -*- coding: utf-8 -*-
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import json
import os.path as path
import time

from common import azkabanHome, azkabanExecTarUrl, azkabanExecTarName, azkabanConfPath
from resource_management.core.exceptions import ExecutionFailed, ComponentIsNotRunning
from resource_management.core.resources.system import Execute
from resource_management.libraries.script.script import Script


class ExecutorServer(Script):
    def install(self, env):
        Execute('yum install -y python-requests')

        tmpAzkabanExecTarPath = '/tmp/' + azkabanExecTarName
        Execute('mkdir -p {0}'.format(azkabanHome))
        Execute('wget --no-check-certificate {0} -O {1}'.format(azkabanExecTarUrl, tmpAzkabanExecTarPath))
        Execute('tar -xf {0} -C {1} --strip-components=1'.format(tmpAzkabanExecTarPath, azkabanHome))

        Execute(
            'cd {0} && '
            'chmod +x bin/start-exec.sh && '
            'chmod +x bin/shutdown-exec.sh && '
            'chmod +s bin/internal/internal-start-executor.sh'.format(azkabanHome)
        )

        self.configure(env)

    def stop(self, env):
        Execute('cd {0} && ./bin/shutdown-exec.sh'.format(azkabanHome))

    def start(self, env):
        self.configure(env)
        Execute('cd {0} && ./bin/start-exec.sh'.format(azkabanHome))
        from params import azkaban_executor_properties
        executor_port = int(azkaban_executor_properties['executor.port'])
        url = 'http://127.0.0.1:{0}/executor?action=ping'.format(executor_port)
        maxRetryCount = 10
        retryCount = 0
        import requests
        while True:
            try:
                resp = requests.get(url)
                print(resp.text)
                if json.loads(resp.text)['status'] == 'alive':
                    print('executor is alive')
                    Execute('curl -G "localhost:{0}/executor?action=activate" && echo'.format(executor_port))
                    print('after activate')
                    break
            except:
                print('executor is not alive')
            time.sleep(1)
            retryCount += 1
            if retryCount > maxRetryCount:
                raise Exception('web start failed')

    def status(self, env):
        # TODO  Can be optimized , and start Logic is consistent 
        try:
            Execute(
                'export AZ_CNT=`ps -ef |grep -v grep |grep azkaban-exec-server | wc -l` && `if [ $AZ_CNT -ne 0 ];then exit 0;else exit 3;fi `'
            )
        except ExecutionFailed as ef:
            if ef.code == 3:
                raise ComponentIsNotRunning("ComponentIsNotRunning")
            else:
                raise ef

    def configure(self, env):
        from params import azkaban_executor_properties, log4j_properties, azkaban_common, global_properties
        key_val_template = '{0}={1}\n'

        with open(path.join(azkabanConfPath, 'azkaban.properties'), 'w') as f:
            for key, value in azkaban_common.iteritems():
                f.write(key_val_template.format(key, value))
            for key, value in azkaban_executor_properties.iteritems():
                if key != 'content':
                    f.write(key_val_template.format(key, value))
            if azkaban_executor_properties.has_key('content'):
                f.write(str(azkaban_executor_properties['content']))

        with open(path.join(azkabanConfPath, 'log4j.properties'), 'w') as f:
            if log4j_properties.has_key('content'):
                f.write(str(log4j_properties['content']))

        with open(path.join(azkabanConfPath, 'global.properties'), 'w') as f:
            if global_properties.has_key('content'):
                f.write(str(global_properties['content']))


if __name__ == '__main__':
    ExecutorServer().execute()
2.2.3.2 azkaban_web.py To configure 
# -*- coding: utf-8 -*-
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import json
import os.path as path
import time

from common import azkabanHome, azkabanWebTarUrl, azkabanWebTarName, \
    azkabanConfPath
from resource_management.core.exceptions import ExecutionFailed, ComponentIsNotRunning
from resource_management.core.resources.system import Execute
from resource_management.libraries.script.script import Script


class WebServer(Script):
    def install(self, env):
        Execute('yum install -y python-requests')

        tmpAzkabanWebTarPath = '/tmp/' + azkabanWebTarName
        Execute('mkdir -p {0}'.format(azkabanHome))
        Execute('wget --no-check-certificate {0} -O {1}'.format(azkabanWebTarUrl, tmpAzkabanWebTarPath))
        Execute('tar -xf {0} -C {1} --strip-components=1'.format(tmpAzkabanWebTarPath, azkabanHome))

        Execute(
            'cd {0} && '
            'chmod +x bin/start-web.sh && '
            'chmod +x bin/shutdown-web.sh && '
            'chmod +s bin/internal/internal-start-web.sh'.format(azkabanHome)
        )

        self.configure(env)

    def stop(self, env):
        Execute('cd {0} && ./bin/shutdown-web.sh'.format(azkabanHome))

    def start(self, env):
        self.configure(env)

        maxRetryCount = 10
        retryCount = 0

        from params import host_info, azkaban_executor_properties
        executor_port = int(azkaban_executor_properties['executor.port'])
        execHosts = host_info['azkaban_executor_hosts']
        urlTmpl = 'http://{0}:{1}/executor?action=getStatus'
        import requests
        while True:
            for execHost in execHosts:
                try:
                    url = urlTmpl.format(execHost, executor_port)
                    print(url)
                    resp = requests.get(url)
                    print(resp.text)
                    if json.loads(resp.text)['isActive'] == 'true':
                        Execute('cd {0} && ./bin/start-web.sh'.format(azkabanHome))
                        return
                except:
                    print('web is not alive')
                time.sleep(0.5)
            time.sleep(1)
            retryCount += 1
            if retryCount > maxRetryCount:
                raise Exception('web start failed')

    def status(self, env):
        try:
            Execute(
                'export AZ_CNT=`ps -ef |grep -v grep |grep azkaban-web-server | wc -l` && `if [ $AZ_CNT -ne 0 ];then exit 0;else exit 3;fi `'
            )
        except ExecutionFailed as ef:
            if ef.code == 3:
                raise ComponentIsNotRunning("ComponentIsNotRunning")
            else:
                raise ef

    def configure(self, env):
        from params import azkaban_common, azkaban_web_properties, azkaban_users, global_properties, log4j_properties
        key_val_template = '{0}={1}\n'

        with open(path.join(azkabanConfPath, 'azkaban.properties'), 'w') as f:
            for key, value in azkaban_common.iteritems():
                f.write(key_val_template.format(key, value))
            for key, value in azkaban_web_properties.iteritems():
                if key != 'content':
                    f.write(key_val_template.format(key, value))
            if azkaban_web_properties.has_key('content'):
                f.write(str(azkaban_web_properties['content']))

        with open(path.join(azkabanConfPath, 'azkaban-users.xml'), 'w') as f:
            if azkaban_users.has_key('content'):
                f.write(str(azkaban_users['content']))

        with open(path.join(azkabanConfPath, 'global.properties'), 'w') as f:
            if global_properties.has_key('content'):
                f.write(str(global_properties['content']))

        with open(path.join(azkabanConfPath, 'log4j.properties'), 'w') as f:
            if log4j_properties.has_key('content'):
                f.write(str(log4j_properties['content']))


if __name__ == '__main__':
    WebServer().execute()
2.2.3.3 common.py To configure 
# -*- coding: utf-8 -*-
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import os

import ConfigParser

script_dir = os.path.dirname(os.path.realpath(__file__))
config = ConfigParser.ConfigParser()
config.readfp(open(os.path.join(script_dir, 'download.ini')))

azkabanHome = '/data/azkaban'
azkabanConfPath = azkabanHome + '/conf'

azkabanWebTarUrl = config.get('download', 'azkaban_web_tar_url')
azkabanExecTarUrl = config.get('download', 'azkaban_executor_tar_url')

azkabanWebTarName = azkabanWebTarUrl.split('/')[-1]
azkabanExecTarName = azkabanExecTarUrl.split('/')[-1]
2.2.3.4 params.py To configure 
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from resource_management.libraries.script.script import Script

# config object that holds the configurations declared in the config xml file
config = Script.get_config()

azkaban_web_properties = config['configurations']['azkaban-web.properties']
azkaban_executor_properties = config['configurations']['azkaban-executor.properties']
azkaban_users = config['configurations']['azkaban-users']
azkaban_common = config['configurations']['azkaban-common']
global_properties = config['configurations']['global.properties']
log4j_properties = config['configurations']['log4j.properties']

host_info = config['clusterHostInfo']
host_level_params = config['hostLevelParams']
java_home = host_level_params['java_home']
2.2.3.5 download.ini To configure 
#  The configuration here is 1.2.1 In process /var/www/html/azkaban Two files in the directory 
[download]
azkaban_web_tar_url = http://ip/azkaban/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
azkaban_executor_tar_url = http://ip/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz

2.2.4 quicklinks.json File configuration 

{
    
  "name": "default",
  "description": "default quick links configuration",
  "configuration": {
    
    "protocol": {
    
      "type": "HTTP_ONLY"
    },
    "links": [
      {
    
        "name": "azkaban_web",
        "label": "Azkaban Web",
        "component_name": "AZKABAN_WEB",
        "requires_user_name": "false",
        "url": "%@://%@:%@:10200",
        "port": {
    
          "http_property": "jetty.port",
          "regex": "^(\\d+)$",
          "site": "azkaban-web.properties"
        }
      }
    ]
  }
}

2.3 restart ambari-server

ambari-server restart

Sign in ambari after , Already available in add service The interface finds Azkaban service

Insert picture description here

3、 ... and . Start the installation according to the interface prompts Azkaban

Follow the interface prompts step by step , To initialize Azkaban
here , When an error occurs, you can check it step by step according to the log prompts :

As shown in the figure below ,Azkaban Installation started successfully !

Insert picture description here

according to 2.2.2.3 azkaban-executor.properties.xml Configuration item azkaban.webserver.url Configured value Value access web Interface

according to 2.2.2.1 azkaban-users.xml Login with the configured account name and password azkaban

As shown in the figure below , Configuration complete !
Insert picture description here

Four . Ambari Integrate Azkaban Problems encountered in the process

4.1 azkaban Source code clean Operation error 

./gradlew clean Exception in thread "main"
java.lang.RuntimeException: Timeout of 120000 reached waiting for exclusive access to file:
/root/.gradle/wrapper/dists/gradle-4.1-all/bzyivzo6n839fup2jbap0tjew/gradle-4.1-all.zip at
org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:65) at
org.gradle.wrapper.Install.createDist(Install.java:48) at
org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)

terms of settlement :

  1. Delete the folder pointed to by the error 
/root/.gradle/wrapper/dists/gradle-4.1-all
  1. Execute the order again
    Insert picture description here

Problem solving :

4.2 azkaban Installation initialization error :

File "/var/lib/ambari-agent/cache/stacks/HDP/2.6/services/AZKABAN/package/scripts/azkaban_web.py", line 67
    self.configure(env)
       ^
IndentationError: expected an indented block

Most of the similar problems are python Script indentation problems ,python The requirements for indentation are very strict , Modify the script at the corresponding position according to the error prompt

原网站

版权声明
本文为[New objects first]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/02/202202161410258646.html

边栏推荐

猜你喜欢

随机推荐