Data warehouse 4.0 notes - user behavior data generation

（ Shang Silicon Valley digital warehouse 4.0 video ）

1 Cluster synchronization script

[[email protected] ~]$ mkdir bin

stay /home/atguigu/bin Create under directory xsync file , So that the global call

[[email protected] ~]$ cd /home/yu/bin

[[email protected] ~]$ vim xsync

  Write the following code in this file

#!/bin/bash
#1.  Judge the number of parameters 
if [ $# -lt 1 ]
then
  echo Not Enough Arguement!
  exit;
fi
#2.  Traverse all machines in the cluster 
for host in node02 node03 node04
do
  echo ====================  $host  ====================
  #3.  Traverse all directories , Send... One by one 
  for file in [email protected]
  do
    #4  Judge whether the file exists 
    if [ -e $file ]
    then
      #5.  Get parent directory 
      pdir=$(cd -P $(dirname $file); pwd)
      #6.  Get the name of the current file 
      fname=$(basename $file)
      ssh $host "mkdir -p $pdir"
      rsync -av $pdir/$fname $host:$pdir
    else
      echo $file does not exists!
    fi
  done
done

  Modify the script xsync Have Execution Authority

[[email protected] bin]$ chmod 777 xsync

 2 SSH Password free login configuration

 node02 Generate public and private keys on ：

[[email protected] .ssh]$ ssh-keygen -t rsa

Copy to the target machine to be logged in without secret :

 [[email protected] .ssh]$ ssh-copy-id node02

[[email protected] .ssh]$ ssh-copy-id node03

[[email protected] .ssh]$ ssh-copy-id node04

Just test it

 

3 Analog data

3.1 Generate logs

take application.yml、gmall2020-mock-log-2021-01-22.jar、path.json、logback.xml Upload to hadoop102 Of /opt/module/applog Under the table of contents

establish applog route

[[email protected] module]$ mkdir /opt/module/applog

Drag the four files directly to /opt/module/applog Catalog

[[email protected] applog]$ java -jar gmall2020-mock-log-2021-01-22.jar

  Check the log

3.2 Cluster log generation script

  Write the following in the script

#!/bin/bash
for i in hadoop102 hadoop103; do
    echo "========== $i =========="
    ssh $i "cd /opt/module/applog/; java -jar gmall2020-mock-log-2021-01-22.jar >/dev/null 2>&1 &"
done 

  Modify the permissions [[email protected] bin]$ chmod 777 lg.sh

Delete previous logs

 [[email protected] bin]$ cd /opt/module/applog/

[[email protected] applog]$ rm -rf log

  Sync information

 104 I don't want it , Delete

 103 There are

  Execute the script , See if it works

 

  Check the log