当前位置:网站首页>Elk distributed log analysis system deployment (Huawei cloud)
Elk distributed log analysis system deployment (Huawei cloud)
2022-07-05 19:52:00 【51CTO】
Architecture diagram
One 、 Environmental preparation
1. Host list
Host name | IP Address | To configure |
es-0001 | 192.168.1.41 | 1cpu,1G Memory ,10G Hard disk |
es-0002 | 192.168.1.42 | 1cpu,1G Memory ,10G Hard disk |
es-0003 | 192.168.1.43 | 1cpu,1G Memory ,10G Hard disk |
es-0004 | 192.168.1.44 | 1cpu,1G Memory ,10G Hard disk |
es-0005 | 192.168.1.45 | 1cpu,1G Memory ,10G Hard disk |
Kibana | 192.168.1.46 | 1cpu,1G Memory ,10G Hard disk |
logstash | 192.168.1.47 | 2CPU、2G Memory ,10G Hard disk |
apache | 192.168.1.48 | 1PU、1G Memory ,10G Hard disk |
2. Software inventory
- elasticsearch-2.3.4.rpm
- logstash-2.3.4-1.noarch.rpm
- kibana-4.5.2-1.x86_64.rpm
- filebeat-1.2.3-x86_64.rpm
3. Plug in list :
- [x] elasticsearch plug-in unit :
software package | describe |
| bigdesk-master.zip | yes ES Cluster monitoring tools |
| elasticsearch-kopf-master.zip | One ElasticSearch Management tools , Provide right ES Cluster operated API |
| elasticsearch-head-master.zip | show ES Clustered # Display help information ?help topology , Indexable (index)、 node (node) Level of operation |
4. Huawei cloud jump board
To configure yum repositories :
[[email protected] <sub>] # rm -rf /etc/yum.repos.d/*.repo
[[email protected] </sub>] # curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.myhuaweicloud.com/repo/CentOS-Base-7.repo
[[email protected] <sub>] # yum clean all
[[email protected] </sub>] # yum makecache
[[email protected] <sub>] # yum install -y net-tools lftp rsync psmisc vim-enhanced tree vsftpd bash-completion createrepo lrzsz iproute
[[email protected] </sub>] # mkdir /var/ftp/localrepo
[[email protected] <sub>] # cd /var/ftp/localrepo
[[email protected] </sub>] # createrepo .
[[email protected] <sub>] # createrepo --update . # to update
[[email protected] </sub>] # systemctl enable --now vsftpd
[[email protected] ~] # cp -a elk /var/ftp/localrepo/elk
[[email protected] elk] # cd /var/ftp/localrepo/
[[email protected] localrepo] # createrepo --update .
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
Optimize system services
[[email protected] <sub>] # systemctl stop postfix
[[email protected] </sub>] # yum remove -y postfix firewalld-*
[[email protected] <sub>] # yum install chrony
[[email protected] </sub>] # vim /etc/chrony.conf
# Comment out server Start line , Add the following configuration
server ntp.myhuaweicloud.com minpoll 4 maxpoll 10 iburst
[[email protected] <sub>] # systemctl enable --now chronyd
[[email protected] </sub>] # chronyc sources -v // Verify configuration results ^* On behalf of success
[[email protected] <sub>] # vim /etc/cloud/cloud.cfg
# manage_etc_hosts: localhost // Comment out this line
[[email protected] </sub>] # reboot
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
Installation configuration ansible Management host
[[email protected] <sub>] # cp -a ansible_centos7 /var/ftp/localrepo/ansible
[[email protected] </sub>] # cd /var/ftp/localrepo
[[email protected] <sub>] # createrepo --update .
[[email protected] </sub>] # vim /etc/yum.repos.d/local.repo
[local_repo]
name =CentOS- $releasever – Localrepo
baseurl =ftp://192.168.1.252/localrepo
enabled = 1
gpgcheck = 0
[[email protected] <sub>] # yum makecache
[[email protected] </sub>] # yum install -y ansible // Go to Huawei cloud website to download the secret key , And upload the secret key to the springboard
[[email protected] <sub>] # mv luck.pem /root/.ssh/id_rsa
[[email protected] </sub>] # chmod 0400 /root/.ssh/id_rsa
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
5. Huawei cloud template machine ( Mirror image ) To configure
[[email protected] <sub>] # passwd root
[[email protected] </sub>] # rm -rf /etc/yum.repos.d/*.repo
[[email protected] <sub>] # curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.myhuaweicloud.com/repo/CentOS-Base-7.repo
[[email protected] </sub>] # vim /etc/yum.repos.d/local.repo
[local_repo]
name =CentOS- $releasever – Localrepo
baseurl =ftp://192.168.1.252/localrepo
enabled = 1
gpgcheck = 0
[[email protected] <sub>] # yum clean all
[[email protected] </sub>] # yum makecache
[[email protected] <sub>] # yum repolist
[[email protected] </sub>] # yum install -y net-tools lftp rsync psmisc vim-enhanced tree lrzsz bash-completion iproute
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
Optimize system services
[[email protected] <sub>] # systemctl stop postfix atd tuned
[[email protected] </sub>] # yum remove -y postfix at audit tuned kexec-tools firewalld-*
[[email protected] <sub>] # yum install chrony
[[email protected] </sub>] # vim /etc/chrony.conf
# Comment out server Start line , Add the following configuration
server ntp.myhuaweicloud.com minpoll 4 maxpoll 10 iburst
[[email protected] <sub>] # systemctl enable --now chronyd
[[email protected] </sub>] # chronyc sources -v
# Verify configuration results ^* On behalf of success
[[email protected] <sub>] # vim /etc/cloud/cloud.cfg
# manage_etc_hosts: localhost Comment out this line
[[email protected] </sub>] # yum clean all
[[email protected] ~] # poweroff
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
1. Host name resolution ( mutual ping through )
Two 、【 Deploy Elasticsearch】( all es node )
A. Deploy elasticsearch service
Install the software
Modify the configuration
# vim /etc/elasticsearch/elasticsearch.yml
17| cluster.name: my-ES // Configuration cluster name
23| node.name: es1 // Current host name
55| network.host:0.0.0.0 // Listen to all addresses / This machine IP
68| discovery.zen.ping.unicast.hosts: [ "es1", "es2", "es3"] // Declare cluster members ( Not all )
perhaps
# sed -i '/cluster.name/s/# cluster.name/cluster.name/' /etc/elasticsearch/elasticsearch.yml // Remove annotations
# sed -i '/cluster.name/s/my-application/my-ES/' /etc/elasticsearch/elasticsearch.yml
# sed -i '/node.name/s/# node.name: node-1/node.name: es-0002/' /etc/elasticsearch/elasticsearch.yml
# sed -i '/network.host/s/# network.host: 192.168.0.1/network.host: 0.0.0.0/' /etc/elasticsearch/elasticsearch.yml
# sed -i '/discovery.zen.ping/a discovery.zen.ping.unicast.hosts: ["es-0001", "es-0002"]' /etc/elasticsearch/elasticsearch.yml // Add configuration under the line
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
Start the service
Service validation
# ss -ntulp | grep 9200
# ss -ntulp | grep 9300
# curl http://es-0001:9200/
# curl http://192.168.1.41:9200/_cluster/health?pretty
{
"cluster_name" : "my-ES", // Cluster identity / name
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3, // Number of cluster members
"number_of_data_nodes" : 3, // How many nodes are there in the cluster
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
cluster: colony
health: health
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
B. Plug in deployment language application
Which machine is the plug-in installed on , Only use on which machine
For local installation file; For remote installation ftp.
# /usr/share/elasticsearch/bin/plugin install file:///root/file/bigdesk-master.zip
# /usr/share/elasticsearch/bin/plugin install file:///root/file/elasticsearch-kopf-master.zip
# /usr/share/elasticsearch/bin/plugin install file:///root/file/elasticsearch-head-master.zip
# /usr/share/elasticsearch/bin/plugin list // Check the installed plug-ins
Installed plugins in /usr/share/elasticsearch/plugins:
- kopf
- bigdesk
- head
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
Access plug-ins
1、 Huawei cloud is bound to the elastic public network IP to es-0005 node
2、http:// Elastic public network IP:9200/_plugin/ The plug-in name [bigdesk|head|kopf]
http:// Public network IP:9200/\_plugin/kopf
http:// Public network IP:9200/\_plugin/head
http:// Public network IP:9200/\_plugin/bigdesk
http:// Public network IP/info.php // Request method
Elasticsearch Basic operation
Inquire about _cat Method
[[email protected] <sub>] # curl -XGET http://es-0001:9200/_cat/ # Query supported keywords
[[email protected] </sub>] # curl -XGET http://es-0001:9200/_cat/master # Check the specific information
[[email protected] <sub>] # curl -XGET http://es-0001:9200/_cat/master?v # Show details ?v
[[email protected] </sub>] # curl -XGET http://es-0001:9200/_cat/master?help # Display help information ?help
- 1.
- 2.
- 3.
- 4.
Create index
Specify the name of the index , Specify the number of tiles , Specify the number of copies
Create an index using PUT Method , After the creation is completed, pass head Plug in validation
[[email protected] ~] # curl -XPUT http://es-0001:9200/tedu -d \
'{
"settings":{ // establish
"index":{ // Indexes
"number_of_shards": 5, // Number of slices
"number_of_replicas": 1 // Copy number
}
}
}'
It is equivalent to building a database
# curl -XGET http://es-0001:9200/_cat/indices?v // View index details
Browser access :http://122.9.96.62:9200/_plugin/head/ The change of , The block depth is the source data , Shallow copy
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
![image-20210512185824453](C:\Users\ Uncle gou \AppData\Roaming\Typora\typora-user-images\image-20210512185824453.png)
Add data
Query data
Modifying data
[[email protected] ~] # curl -XPOST http://es-0001:9200/tedu/teacher/1/_update -d \
'{
"doc": {
" years ": " A.D. 701"
}
}'
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
Delete data
[[email protected] <sub>] # curl -XDELETE http://es-0001:9200/tedu/teacher/1 // Delete one
[[email protected] </sub>] # curl -XDELETE http://es-0001:9200/tedu // Delete index
- 1.
- 2.
3、 ... and .Kibana Deploy
Upload kibana-4.5.2-1.x86_64.rpm
# yum install -y kibana
# vim /opt/kibana/config/kibana.yml // Modify the configuration file
2| server.port: 5601 // port
5| server.host: "0.0.0.0"
15| elasticsearch.url: http://es1:9200 // colony
23| kibana.index: ".kibana"
26| kibana.defaultAppId: "discover" //kibana default page
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
Service startup / verification
Access test
Import log data
Copy cloud disk public/elk/logs.jsonl.gz To the springboard machine
[[email protected] ~] # gunzip logs.jsonl.gz
# curl -XPOST http://192.168.1.41:9200/_bulk --data-binary @logs.jsonl // It will take a few minutes
- 1.
- 2.
Successful import :
![image-20210512214744461](C:\Users\ Uncle gou \AppData\Roaming\Typora\typora-user-images\image-20210512214744461.png)
To configure kibana, wildcard
![image-20210512215605715](C:\Users\ Uncle gou \AppData\Roaming\Typora\typora-user-images\image-20210512215605715.png)
The modification time is consistent with the log
![image-20210512215914935](C:\Users\ Uncle gou \AppData\Roaming\Typora\typora-user-images\image-20210512215914935.png)
【logstash】
install logstash
# yum install -y java-1.8.0-openjdk logstash
# vim /etc/logstash/logstash.conf // Create the configuration file manually
input {
stdin {}
}
filter{ }
output{
stdout{}
}
stdin The standard input (0)
stdout standard output (1)
stderr Error output (2)
# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf // Start service from profile
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
Plug in and debugging format
Use json Format string test {"a":"1", "b":"2","c":"3"}
边栏推荐
- MMO項目學習一:預熱
- What does software testing do? What are the requirements for learning?
- Force buckle 729 My schedule I
- What is the function of okcc call center
- CADD课程学习(7)-- 模拟靶点和小分子相互作用 (半柔性对接 AutoDock)
- S7-200smart uses V90 Modbus communication control library to control the specific methods and steps of V90 servo
- 函数的概念及语法
- Analysis of openh264 decoded data flow
- Is the education of caiqiantang reliable and safe?
- Android interview classic, 2022 Android interview written examination summary
猜你喜欢
Win10 x64环境下基于VS2017和cmake-gui配置使用zxing以及opencv,并实现data metrix码的简单检测
What is the core value of testing?
Parler de threadlocal insecurerandom
建议收藏,我的腾讯Android面试经历分享
third-party dynamic library (libcudnn.so) that Paddle depends on is not configured correctl
Bitcoinwin (BCW) was invited to attend Hanoi traders fair 2022
众昂矿业:2022年全球萤石行业市场供给现状分析
Necessary skills for interview in large factories, 2022android will not die, I will not fall
After 95, Alibaba P7 published the payroll: it's really fragrant to make up this
IBM has laid off 40 + year-old employees in a large area. Mastering these ten search skills will improve your work efficiency ten times
随机推荐
Which securities company is better and which platform is safer for mobile account opening
Wildcard selector
The city chain technology Digital Innovation Strategy Summit was successfully held
webuploader文件上传 拖拽上传 进度监听 类型控制 上传结果监听控件
openh264解码数据流向分析
Is it safe for Guosen Securities to open an account online?
What does software testing do? What are the requirements for learning?
Where is the operation of new bonds? Is it safer and more reliable to open an account
Apprentissage du projet MMO I: préchauffage
秋招字节面试官问你还有什么问题?其实你已经踩雷了
Concept and syntax of function
Gstreamer中的task
S7-200smart uses V90 Modbus communication control library to control the specific methods and steps of V90 servo
Using repositoryprovider to simplify the value passing of parent-child components
Realizing deep learning framework from zero -- LSTM from theory to practice [practice]
C#应用程序界面开发基础——窗体控制(5)——分组类控件
【obs】libobs-winrt :CreateDispatcherQueueController
Reptile exercises (II)
[untitled]
Autumn byte interviewer asked you any questions? In fact, you have stepped on thunder