Operation and maintenance security, not so simple

With IT Technology and business development , And the emergence of all kinds of security vulnerabilities , Operation and maintenance and safety are increasingly integrated , People pay more and more attention to operation and maintenance security , There is a new cross cutting area called “ O & M security ”. hackers 、 White hat is busy digging security holes in operation and maintenance , The enterprise is busy building the operation and maintenance security system , At that time, countless loopholes came in a stream , Every fortress rises from the ground . The author bases on his own years of operation and maintenance safety practice , Let's also discuss one or two . This article follows the train of thought from asking questions to responding to answers , First throw out the author's understanding of operation and maintenance security , It also explains the reasons for paying attention to operation and maintenance safety . Then, according to the work habits found in the front line of operation and maintenance security and the common problems faced by the enterprise , Sort out the classification of general operation and maintenance security issues . Then suit the remedy to the case , Put forward a good operation and maintenance security form ： It's not just the safety awareness of Engineers , It is more about a relatively complete operation and maintenance security system , From process to technology , Points, lines and surfaces are integrated to create .

What is O & M security ？

Let's first look at a Venn diagram , Real business 、 Operation and maintenance 、 The relationship of security is interrelated 、 Interdependent . From this picture , Three different safety related sub specialties have been derived ：“ Operation and maintenance + Security ”,“ Security + Operation and maintenance ”,“ Business + Operation and maintenance + Security ”. In the recruitment position of the Internet company , What we often see is the operation and maintenance security engineer 、 Safety operation and Maintenance Engineer , These two posts are better suited to each other . and “ Business + Operation and maintenance + Security ”, It is usually included in the position of safety engineer , Application, operation and maintenance security engineers appeared in recent years , In contrast, it is more in line with “ Business + Operation and maintenance + Security ” The positioning of .

O & M security = Operation and maintenance + Security

Operation and maintenance security studies the discovery of security problems related to operation and maintenance 、 Analysis and blocking ： Such as operating system or application version vulnerability 、 Access control vulnerability 、DDoS Attack, etc . obviously , Operation and maintenance security is based on operation and maintenance , In terms of enterprise architecture, it usually belongs to the operation and maintenance department or infrastructure department , The professional sequence of O & M safety engineers generally belongs to O & M engineers .

Safety operation and maintenance = Security + Operation and maintenance

Security operation and maintenance studies the operation and maintenance of security systems or equipment ： Like firewalls 、 Vulnerability scanner maintenance , Vulnerability excavation and emergency response . This is also obvious , Security operation and maintenance is under the security department , The professional sequence of safety operation and maintenance engineers also belongs to safety engineers .

Application operation and maintenance security = Business + Operation and maintenance + Security

The research of application operation and maintenance security is the operation and maintenance and security of business , It mainly includes safety risk assessment, safety scheme planning, design and implementation . In the organizational structure, the position belongs to the safety department , There are also business departments , The corresponding discipline sequence includes those belonging to the safety engineer , There are also development engineers .

by force of contrast “ Operation and maintenance + Security ”,“ Security + Operation and maintenance ”,“ Business + Operation and maintenance + Security ” The difference between the three sub specialties , We have defined the research fields and job responsibilities of operation and maintenance security . See here , Maybe you have questions , What causes O & M security to be so “ scenery ”？

Why do we attach importance to operation and maintenance security ？

so to speak ,2013 year -2014 The year is a watershed in the development of operation and maintenance security . The special feature of the past two years is that several major applications as Internet infrastructure have been exposed or attacked one after another , for example Struts2 Remote code execution vulnerability 、Openssl Blood dripping from the heart 、Bash Broken shell , And then “ The largest in history DDoS attack ” Leading to a large number of .cn and .http://com.cn Domain name cannot be resolved . After this , Enterprises have rapidly increased their investment in operation and maintenance security , Various operation and maintenance security issues have also attracted wide attention . Until today, , Operation and maintenance security has become the top priority of enterprise security construction .

Flawed software supply chain

• struts2 Remote code execution vulnerability

That year S2 When a loophole comes out , The whole Internet is crying . Here are the affected enterprises , Hardly anyone you don't know .

• openssl Blood dripping from the heart

Follow Struts2 The holes are the same , It's very lethal , At that time, the global Internet was greatly affected .

• xcode Developed ios app Infected Trojan horse

Researchers found that AppStore Upper TOP5000 Applications include 76 Money is infected . Later, it was found that the culprit was that developers downloaded from non Apple official channels xcode development environment .

The proportion of O & M security vulnerabilities is obvious

Now one day, a certain law, a certain eye , The statistical analysis data of domestic security situation that can be queried is very limited . Even one day , The user experience is not good enough , The statistical analysis function is not satisfactory . The rest , Various research reports have never included O & M security issues in a separate statistical category , So here we borrow 2016 year CNVD The statistics of , Network device vulnerabilities and operating system vulnerabilities that obviously belong to O & M security issues can be found , The proportion has exceeded 20%, Plus various application version vulnerabilities included in the application vulnerability , It is believed that the proportion of vulnerabilities belonging to the operation and maintenance security field will be extremely considerable .

High cost performance of O & M security vulnerability utilization

Attacks against O & M security vulnerabilities are typical “ A thousand gold coins ”, Its ROI Very high ： Small investment 、 Easy to find and use 、 Cause great harm .

According to Microsoft DREAD The model to measure the risk of O & M security vulnerabilities is as follows ：

Common O & M safety habits

Frequent O & M security incidents , On the one hand, the operation and maintenance or safety specifications are blank or not implemented , On the other hand, the O & M personnel lack strong O & M safety awareness , There are various safety habits in daily work, which lead to . You can take your seat according to the number , Think about whether you have stepped on the same pit before ？

modify iptables The configuration was not restored after , Even clear off iptables

Temporary emptying is required for testing iptables You can understand , But many people forget to restore , There is no automatic restore mechanism

iptables -F

The script does not check “*”、 Space 、 Variable

If we accept “ Not only is user input untrustworthy , Your input is also untrustworthy ”, Such a pit will be less trampled .

rm -rf /$var1/$var2

The service starts listening to all addresses by default

This is the case with most application default configurations , Enable listening to all addresses without effective access control , The danger is not far away .

bind-address 0.0.0.0

When too much permission is given to a file , Anyone can read and write

This heel phpinfo It's kind of like , Can give an intruder a push .

chmod 777 $dir || chmod 666 $script

use root Start the service

For most o & M personnel , As soon as you get on the machine root, Later use root Starting the service seems like a one-stop process .

#nohup ./server &

Too much trouble to be certified , No access control

This is more like listening to arbitrary addresses , It is usually the default configuration , The user is not aware of reinforcement .

#requirepass test

Stand alone installation docker Then ignore the check iptables, Lead to docker modify iptables Open Internet

docker It goes without saying that technology brings us convenience , But because docker But the security risks are not small . and ,docker daemon The default is to control the host iptables Of , If docker daemon Use tcp socket Or the launched container can be accessed externally , Then even the host The fall of the enemy is no problem . For example, the following boot container will tcp/443 The port is open to the public .

docker restart

*nat
:PREROUTING ACCEPT [8435539:534512144]
:INPUT ACCEPT [1599326:97042024]
:OUTPUT ACCEPT [4783949:343318408]
:POSTROUTING ACCEPT [4783949:343318408]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.1/32 -d 172.17.0.1/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.23.0.3/32 ! -i br-1bf61a2fa2e7 -o br-1bf61a2fa2e7 -p tcp -m tcp --dport 443 -j ACCEPT
*filter
:INPUT ACCEPT [1599326:97042024]
:OUTPUT ACCEPT [4783949:343318408]
-A INPUT -s 10.0.0.0/8 -j ACCEPT
-A INPUT -s 127.0.0.1 -j ACCEPT
-A INPUT -j DROP
# The last rule was bypassed 

Copy

sudo Authorization is too large , This causes the custom script to be authorized

If an attacker can modify the contents of the script, it will be as easy as a palm to raise the right .

sudo script.sh

For development or QA to grant authorization root jurisdiction , He makes trouble for you ？

We have always emphasized RBAC, But o & M is too busy , When there are too many requirements for development testers , Many o & M personnel will directly authorize them root jurisdiction , And they don't know much about system level access control , The resulting vulnerabilities are considerable .

[email protected]:/home/dev$su
[email protected]:/home/dev#whoami
root

Copy

key/token/ssh The private key is stored in txt In the document , There are also individuals ssh The private key is placed on the server

[email protected]:/home/op$ls ~/.ssh
id_rsa id_rsa.pub

Copy

Release the working code

After encountering interns, submit the project code github 了 , The reason for reply is git Mismatched . Although I don't know the truth , But I think , At least they are not aware of safety .

git remote add origin https://github.com/secondwatchCH/EFS.git
git push origin master

Copy

personal home Directories are so sensitive , Others use direct hosting services , At least .bash_history Leakage cannot escape

[email protected]:/home/dev$python -m HTTPSimpleServer

Safety risk is not considered in application and model selection

Apache Struts Version：Struts 2.5 - Struts 2.5.12 # Online business use is subject to S2-052 Affected S2 edition

No concept of software supply chain security

from xcode The event to the pip The authorities found malicious ssh library , Are showing us a truth ： The security risk of software supply chain is great . At present, there are some common problems among the operation and maintenance personnel ：

• ssh Client or development IDE Download from Baidu online disk
• Close your eyes , hold github/pypi/dockerhub And so on / library / The image is directly used in the production environment
• The default password or default configuration is not cleared

Common O & M security issues

We talked about operation and maintenance earlier 、 Some bad habits in thinking , Or lack of safety awareness , The following is a combination of vulnerability analysis and response , Common O & M security problems can be divided into the following categories ：

Sensitive ports are open to the public

db perhaps cache It is a sensitive application , It is usually deployed in the intranet , But if the deployed machine has an intranet and an intranet ip, And the default listening address is 0.0.0.0 Words , Then the sensitive port will be opened to the public . Such as mysql/mongodb/redis/rsync/docker daemon api And so on .

No certification for sensitive applications 、 Empty password or weak password

ditto , If sensitive applications use the default configuration , Authentication will not be enabled ,mysql/mongodb/redis/rsync/supervisord rpc/memcache And other applications without certification . Sometimes it is convenient to test , A weak password or an empty password is configured , Then the authentication is in vain .

Leakage of sensitive information , Such as code backup 、 Version tracking information 、 Authentication information disclosure

web.tar.gz/backup.bak/.svn/.git/config.inc.php/test.sql Information leakage can be seen everywhere , Everyone knows the danger , But from time to time, someone will step on the pit .

The application default configuration is not cleared

jenkins script/apache server-status And other default functions are not cleaned up , For example, the following figure can directly execute the command

The application system opens debug Pattern

Django debug Mode on exposure uri route ,phpinfo() Expose server information and even webroot etc. , The attacker can then use this to further penetrate , Many white hats should feel the same way , Found out sql Inject but not write webshell, If I can meet a phpinfo() That's the best thing .

Application vulnerabilities are not upgraded in time

The more general applications , The more often loopholes break out . There's a good saying ： It is not because of hackers that the world is not safe , But because of insecurity, there are hackers , There will be hackers to uncover that illusion , Let us find out that there is so much insecurity . therefore Struts2、OpenSSL、Apache、Nginx、Flash wait CVE follow close on succession .

Loose authority management

Do not follow the principle of minimum authority , Provide... For development root Authority or authorization to business account admin jurisdiction .

DDoS attack

DDoS Attacks are important to O & M personnel , It is a familiar safety problem . We all know that by taking up full bandwidth 、 Running out of resources and other methods can make the server unable to respond to normal requests , In the final analysis, it is an attack method of resource confrontation . If you only rely on server resources to resist , To filter , Here's the picture , In high traffic 、 Under high concurrency , It will only lead to avalanches . add DDoS There are a large number of attack platforms , And it's cheap , This makes DDoS Attacks become a way to suppress competitors 、 revenge 、 Blackmail is the first choice for schemers .

Traffic hijacking

Remember 2015 Nian Xiaomi 、 tencent 、 Microblogging 、 Today, headlines and other six companies jointly issued a statement calling on telecom operators to crack down on traffic hijacking ？ Here are three common traffic hijacking methods , This is also a chronic disease that has plagued the operation and maintenance security personnel for many years .

• arp hijacked ：ARP The basic function of the protocol is through the target device IP Address , Query the... Of the target device MAC Address , In order to ensure the communication . be based on ARP This working feature of the protocol , Hackers constantly send fraudulent messages to each other's computers ARP Data packets , Fake target IP Conduct ARP Respond to , So as to realize man in the middle attack .
• Domain name hijacking ： By hijacking the domain name DNS Analysis results , take HTTP Request hijacking to specific IP On , Causes the client and the attacker's server to establish TCP Connect , Instead of connecting directly to the target server .
• HTTP hijacked / Direct flow modification ： Fixed content insertion on the page in the data path , Such as advertising pop-up window and so on .

Case study

Previously, we discussed many o & M security habits and problem classifications , I'm going to talk about , These are the most familiar cases , Let's see how the O & M security vulnerabilities “ Cost performance ” Extremely high .

svn

• Deploy web Error in code .svn Directory upload
• Use rsync There was no... When uploading the code exclude fall .svn Catalog ,svn The warehouse is not used svn propedit svn:ignore < Directory or file > The way ignore Delete files or directories that should not be uploaded .
• The attacker took advantage of svn Information disclosure tools Svn-Tool perhaps svn-extractor Restore code

rsync

• rsync Use root User start , The module is not configured with authentication , Also open the default port 873
• The attacker took advantage of rsync Write crontab The task bounces successfully shell, And planted a mining Trojan horse

redis

• redis Use root User start , No authentication configured , Also open the default port 6379
• The attacker took advantage of redis Write ssh Public key to root User .ssh The directory was successfully logged onto the machine
• General deployment redis All the machines have an intranet ip, The attacker can use this to roam the Intranet

kubernetes

• k8s Of api Opening to the outside world , At the same time, authentication is not enabled
• Attacker calls api Create a container , Mount the container file system root directory in the host root directory , The attacker used to write crontab The task bounces successfully shell, And planted a mining Trojan horse on the host
• Sometimes the container is full of uncompiled code or private files can be pulled from the fallen machine docker Any image of the image warehouse , The consequences will be unimaginable , As below k8s Of api, It is very simple to call .

that , How to do a good job in operation and maintenance security ？ There is a saying in traditional Chinese medicine called "suit the remedy to the case" . We spend a lot of time dissecting the problem , It must start with the problem , By correcting or cultivating good o & M safety habits , Combined with the complete operation and maintenance security technology system , Is the way out of the problem .

Cultivate good operation and maintenance safety habits

Port open

• By default, it listens to the intranet or local network
• If you need to monitor all external networks ,iptables、password and acl Can add add

iptables

stay cmdb Designed for a machine or service iptables The rules , At the same time, the synchronization mechanism ：

• Use when deploying services cmdb Generated iptables Agree to update
• Once cleared during the test iptables Then use the automatic or manual method to brush back to the standard iptables

Rights management

• use puppet、ansible perhaps saltstack And other cluster management tools to uniformly manage the operating system permissions
• When advanced permissions are needed temporarily, add scheduled recycling manually , When the quantity is large, it is configured automatically

Script security

• Verify variables , Especially high-risk operation
• In principle, the script is not authorized sudo Password or grant 666 The permission bit of

Key management

• Don't let ssh The private key leaves your office computer
• listen IT Words , Revise your... Regularly corp Or domain password
• One reason to separate configuration from code is ： The account password cannot be written in the code

Service management

• Can not use root It's better not to start with root
• Do not put the service root directory in your home Catalog

Code management

• It is forbidden to upload work-related codes github！！！
• Study carefully git/svn And other version management tools
• Define your .gitignore, Especially delete your .DS_Store

application sizing

• Security is an important consideration in application selection
• Open source software that is less active in bug fixes and security coding , No matter how easy it is to use

Pay attention to the application security configuration document

• The official documentation for general applications will contain a chapter on security configuration , It needs to be deployed step by step , Configure the security section according to best practices , Instead of skipping over the trouble .

Enterprise level operation and maintenance security system

Security system , It's a big set of concepts . From the process specification , To the technical architecture , This article can not explain clearly . therefore , The enterprise level operation and maintenance security system discussed below , I will give a general introduction to the schemes that I have come into contact with or that have been put into practice , Involving the specific landing , I will write an article to discuss it in detail later .

First , Complete operation and maintenance security system , Actually, it belongs to a part of the enterprise security system , So generally, there will not be much difference in thinking . secondly , O & M security , More concerned about “ Operation and maintenance ”, So like business risk control 、 Anti fraud 、app Decompilation is not considered . Let's take a look at what a complete enterprise level operation and maintenance security system looks like .

Process specification

Operation and maintenance specifications are like human laws ,“ Freedom in life , But they are always in chains ”. This set of norms , Not just constraints 、 Guide the operation and maintenance personnel , It's also a constraint 、 Guide development testers , And all the participants in the production activities .

train

The training here is not a substitute for the safety awareness training of employees conducted by the security department , It is not suitable for R & D safety training for development testers , But only for the awareness and technical training of operation and maintenance personnel . For example, the security habits and security habits mentioned earlier in this article , Can be used as a blueprint for awareness training . And the technical system mentioned later , Can be used as the basis for technical training . This kind of training can be put into the school recruitment training course , It can also be put in the Department salon lecture .

The examination and approval + to examine + assessment

First , Review or approve , Not to hinder business development , It's not for nothing , Instead, we hope to reduce or avoid the neglect of safety caused by human factors through the process . Therefore, the permission application should be approved by the superior 、 The function opening shall be reviewed by security personnel or colleagues in the same group 、 The function online shall be evaluated and tested by the safety personnel . Of course , The implementation methods can be flexible and diverse , For example, by default , You can start approval according to product or business needs 、 Audit mechanism , Then put the evaluation mechanism into the business launch process , Only through evaluation can we go online . Enterprises that are relatively strong in the security sector or relatively attach importance to security , I believe the above mechanisms have been put into place .

Security report

Security visualization 、 Data is very important , It is one of the forms that embody the safety value , Therefore, through cooperation with enterprises SRC Or the docking of the security department , You can obtain O & M related vulnerabilities 、 Security incident statistics , Then conduct secondary processing according to internal requirements , Then, it will be sent to the operation and maintenance personnel or department leaders or even the technical director in the form of regular reports for review , On the one hand, let them understand the operation and maintenance security situation , This is usually seen as a lack of security , So that we can get a warning from the data , Or get the attention of superiors , Thus, it is possible to obtain more resources or promote the implementation of Safety specifications from top to bottom .

The implementation of process specifications includes but is not limited to the above points , But I think these are the most important .

Technical system

Access control

Network isolation under security domain partition

• The network layer ：192.168 Divided into office areas 、 Office service area and development machine network , Partial isolation ;10.x It is divided into IDC Physical intranet 、IDC Virtual intranet and public cloud virtual intranet , adopt IGP Interworking , You can apply for port mapping to the Internet ; Public network IP Only used for business extranet , It is forbidden to use the public network environment in the development and test environment ！
• System level ： The firewall is enabled by default for the installed image , Open only ssh、rdp Management port .ssh All users log in with public key , Disable enabling password authentication ; Authorize system permissions by role .
• application layer ： database 、 Cache applications are deployed on the intranet IP, The management interface shall not be opened to the public , Authorize according to the principle of minimum authority

Unified access control at entrance and exit level

• Building IDC Level unified entry , combination NAT Gateway to achieve access control .

at present BATJ Have their own enterprise level GW As a unified application layer portal , Use at the same time NAT Gateway outbound traffic .GW There are many ways to realize open source , Once as an enterprise GW It still needs self-study . and NAT gateway , Can be purchased with API Distributed hardware firewall or self-developed NAT gateway , solve IDC Internal network outgoing traffic RS There is no external network when directly returning to the external network IP The problem of , Or the server directly initiates external requests , And then adopt unified system management . At present, there are many sharing in the industry , Relevant ideas are not difficult to find .

• Sensitive port access control

Once there is a unified entrance and exit , The whole production network is like an office network , Sensitive port access can be shielded , Limit the outward flow to the inside , Effective in risk mitigation and attack blocking uplink .

Application layer access control

adopt WAF The brush 、 Current limiting is a general scheme , without WAF Can be applied acl Self control , such as nginx Of limit_rate perhaps haproxy Of acl.

Fortress machine

• The fortress machine can realize the unification of the operation and maintenance entrance , It can also achieve centralized access control and audit .
• The server ssh The port or service management background can only open the white list to the fortress machine .

Baseline audit and intrusion detection

I think baseline audit and intrusion detection are two different concepts , The former lies in the post audit , It is not qualified , The latter lies in pre prevention and in-process detection and response . On the concrete landing , Baseline audits usually rely on Fortress machines , Intrusion detection usually relies on security agent.

Fortress machine

Usually fortress machines have access control 、 Log audit 、 Operational behavior audit 、 Data upload, download, audit and authority management . however , System patch update and application version update , Is not covered by the fortress machine .

For the landing of Fortress aircraft , Purchasing equipment is the second , The key point is to integrate the entire operation and maintenance system , For some years, the cost of enterprise transformation is too high , And people are worried about its performance and availability .

Security agent

Of course , The system patch update and application version update mentioned above , Can be handed over to security agent To do . Intrusion detection 、 Baseline audit , Security agent It can fully cover . But because I have to run agent, Generally, no one wants to run commercial intrusion detection systems on their own machines , If self research, the development cycle will be long , It will also cause business concern ： Server monitoring agent、 Data upload agent And then we have to run safely agent, In case agent Can avalanche be caused by avalanche ？ At the end of the day , To gain the trust of the product , You have to be strong enough .

that , What kind of solution can everyone talk about ？ stay google Put forward beyondcorp after , The problem may have taken a turn for the better , That is to use light weight agent Gather information , Calculate 、 analysis 、 The decision is left to the big data background . Of course , It's hard for us to be like google That's based on rpc Protocol to do access control 、 Identity Authentication , So on top of the traditional fortress machine scheme , Combined with lightweight agent, Maybe a better way . Of course , Or the above sentence , If you have a solid foundation , Can win the trust of everyone , That's another matter .

Vulnerability scanning

At present, large and medium-sized enterprises who do not have their own vulnerability scanner , It will not develop the head office for commercial use ？ But I think there may be a common problem , The vulnerability scanner is too heavy . If you can emancipate your mind , Maybe you can try to start from the positioning of the scanner , In efficiency 、 Coverage , For example, large scanners are specialized in long-term 、 Wide coverage scanning is required , The lightweight scanner is aimed at high efficiency 、 Directional scanning . Now it's not just waf In combination with machine learning , Vulnerability scanners can also be combined with machine learning or big data analysis , According to the scanning log or experience , Do the automatic generation of strategies , Realize lightweight and accurate scanning rules .

CI/CD Security

CI/CD It is an important part of operation and maintenance . stay CI/CD There are many security vulnerabilities on the . Let's discuss how to publish and deploy applications safely .

Leakage of sensitive information

We all know that publishing code should exclude ： Source files and temporary files , Such as .py、.cc、*.swp(vim The temporary file ), Upload information files related to version management ( Such as .svn/.git), And packing / Backup file （ Such as .gz/.bak). This looks more like a specification , It's not , By adding hooks or filter modules to the code distribution system , You can find sensitive information in advance . For example, the code is submitted ssh Private key or account password configuration file , Just one webhook Can detect . The cost of implementation compared to the cost of problems , It's nothing .

Security audit of code or image

With docker Wide application of container technology ,CI/CD The safe landing is more hopeful . We all know , Use docker Containers need to be written dockerfile/docker-compose file ,docker build Then there is the mirror image , And then again docker pull、docker run Deployment Services , In fact, it can be combined with jenkins etc. CI/CD Tool tune CoreOS Official Clair Mirror the security audit tool for vulnerability scanning . Besides , Of course, RASP etc. Runtime Dynamic detection mechanism of mechanism , Also have foritity perhaps Cobra And other commercial or open source code auditing tools , It can also be used in combination with .

Certificate authority

Authentication and authorization mechanism , The main ideas shared are as follows ：

• SSH Password login is not allowed , You must log in with a public key
• The concept of creating a personal account , You must have one account for each person , Multiple people are not allowed to share a personal account
• Public accounts should be separated from personal accounts , Direct login is not allowed
• Password security requires attention to complexity verification
• Access control cannot be performed through the network layer or application layer , The authentication and authorization mechanism should be added
• RBAC： According to the role authorization
• The principle of minimum authority ： It is forbidden to configure the business root/admin Level database account , Authorize corresponding authorities according to business requirements .
• White list mechanism ： At the same time limit root/admin Level database accounts can only pass through the white list ip visit . If there is a default account and password, it should be deleted at the same time .
• Certification information management ： Speaking of docker Container piece , at present kubernetes Provides ConfigMap, Can be used to pass authentication configuration path or other indirect variables , Used to calculate authentication information . It can also be used. Hashicorp Vault Manage authentication information

DDoS defense

DDoS The defense follows the network architecture , It can be divided into cloud cleaning or IDC There are two cleaning modes , The former passes through DNS Or instead, the target IP Replace with a cloud VIP The way of drainage , The corresponding defense processes are divided into ： Flow analysis -> Traffic collection -> Flow suppression and other steps . The latter leads through the route traction mode , The corresponding defense processes are divided into ： Traffic collection -> Flow analysis -> Flow pull -> Flow suppression and other steps . The following is the flow collection 、 Flow analysis 、 Traffic traction, attack blocking and filtering are briefly introduced .

Traffic collection

• Cloud cleaning
• DNS： Usually web service , Use the domain name to provide external services , Only need to dns A The record points to advanced anti DDoS or cleaning VIP, perhaps dns cname Go to the cloud to clean the domain name .
• Reverse proxy ： Configure reverse generation , Usually used for those who take IP Directly providing external services , Such as game .
• IDC cleaning
• Traffic image / Flow spectrophotometer ： This way requires IDC Deploy cleaning or advanced anti DDoS cluster in the computer room , Detect abnormal traffic by mirroring traffic or splitting traffic on network devices .

Flow analysis

• Packet capture and capture 、 Packet analysis 、 Session restore and reorganization ： Recommended in actual production environment nDPI+PF_RING Realization , Of course ,Intel DPDK The technology is also very mature , The latter is becoming more and more popular .
• Application layer protocol analysis ： It is understood that some companies use Bro Analyze traffic , The test results show that the peak value is tens of Gbps The performance is also good . Of course ,Bro It can also be used. PF_RING Performance acceleration , There are also plug-ins that can be vomited to kafka analysis .
• The abnormal data flow is identified through the traffic analysis here , Then trigger the alarm , Proceed to the next step .

Flow pull

This is only for IDC Cleaning is effective , It is usually cleaning equipment and IDC Export equipment establishment BGP agreement , The cleaning equipment is directed to IDC The exit sends the traction route , that , Flow to the target IP All the flow will be sent to the cleaning equipment for filtration .

Attack blocking and filtering

Attack blocking is mainly black hole routing , Flow filtering mainly uses adaptive cleaning algorithm and various algorithm thresholds , This distinguishes between normal flow and abnormal flow , Then discard the abnormal traffic , Return normal flow .

Data security

Data security , It's better to develop 、 Joint planning and design scheme for business security . Generally, what O & M security can cover is access control 、 Certificate authority 、 Backup 、 Encryption, etc .

• Access control ： Distinguish between data sensitivity , Implement different levels of access control . But it should be strictly in accordance with db The principle of placing in the intranet .
• Certificate authority ： be based on RBAC To authorize . If it is more mature db Or big data clusters , You can also use dynamic calculation permissions 、 How to dynamically distribute permissions , Authorize only when necessary 、 Recycle after use .
• Backup ： Local backup and remote backup , The business needs to decide whether to encrypt the backup .
• encryption
• transmission ： Usually use https Achieve channel security . About https Yes 2 The best events ：1. Certificate procurement ： The development test environment or non important business can be used for free SSL certificate Let's Encrypt, The scheme supports automatic issuance 、 Renewal , Cross certification enables compatibility with most client environments , In addition, you can use https://www.ssllabs.com/ Conduct site security scanning and compatibility testing .2. Certificate deployment ： For site access CDN You need to put the certificate private key in CDN, perhaps tls The handshake phase consumes the performance of the server and may affect the business , have access to cloudflare Of keyless programme , Transfer computing pressure to a dedicated cluster , The cluster can use Intel QAT Hardware acceleration , At the same time, targeted optimization shall be made at the protocol level , So as to realize pressure transfer and performance optimization .
• Storage ： This is basically a development level or business security level consideration , But if the operation and maintenance security does it , It is usually only encrypted at the file system level , For example, use enterprise level solutions ecryptfs.
• desensitization ： Developers and testers need to pull data from backup data or logs for other purposes , At this time, it is necessary to pay attention to desensitization . It is usually used to replace 、 Add or delete fields 、 Remove features and remove correlations .

Emergency response to security incidents

The following is a general emergency response process for security incidents , Obviously, the operation and maintenance personnel 、 Security personnel need to cooperate with a lot of work , Among them, we need to pay attention to ：

• Protection site , The backup data
• Contact the product to assess the scope of impact
• Confirm whether it can be sealed first iptables Restrict Internet access
• Confirm the baseline audit and intrusion detection of the hacked machine
• Confirm whether there is any data leakage 、 Machine is root, Added abnormal users 、 Exception process 、crontab, Open exception port
• Confirm whether the hacked machine has an intranet ip, Check the monitor to see if it is used as a springboard
• Create an O & M work order , Track and recover the occurrence and handling of vulnerabilities

External cooperation

O & M security , The first is operation and maintenance . In daily work with IT、 Security and network departments are closely related , It is very important to keep good communication and information sharing with brother departments . Let's discuss the possibility of cooperating with them .

And IT department

Mainly office network security , In especial NAC： Network access system , Usually IT maintain , However, due to historical reasons or technical support needs ,NAC It may require technical support from O & M security personnel , For example, the fortress Machine Service mentioned above .

With the security department

Operation and maintenance security is a branch of security , But not under the management of the security department , But it is very close to the security department , It can be said that whether it is business security , Or operation and maintenance security , All are “ Standing on a giant ”.

• The security department provides infrastructure such as DDoS The defense system and external unified interfaces are as follows SRC etc.
• The security department provides SDL Support , The O & M department is more closely connected with the product department than the security department , Most of the time, it is necessary to arrive at the O & M first , To be safe , Therefore, we will promote safety training through O & M safety 、 Security architecture design and implementation 、 Penetration testing is not uncommon .
• Corresponding , Operation and maintenance security can also realize refined vulnerability operation according to the specific conditions of operation and maintenance departments and products , At the same time, it promotes the efficient repair of vulnerabilities .

With the network department

The operation and maintenance and network of many enterprises have been under the same department for a long time , Even after the split , The cooperation between the two is also the most . For O & M security , In access control and DDoS In terms of defense, it needs the support of the network department very much .

• Access control
• Such as the implementation of network isolation and unified access control
• DDoS defense
• Get through the Internet 、 Flow collection and includes ip Data sharing including asset information

This paper starts with the concept of operation and maintenance security , It is emphasized that the operation and maintenance security dilemma has led to our attention , The reasons for this dilemma are also analyzed from the aspects of security awareness and infrastructure construction , Then talk about the matter , I hope to cultivate the safety awareness through operation and maintenance 、 Construction of O & M Safety specifications and O & M safety technical system , To ensure the effective operation of a complete operation and maintenance security system , Escort business development .

This article comes from an internal training , From conception to writing , from ppt To the article , It took a few weeks , In the middle, intermittently , Barely written . Limited by the author's cognitive ability and technical precipitation , And the length of the article , Many places may not be clear enough or there may be mistakes and omissions . Once again, I will throw a brick to attract jade , I hope to get more guidance from you . meanwhile , I also hope to take this article to refresh your understanding of operation and maintenance security ： O & M security , It's not that simple .