Gstore weekly gstore source code analysis (4): black and white list configuration analysis of security mechanism

​ In the previous chapter, we introduced the user rights management of the security mechanism , Next, we will analyze the source code of the black-and-white list configuration in the security mechanism .

1.1 brief introduction

​ stay gstore The black and white list in the security mechanism refers to access IP Blacklist database and whitelist database ; Blacklist rules are ： In the blacklist library IP Will be denied access to , The white list rule is ： Only allow... In the whitelist library IP visit ; In principle, the black-and-white list is configured as one of two , If both are enabled, the white list takes precedence .

1.2 Configure black and white lists

​ First, through gstore Root directory init.conf File ： Start the blacklist or whitelist mode ; The default mode is blacklist , As shown in the following configuration file ：

[ghttp]
......
#  White list library file path , Blank means that... Is not enabled 
ip_allow_path=""
#  Blacklist library file path , Blank means that... Is not enabled 
ip_deny_path="ipDeny.config"
......

​ Then, according to the configured mode , You can copy the corresponding black-and-white list library sample file for configuration , Configuration supports single IP And a certain range of IP Group , As shown below ：

#ipDeny.config.example
#two kinds of ip format
#1. a single ip address 
1.2.2.1
#2. a closed intervel represent by "-", to allow for a segment of ips or deny a segment of ips
1.1.1.1-1.1.1.2


###########################################################
#                  Notice!!!!                             #
#    1. use "#" to comment out a Whole line               #
#    2. ip address write in one line                      #
#    3. do not include extra space or other char          #
#    4. keep it only numbers or with one '-'              #
#    5. to enable black list, use "--ipDeny=<filename>"   #
###########################################################

​

1.3 Black and white list management

​ After configuring the black and white list , We can also dynamically configure through interfaces ,ghttp The network service provides an interface for querying and updating the black and white list .

​ When ghttp When the network service starts , The black and white list will be initialized , Read configuration file init.conf The configured black and white list library path , And read and parse the database data of the black and white list , Some key codes are as follows ：

//  Global parameters related to the black and white list 
//  Whether to enable blacklist 
int blackList = 0;
//  Whether to enable the white list 
int whiteList = 0;
//  White list library path 
string ipBlackFile = "ipDeny.config";
//  Blacklist library path 
string ipWhiteFile = "ipAllow.config";
//  White list Library 
IPWhiteList* ipWhiteList;
//  Blacklist database 
IPBlackList* ipBlackList;

​ ghttp The function... Will be called when the server starts ghttp::initialize To initialize ：

int initialize(int argc, char *argv[])
{
    cout << "ghttp begin initialize..." << endl;
	cout << "init param..." << endl;
    Util util;
    //  Read init.conf Configuration parameters in the file 
    Util::configure_new();
    ......
    else
    {
        ......
        ipWhiteFile = Util::getConfigureValue("ip_allow_path");
        ipBlackFile = Util::getConfigureValue("ip_deny_path");
        cout<<"ipWhiteFile:"<<ipWhiteFile<<endl;
        cout<<"ipBlackFile:"<<ipBlackFile<<endl;
        //  Determine whether to configure the white list database 
        if (ipWhiteFile.empty()) {
            whiteList = 0;
        } else {
            whiteList = 1;
        }
        //  Determine whether to configure the blacklist library 
        if (ipBlackFile.empty()) {
            blackList = 0;
        } else {
            blackList = 1;
        }
        ......
    }
    //  Determine the enabling rules of the black and white list , When both are enabled , White list first 
    if (whiteList) {
        cout << "IP white List enabled." << endl;
        ipWhiteList = new IPWhiteList();
        ipWhiteList->Load(ipWhiteFile);
    } else if (blackList) {
        cout << "IP black list enabled." << endl;
        ipBlackList = new IPBlackList();
        ipBlackList->Load(ipBlackFile);
    }
}

​ When initializing the library of the whitelist IPWhiteList::Init and IPWhiteList::Load Function to parse （ The blacklist database is similar ）：

// IPWhiteList Parameters in ipList Used to hold IP library 
std::set<std::string> ipList;

// IPWhiteList Constructors 
IPWhiteList::IPWhiteList(){
    this->Init();
}
//  Initialize parameters ipList
void IPWhiteList::Init(){
    ipList.erase(ipList.begin(), ipList.end());
}
//  Documented IP library 
void IPWhiteList::Load(std::string file){
    this->Init();
    this->ReadIPFromFile(file);
}
//  analysis IP library 
void IPWhiteList::ReadIPFromFile(std::string file){
    ifstream infile(file.c_str());
    string line;
    if (!infile.is_open())
    {
        cout << "open white list file failed." << endl;
        return;
    }
    while(getline(infile, line)) {
        if (line.length() >= 7) {
            int pos = line.find("#");
            if(pos != -1)
                continue;
            this->ipList.insert(line);
        }
    }
    infile.close();
}

​ After initialization , If you need to update the black and white list database data in real time , You can call api Interface to modify , Will call ghttp::IPManager_thread_new（ explain ： The enabling rule of the black and white list does not support dynamic switching , Only through init.conf Make changes , Need to restart after modification ghttp The service takes effect ）, The code is as follows ：

/**
 * @brief IP manager
 * 
 * @param response 
 * @param ips ip string
 * @param ip_type 1-black ip 2-white ip
 * @param type  1-query 2-save
 */
void IPManager_thread_new(const shared_ptr<HttpServer::Response>& response, string ips, string ip_type, string type) {
	Document all;
	Document::AllocatorType& allocator = all.GetAllocator(); 
	all.SetObject();
    //  The query interface will return whether the currently effective rule is a white list or a blacklist , And the corresponding effective rules IP Library list 
	if ( type == "1") { // query
		Document responseBody;
		Document listDoc;
		responseBody.SetObject();
		listDoc.SetArray();
        //  During initialization, set the parameter change value by configuring the whitelist path 
		if (whiteList) // white IP
		{
			cout << "IP white List enabled." << endl;
			responseBody.AddMember("ip_type", "2", allocator);
			for (std::set<std::string>::iterator it = ipWhiteList->ipList.begin(); it!=ipWhiteList->ipList.end();it++)
			{
				Value item(kStringType);
				item.SetString((*it).c_str(), allocator);
				listDoc.PushBack(item, allocator);
			}
		}
        //  During initialization, set the parameter change value by configuring the blacklist path 
        // ( If the white list is in effect , The white list is preferred , Even if the blacklist library path is configured )
		else if (blackList) // black IP
		{
			cout << "IP black List enabled." << endl;
			responseBody.AddMember("ip_type", "1", allocator);
			for (std::set<std::string>::iterator it = ipBlackList->ipList.begin(); it!=ipBlackList->ipList.end();it++)
			{
				Value item(kStringType);
				item.SetString((*it).c_str(), allocator);
				listDoc.PushBack(item, allocator);
			}
		}
		......
		responseBody.AddMember("ips", listDoc, allocator);
        ......
	}
	else if (type == "2") { // save
		......
		vector<string> ipVector;
		Util::split(ips,",", ipVector);
		if (ip_type == "1") { // black IP
            //  Update the blacklist library and reload 
			if (blackList) {
				ipBlackList->UpdateIPToFile(ipBlackFile, ipVector, "update by wrokbanch");
				// realod ip list
				ipBlackList->Load(ipBlackFile);
			}
            ......
		}
		else if (ip_type == "2") { // white IP
            //  Update the whitelist library and reload 
			if (whiteList) {
				ipWhiteList->UpdateIPToFile(ipWhiteFile, ipVector, "update by wrokbanch");
				// realod ip list
				ipWhiteList->Load(ipWhiteFile);
			}
			......
		}
		......
	}
}

​ When updating the black and white list library, it will call IPWhiteList::UpdateIPToFile and IPBlackList::UpdateIPToFile function , Update the latest configuration data to the file , Some of the code snippets are as follows ：

/**
 * @brief 
 * 
 * @param file
 * @param ips
 * @param reason
 */
void IPWhiteList::UpdateIPToFile(std::string file, vector<std::string>& ips, std::string reason)
{
    ofstream outfile;
    outfile.open(file.c_str());
    if (!outfile)
    {
        cout << "open white list file failed." << endl;
        return;
    }
    //  Record the reason for each change 
    outfile << "#" << reason << "\n";
    for(vector<std::string>::iterator it = ips.begin(); it != ips.end(); it++)
    {
        outfile << (*it) << "\n";
    }
    outfile.close();
}

1.4 Black and white list verification

​ Black and white list verification is generally enabled gstore Network services （ Such as ghttp service ） after , When the database is operated through an external interface , The client that initiated the request IP To verify , Whether the black and white list rules are met , Some of the code snippets are as follows ：

//  In the sub thread of request processing, the interface of the request will be IP verification 
void request_thread(const shared_ptr<HttpServer::Response>& response, 
const shared_ptr<HttpServer::Request>& request, string RequestType)
{
    //  verification IP Whether it complies with the access rules of the black and white list 
	if (!ipCheck(request)) {
		string error = "IP Blocked!";
		sendResponseMsg(1101, error, response,remote_ip,"ipCheck");
		return;
	}
    ......
}

​ In function ghttp::ipCheck in , The rules that will determine the effectiveness of the current black and white list , Then verify according to the effective rules , The code is as follows ：

bool ghttp::ipCheck(const shared_ptr<HttpServer::Request>& request){
    //  Obtain requested IP Address 
	string remote_ip;
	unordered_multimap<std::string, std::string, case_insensitive_hash, case_insensitive_equals> m=request->header;
	string map_key = "X-Real-IP";
	pair<std::unordered_multimap<std::string, std::string, case_insensitive_hash, case_insensitive_equals>::iterator,std::unordered_multimap<std::string, std::string, case_insensitive_hash, case_insensitive_equals>::iterator> lu = m.equal_range(map_key);
	if(lu.first != lu.second)
		remote_ip = lu.first->second;
	else
		remote_ip = request->remote_endpoint_address;
	//  Implement the white list rule 
	if(whiteList == 1) {
		return ipWhiteList->Check(remote_ip);
	}
    //  Execute the blacklist rules 
	else if(blackList == 1){
		return ipBlackList->Check(remote_ip);
	}
	return true;
}

//  White list rules match ： Hit rule returns true, Otherwise return to false
bool IPWhiteList::Check(std::string ip){
    if(this->ipList.empty())
        return false;
    else if(this->ipList.find(ip) != this->ipList.end())
        return true;
    for(std::set<string>::iterator it=this->ipList.begin(); it!=this->ipList.end(); ++it) {
        // case for xxx.xxx.xxx.xxx-xxx.xxx.xxx.xxx
        string test = *it;
        if(test.find("-") != -1){
            std::vector<std::string> fields;
            std::vector<std::string> start;
            std::vector<std::string> end;
            std::vector<std::string> test_ip;
            boost::split( fields, test, boost::is_any_of( "-" ) );
            boost::split( start, fields[0], boost::is_any_of( "\\." ) );
            boost::split( end, fields[1], boost::is_any_of( "\\." ) );
            boost::split( test_ip, ip, boost::is_any_of( "\\." ) );
            bool res = true;
            for(int i = 0; i < 4; i++){
                int s = std::stoi(start[i]);
                int e = std::stoi(end[i]);
                int candidate = std::stoi(test_ip[i]);

                if(!(s <= candidate && candidate <= e)){
                    res = false;
                    break;
                }
            }
            if(res)
                return true;
        }
    }
    return false;
}

//  Blacklist rules match ： Hit rule returns false, Otherwise return to true
bool IPBlackList::Check(std::string ip){
    if(this->ipList.empty())
        return true;
    else if(this->ipList.find(ip) != this->ipList.end())
        return false;
    for(std::set<string>::iterator it=this->ipList.begin(); it!=this->ipList.end(); ++it){
        // case for xxx.xxx.xxx.xxx-xxx.xxx.xxx.xxx
        string test = *it;
        if(test.find("-") != -1){
            std::vector<std::string> fields;
            std::vector<std::string> start;
            std::vector<std::string> end;
            std::vector<std::string> test_ip;
            boost::split( fields, test, boost::is_any_of( "-" ) );
            boost::split( start, fields[0], boost::is_any_of( "\\." ) );
            boost::split( end, fields[1], boost::is_any_of( "\\." ) );
            boost::split( test_ip, ip, boost::is_any_of( "\\." ) );
            bool res = false;
            for(int i = 0; i < 4; i++){
                int s = std::stoi(start[i]);
                int e = std::stoi(end[i]);
                int candidate = std::stoi(test_ip[i]);
                if(!(s <= candidate && candidate <= e)){
                    res = true;
                    break;
                }
            }
            if(!res)
                return false;
        }
    }
    return true;
}

1.6 Summary

​ This chapter introduces the black and white list module of the security mechanism , Analyzed how to configure the black and white list 、 How to manage black and white lists and how to use black and white list rules to verify requests , It is suggested to combine the source code while reading Main/ghttp.cpp、Util/IPWhiteList.cpp、Util/IPBlackList.cpp Analyze it together , It will be easier to understand . The next chapter will analyze gstore Log tracing module in security mechanism .