2020-09-18 referer authentication URL escape

Mission 1：referer Extract and log

analysis

1. How to get it referer
2. How to write Sinan

Code disassembly

// In the parameter HttpRequest Type below 
bool BidRequestHandler::ProcessBidRequest(const shared_ptr<HttpRequest> request, shared_ptr<HttpResponse> response) {
    
	……
	BidHandlerFactory* factory = GetFactory(*request);//GetFactory Factory means instantiation 
	……
	shared_ptr<BidRequestHandler> handler = factory->NewHandler(*request);// hold handler“ operator ” The processor that handles this thing . Here is the bidrequest Bidding request processor 

// In the upper parameter HttpRequest type 
class HttpRequest : public HttpBaseMessage {
    
public:
    const std::string& uri() const;
    const std::string& refer() const;//refer() and uri（） The statement is the same , By reference uri（） The usage of refer（）
    ……

Sometimes there is no refer Of , You can see from the bottom that there is no time refer（） Returns an empty string ：

const std::string& HttpHeader::header(const std::string& name) const {
    
    const static std::string empty_str("");
    std::vector<std::pair<std::string, std::string> >::const_iterator iter;
    for (iter = headers_.cbegin(); iter != headers_.cend(); ++iter){
    
        if(iter->first.size()== name.size() &&
         boost::iequals(name, iter->first)){
    
            return  iter->second;
        }    
    }
    return empty_str;
}

There is no need to judge , An empty string can also fall into the log .

Realization

1. extract referer
// I see refer() and uri（） The statement is the same , By reference uri（） The usage of refer（）
//NETGO_LOG(debug) << "request uri is : " << request->uri();
2. Write Sinan
// There are several ways to write Sinan logs , This is written in （ key , value ）
//SetCompassGuardAttr(“is_retarget”, “true”);

So the task of writing logs is just one sentence ：

SetCompassGuardAttr<string>("gateway_refer", request->refer());

Expand ： The principle of log real-time analysis platform

How Sinan works
Reference resources ： Real time log analysis platform
https://cloud.tencent.com/developer/article/1082326
Logstash->Elasticsearch->Kibana
Logstash： Data collection engine . It supports dynamic data collection from various data sources , And filter the data 、 analysis 、 Enrich 、 Unified format and other operations , Then store it in the location specified by the user ;
Elasticsearch： Distributed search and analysis engine , It's highly scalable 、 High reliability and easy management . Near real time storage 、 Search and analysis operations . It is often used as a basic search engine for some applications , Make it have complex search function ;
Kibana： Data analysis and visualization platform . Usually with Elasticsearch In combination with , Search the data 、 Analyze and present in statistical charts ;
Filebeat：ELK New members of the protocol stack , A lightweight open source log file data collector , be based on Logstash-Forwarder Source code development , It's a replacement .

Mission 2：referer Blacklist judgment

（ Not many , Single digit ）

Realization

Blacklist format design

The following method will find the configuration file , We put... In this configuration file referer The blacklist of .
One problem is that the values of key value pairs in this configuration file only support int,double and string These three basic types , So there is no way to save the blacklist as list perhaps set. Plan to use string To store the blacklist , Use spaces to split （ because url There are no spaces in the , Will be escaped as + Number ）

  // This method will read from a configuration file key Corresponding value, In this case key by “ad_traffic_type”, If it does not exist, the default value is 0. You can use this method to get the blacklist 
  int ad_traffic_type =
      NS_ADPOS::AdPosComponent<NS_ADPOS::AD_POS>::GetCntlPoint(
          &pos_component_param_, "ad_traffic_type", 0);

Judge referer Whether to hit the... Stored in the blacklist url People

//  Judge header_referer This url Is it on the blacklist .
bool IsRefererInBlacklist(const std::string& header_referer) {
    
    std::string header_referer_black_list = "";
    std::string curr_referer = "";
    //header_referer_black_list = "opqrgo;andfop odjfaop goajgopahdoj a";
    header_referer_black_list = NS_ADPOS::AdPosComponent<NS_ADPOS::AD_POS>::GetCntlPoint(&pos_traffic_info, "header_referer_black_list", header_referer_black_list);
    while (header_referer_black_list.size()){
    
        size_t pos = header_referer_black_list.find_first_of(" ");
        if (pos == std::string::npos) {
    
            if (header_referer_black_list == header_referer) {
    
                return true;
            }
            return false;
        }
        else {
    
            curr_referer = header_referer_black_list.substr(0, pos);
            if (curr_referer == header_referer) {
    
                return true;
            }
        }
        header_referer_black_list = header_referer_black_list.substr(pos+1);
    }
    return false;
}

IsRefererInBlacklist(request->refer());

Expand ：url Special character encoding

website URL Escape encoding of special characters in

character	URL Encoding value
Space	%20
"	%22
#	%23
%	%25
&	%26
(	%28
)	%29
+	%2B
,	%2C
/	%2F
:	%3A
;	%3B
<	%3C
=	%3D
>	%3E
?	%3F
@	%40
\	%5C
|	%7C

Sometimes spaces are encoded as + Number , Sometimes it is encoded as %20.
reason ：W3C The standard stipulates , When Content-Type by application/x-www-form-urlencoded when ,URL Use the plus sign to query the parameter name and the space in the parameter value + replace , So almost all browsers that use this specification after the form is submitted ,URL The spaces in the query parameters will be added +.
And in another specification RFC2396, Definition URI in , URI All reserved characters in must be escaped to %HH Format , So spaces are encoded as %20, plus + It is also compiled as a reserved word %2B.
For some to follow RFC 2396 For standard applications , It may not accept a plus sign in the query string +, Think it's illegal characters . So a safe move is URL In the unified use of %20 To encode space characters .

Other meanings ：
1、 Change the space to a plus sign (+)
2、 Forward slash (/) Separate directories and subdirectories
3、 question mark (?) Separate URL And query
4、 Percent sign (%) Make special characters
5、# No. specifies the bookmark
6、& No. to separate parameters

be familiar with string The operation of is very important ~ I am off work hhh！！