Crawling data encounters single point login problem

Crawling website data is , Find the same token At the same time, it is kept in domainA.cn and domainB.com Next , adopt selenium To achieve login free access to data , Summary of knowledge points to solve cross domain problem finding .

Browser cross domain problem understanding ?

• The same-origin policy :

Three is the same : agreement , port , domain name

• The purpose of formulating homology strategy :

Prevent request forgery : Front end interaction , Get the server token stored in token Next , If there is no homology strategy ,token It can be easily obtained by other molecules .

• Three limitations of non homology :

Can't get the cookie、localstorage and indexedDB

Can't access non homologous pages DOM （iframe）

Can't send... To a non homologous address AJAX request or fetch request （ Can send , But the browser refused to accept the response ）

• How to solve cross domain problems :

• Server implementation CORS Interface understanding

The server interacts with the client ,http Add... To the request header ACCESS Start parameter to interact

A simple request : The actual request contains ACCESS parameter information

Pre inspection request : Before sending the actual request , Send a OPTION request , After passing the inspection , Send the actual request

The preview request with the certificate needs to be in the request header withCredentials Set to true

For details, please refer to the link : Cross source resource sharing （CORS） - HTTP | MDN

• Use JavaScript Of src Properties are implemented without cross domain restrictions

Request header X-Requested-With What is the function of parameters ？

      Can be judged ajax Request or other request . Server interpretation allows browsers to cross source servers , issue XMLHttpRequest request

Documents referenced in the article :

Cross source resource sharing （CORS） - HTTP | MDN

Thoroughly understand the cross domain of browser - Nuggets

Cross domain single sign on solution implementation - Nuggets

Different domain names pass JSONP Cross domain sharing cookie- Huang Yunxin's blog

https://segmentfault.com/a/1190000015597029