Experience record of rural housing integration script

The first day

First, I checked the environment , It is found that the entire working environment is mounted on the browser , It feels easy to use reptiles

Try Google to open , It turns out that it can't , There is a nested web page that cannot be displayed , Only use iE Old version or 360 Browser to open , however 360 What is used is chrome kernel , So first use 360（ Later I found out that 360 Not yet. ）

Technology stack direction , Consider direct http Request or automate selenium To operate . The web page uses aspx To render the , After checking the packet capture, I found that the image was magically encrypted , It's using T：0.314578…, So you can't just http Request completed （ I don't understand my own food , Later, I thought that I could actually go to the tool classes of the static resources to get the encrypted methods ）

Then try it first webdriver use 360, see 360 Of chrome kernel , Be careful 360 To re import options And path selection

Results found 360 There are the following problems

1. There are advertisements on the open page （ Even after adjusting the settings, there is still no way to solve , It may be initialized every time you open it ？）
2. Directly stuck in driver Start the program , No response after opening the page , Without rhyme or reason , And debug Don't come out , Don't complain

So I switched to IE browser

1. ie The version of the chromedriver It's based on selenium Version , It is not determined by the version number of the browser
2. To put ie The certificate of distrust is turned off , The untrusted certificates on all four pages should be closed （ a key ）
3. yes , we have ie It's going to be slow , yes , we have ie It will open very quickly （ It should be a version issue , I didn't want to compare ）
4. Remember to use win32 Version, not x86 edition

the second day

be familiar with selenium The operation of , Complete the body of the code

There are several points to pay attention to

1. iframe The operation of , Remember to go in one iframe Exit after the operation , Use simple attributes to locate iframe More convenient
2. There is one input Box input with caution , Input is not what you see on the surface input, There are several others hidden below input, The true value is converted later
3. Asynchronous waiting , There are three operations , The search has a lot of information , Don't tell me more about （https://www.jianshu.com/p/02005ec29d94）
4. left-click , Use actions().contextclick().perform() To operate

On the third day

Still want to switch back chrome browser

There may be two kinds of errors in searching the Internet

1. The first is because ie Kernel support for activeX To browse , use ietab Google plugin to solve
2. The second is to close iframe Access to the ,chrome stay 80 Series above shutdown prevents iframe Of src Is a cross-domain request

What I do

1. chrome Of ietab Plug in packaging , It can be packaged in developer mode , And then packed crx If the file is directly dragged into , There will be problems that need to be repaired bug, Therefore, the safe method is to directly decompress and then import the decompressed installation package , But it didn't work
2. Check directly on the Internet , To close two things , stay chrome Of extensition in , But it didn't work , Then lower the version number , First down to 78, Again 70, It's no use

give up , Or use ie To operate

The next day, the basic click in operation has been completed , The following is the operation of uploading pictures Information

1. adopt os.listdir Operation can get the names of all files in the directory
2. Prepare for standby sendkeys This package , The result can only be used python2 Version of , So give up this method
3. Then use pywin32 My bag , The operation of this package should cooperate with spy++ To use ,spy++ Want to have one visualstudio Of 2011 Please pay attention to the installation , Or install a 2011 The small patch operation of ,spy++ To get the name of the current window and id, And then use pywin32 Conduct input and submit operation , When uploading, you can access it by entering the absolute path of the file

The fourth day

1. use pandas Import excel Table and filter , Operation of the cycle , After the package is completed

2. Pay attention to str The following file class data , Otherwise, there will be problems when output to the box （ I can't remember clearly. , Is there a problem ）

3. Make a progress bar box , It's using progressbar This plugin , Then see the document for specific operations , But it must be changed to for loop , No use while To show the progress , Taken for granted ,while You can't know the termination condition of the

4. Preprocess the file before operation , Such as file naming conventions and querying empty folders

Some follow-up thoughts

Want to continue some optimization , For example, character recognition directly generates excel Table , What we're going to do is paddleocr

padddleocr Installation tutorial for

But my computer is a business and cannot undertake a large number of computing tasks , And the scanned pictures can not be accurately obtained , Because the number is garbled , Finally gave up the idea

Some novels about the installation package

How to install the package

Remember to install the corresponding version py38, instead of py37 And so on

Yes whlpip install xxx and tagpython setup.py install Two kinds of , recommend whl

Sometimes it is necessary to install one package with another package , So it's best to install it with a network to see what dependencies are needed

1. whl Directly under the corresponding directory pip install Corresponding whl file
2. tag Unpack it before installing use python setup.py install that will do

The requirements part of the package

Use code pip install -r requirements.txt

urllib3
selenium
six
xlrd
python_utils
python_dateutil
pytz
pywin32
numpy
pandas

Code section

Body code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver import ActionChains
from selenium.webdriver.support import expected_conditions as Ec

import win32gui
import win32con
import pandas as pd
import progressbar

import time
import os, sys
import datetime
import logging

options = webdriver.IeOptions()
driver = webdriver.Ie(options=options,executable_path=r"C:\Users\hp\Desktop\IEDriverServer.exe")

# chrome Browser operation 
# options = webdriver.ChromeOptions()
# driver = webdriver.Chrome(options=options,executable_path=r"C:\Users\hp4\Desktop\chromedriver.exe")

driver.implicitly_wait(20)
wait = WebDriverWait(driver,20,0.2)

progressbar.streams.wrap_stderr()
# logging.basicConfig(level=logging.NOTSET)
logging.basicConfig()

def open():
    driver.get('http://10.12.4.71:8080/DL.aspx')
    driver.find_element_by_id("txtPwd").send_keys("123")
    username = driver.find_element_by_id("comEmpTxt")
    username.clear()
    # username.send_keys('nongfang4-1hao')# Ding 
    # username.send_keys('nongfangyiti1hao')# gold 
    # username.send_keys('nongfang5hao')# Dong 
    # username.send_keys('nongfang2hao')#  food 
    # username.send_keys('nongfang3hao')# Xue 
    driver.find_element_by_id("btnlogin").click()

    # time.sleep(20)
    driver.find_element_by_xpath("//div[@class='acc_main_0'][5]/div/div[@class='box-open']/a").click()
    # xitongguanli = wait.until(Ec.presence_of_all_elements_located(By.XPATH,"//div[@class='acc_main_0'][5]/div/div[@class='box-open']/a"))
    # xitongguanli.click()

    driver.find_element_by_id("tree_5_2_span").click()
    # shiliguanli = wait.until(Ec.presence_of_all_elements_located(By.ID,"tree_5_2_span"))
    # shiliguanli.click()


def work(shoulihao,path):
    # shoulihao = str(shoulihao)

    # onreadystatechange = "top.frm_OnReadyStateChange(this);"
    iframe = driver.find_element_by_xpath('//iframe[@onreadystatechange="top.frm_OnReadyStateChange(this);"]')
    # iframe = driver.find_element_by_xpath('//div[@class="tabbody_0 tabbody_1"]/iframe')
    driver.switch_to.frame(iframe)

    # chkPrjId
    # driver.find_element_by_id("chkPrjId").click()
    if(i == 0):
        driver.find_element_by_xpath('//span[@name="chkPrjId"]').click()

    # txtPrjId
    query = driver.find_element_by_id('txtPrjId')
    query.clear()
    query.send_keys(shoulihao)
    query.send_keys(Keys.ENTER)

    driver.find_element_by_xpath('//a[@title=' + shoulihao + ']').click()

    # driver.switch_to.frame(iframe)
    # tabbody_0 tabbody_1
    driver.switch_to.default_content()
    # driver.find_element_by_id("frmMain")
    # driver.find_element_by_id("tab_BodyContent")

    #iframe = driver.find_element_by_xpath('//iframe[@onreadystatechange="top.frm_OnReadyStateChange(this);"]')
    iframe = driver.find_element_by_xpath('//div[@class="tabbody_0 tabbody_1"]/iframe')
    driver.switch_to.frame(iframe)

    # time.sleep(3)
    driver.find_element_by_id("worklayout")

    # title = " Expand the workspace "
    driver.find_element_by_xpath('//img[@title = " Expand the workspace "]').click();

    # title = " The attachment "
    driver.find_element_by_xpath('//img[@title = " The attachment "]').click();

    # id = "divAttachTree_2_a"
    youji = driver.find_element_by_id("divAttachTree_2_a")
    ActionChains(driver).context_click(youji).perform()

    #m_AttachList_addAttach Add the id name 
    driver.find_element_by_id("m_AttachList_addAttach").click()

    # path = r"C:\Users\hp\Desktop\ Guzhuang \00180"

    # Folder location of pictures 
    # listpath = exactpath + "\\" + path
    exactpath = "C:\\Users\\hp\\Desktop\\ Mogu village, Hainan town 2\\" + path
    os.listdir(exactpath)
    str = ""
    for file in os.listdir(exactpath):
        # str += file.split('.')[0] + " "
        str += '"'
        str += exactpath
        str += "\\"
        str += file
        str += '" '

    # print(str)

    dialog = win32gui.FindWindow("#32770"," open ")
    ComboBoxEx32 = win32gui.FindWindowEx(dialog,0,'ComboBoxEx32',None)
    ComboBox = win32gui.FindWindowEx(ComboBoxEx32, 0, 'ComboBox', None)
    Edit = win32gui.FindWindowEx(ComboBox, 0, 'Edit', None)
    button = win32gui.FindWindowEx(dialog, 0, 'Button', None)

    win32gui.SendMessage(Edit,win32con.WM_SETTEXT,None,str)
    win32gui.SendMessage(dialog,win32con.WM_COMMAND,1,button)

    driver.switch_to.default_content()

    driver.find_element_by_xpath('//li[@class="tabtitle_0"]/span/a').click()
    # time.sleep(2)


if __name__ == '__main__':
    # excel Subtract... From the number of rows in the table 2
    # excel Table No 2 The line corresponds to x Of the 0 Number 
    inital = 0
    x = inital
    # exactpath = r"C:\Users\hp\Desktop\ Zhanglian village, Hainan town 02"

    #excel The location of the table 
    io = r"C:\Users\hp\Desktop\ Hainan town （ Mogu ） Gao Jun .xls"

    open()

    data = pd.read_excel(io, sheet_name=0)
    shoulibianhao = data[" Acceptance No "]
    total = len(shoulibianhao)

    #  This is the location of the parcel number ,14 To 19 The last five , Empathy 16：19 The last three , This is where the folder is named 
    zongzhutongyibianma = data[" Unified parcel code "].map(lambda x: x[14:19])

    #  This is where it ends , Start with -2

    for i in progressbar.progressbar(range(total-inital)):
    # for i in progressbar.progressbar(range(14)):
        logging.error(' Acceptance No : %d', shoulibianhao[x])
        # print()
        work(str(shoulibianhao[x]), str(zongzhutongyibianma[x]))

        x = x + 1

Query preprocessing part

import pandas as pd
import os

io = r"C:\Users\hp\Desktop\ Cao Guozhi, Liuji village, Zhangguo Town, Xinghua City .xls"
data = pd.read_excel(io,sheet_name=0)
data = data[[" Acceptance No "," Unified parcel code ","ZL"]]
shoulibianhao = data[" Acceptance No "]
zongzhutongyibianma = data[" Unified parcel code "].map(lambda x: x[14:19])

os.chdir(r'C:\Users\HP\Desktop\ Zhangguo, liujicun ')

for i in zongzhutongyibianma:
    f = str('C:\\Users\\HP\\Desktop\\ Zhangguo, liujicun '+'\\'+i)
    if(os.path.isdir(i)):
        if not os.listdir(f):            
            print("%s, There are no pictures in this folder "%f)
    else:
        print("%d, This folder does not exist "%(str(i)))