Usage of pandas to obtain MySQL data

utilize Pandas obtain Mysql data

stay python In the operation mysql, Although convenient , But frequently get data from the server , Efficiency is very low . I tried to use it today pandas from Mysql Extracting data from the database , It's really convenient , And after getting the data once , utilize pandas Rich functions for various analyses , It's also handy .

1. First, establish a database connection , Establish a public conn As a connection object

import pymysql
import pandas as pd
class cardb():
    conn = None

    def connect_db(self):
        db_host = "XXXX"
        user = "XXXX"
        pw = "XXXX"
        try:
            #  Create connection database 
            self.conn = pymysql.connect(db_host, user, pw, "yiche_car_info", use_unicode=True, charset='utf8mb4')
            return self.conn
        except Exception as e:
            print(" Database connection exception ！ error %s", e)
            return None

2. pandas Use the established connection to obtain data

 try:
            self.conn.ping(reconnect=True)
        except Exception as e:
            print("%s" % e)
            return None
            sql=“select * from viewcarinfo”
        df = pd.read_sql_query(sql % (str_info, str_viewname), con=self.conn)
        print(df)

Got the data , It's a dataframe Format . A row by row index is called index, Columns are column Like table .

       pz_id cartype_id              pz_name  ...     Model level   Body type      headlamps 
0    m139120      m4758       2020 paragraph 1.2L Manual value version   ...      Light-Duty Vehicle     The hatchback       Halogen 
1    m111122      m2790         2014 paragraph 1.3L The standard version   ...      Light-Duty Vehicle     Three compartment       Halogen 
2    m133807      m3067       2019 paragraph 1.5L Manual aggressive version   ...      Light-Duty Vehicle     Three compartment     LED
3    m133409      m4758      2019 paragraph 1.2LAMT Comfortable edition   ...      Light-Duty Vehicle     The hatchback       Halogen 
4    m139423      m3167       2020 paragraph 1.4L Manually refresh the new version   ...      Light-Duty Vehicle     Three compartment       Halogen 
..       ...        ...                  ...  ...     ...  ...    ...
343  m129677      m4586  2018 paragraph 5.3L Hand in hand white house one 4 seat   ...   full-size SUV  SUV    LED
344  m136613      m3859      2020 paragraph 6.0TW12 The standard version   ...      Luxury cars     Three compartment     LED
345  m131852      m4373       2019 paragraph S680 Two tone Collection Edition   ...      Luxury cars     Three compartment    matrix LED
346  m132800      m2078    2019 paragraph GT6.0TW12 Open top version   ...      Luxury cars    convertible    matrix LED
347  m125538      m3044    2017 paragraph 6.8T Hand integrated long wheelbase version   ...      Luxury cars     Three compartment       Xenon 

3. First use python Establish what needs to be saved json Dictionaries

 json_res = {}
        json_res["item_name"] = []
        json_res["item_option"] = {}
        json_res["item_value"] = []
        json_res["car_prosys_name"] = car_prosys_name
        json_res["car_prosys_value"] = []
        json_res["car_prosys_series_value"] = []
        json_res["car_pricezone_name"] = car_pricezone_name
        json_res["car_pricezone_value"] = []
        json_res["car_pricezone_series_value"] = []

4. use pandas Analyze the data
   Usage of data query ：df.loc Is to locate data rows , Add conditions to realize data filtering .
   A. Equal filtering or data size filtering

item=" Halogen "
item1 = df.loc[df[" headlamps "] == item]  

be equal to ==, Greater than less than <,> You can filter out all data rows that are equal or meet the size condition numerically .
n1 = item1[str_feild].count(）
Count all headlights ==“ Halogen ” The number of rows .

B. Text fuzzy search filter
The fuzzy search of text can be used .str.contains This function
Such as

item=" Halogen "
item2 = df.loc[df[" headlamps "].str.contains(item)]

All halogen related data lines can be counted .

n2 = item2[str_feild].count(）

C. Union condition lookup
Calculate that all headlights contain “ Halogen ” The number of rows .
You can also combine 2 Search with more than conditions , Such as "&" yes and The effect of , and “|” yes or The effect of .

item3 = df.loc[(df[str_feild] == item) & (df[prosys_name] == ps_name)]
n3 = item2[str_feild].count(）

Statistics meet df[str_feild] == item as well as df[prosys_name] == ps_name) Data columns for .

5. Export dictionary as json Format , Can be used for flask,Django Data source .

 save_path = sys.path[0] + json_path + str_feild_id + ".json"
        with open(save_path, 'w') as wr:
            json.dump(json_res, wr)

6. use pandas Analyze the data
   pandas There are many usages at the beginning , Can be said to be numpy Upgraded version ,numpy More for arrays and matrices , and pandas More for
   similar excel and sql Advanced data processing , It can also be directly connected with the database , And there are many formats of access and Export （ common
   Such as csv,excel,json), It can be said that a big data processing tool .