当前位置:网站首页>51 lines of code, self-made TX to MySQL software!
51 lines of code, self-made TX to MySQL software!
2022-07-06 15:18:00 【Jane said Python】
Pay attention to me , Learn with an old watch Python、 Cloud server
One 、 Preface
Hello everyone , I'm an old watch , I watched it this morning B standing , I found that the home page gave me a recent attention up Lord (@ Is my _ Is my _ Is my , For the convenience of the following paragraphs, the following words are used Little yes Substitute for ) video , So I opened it , So there is the next story ~
Little yes What I want to achieve is a task given by the teacher : Read txt file , And store it to mysql, Just two days ago, I helped the readers write a excel to sqlserver Software for , And finally Little yes Said the present java Two problems with version :
Can only read string Type data ( I don't quite understand this , It may be read file format or file field type )
Dynamic modification of read file and database configuration is not supported ( Write a gui Or simply write the terminal logic program directly )
Let me think , I can , And then it provides the idea , Occupy a pit , Next is my implementation code .
Two 、 Start thinking
All source code of this project + Environmental Science + The test files are open source , If you don't want to see the code implementation process, you can skip to the next part Direct consumption method
.
2.0 Environmental preparation
What I use here is :
python 3.10
Third party packages and corresponding versions :
pandas==1.3.5
PyMySQL==1.0.2
SQLAlchemy==1.4.30
PySimpleGUI==4.56.0
Facilitate project environmental management , I usually use pipenv Create and manage virtual environments , If you're interested in , You can read what I wrote before pipenv Basic use tutorial .
pipenv install # Creating a virtual environment
pipenv shell # Enter the virtual environment
pip install pandas PyMySQL SQLAlchemy PySimpleGUI # Install the required packages in the virtual environment
exit # Exit virtual environment , Direct closure cmd Can also be
2.1 data fetch
Looking at the sample data, we found that there are 2 Two separators , Spaces and tabs \t
, So we also need to specify two separators when reading data , In addition, this file has no header , So it is convenient for data processing and storage , It is better to add the header , Consistent with the database field name .
The following code :
import pandas as pd
'''
read_csv Parameter interpretation :
1、 To read the file path
2、sep Specify the separator , Reading data , Use | You can add multiple delimiters
3、header=None There is no meter The first line is the header by default
4、engine Set up the program engine
'''
data = pd.read_csv('./resources/ctd2020-09-27.txt', sep=' |\t',header=None, engine='python')
data
It's not hard to see. , There are two columns for direct reading nan, This is because there are two spaces separated , No problem , We delete the entire column as nan The column of , After the data is read without error , We are adding a header , The implementation code is as follows :
# Read the file
def get_txt_data(filepath):
columns = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N']
data = pd.read_csv(filepath, sep=' |\t',header=None,engine='python')
# Delete all data nan The column of ( If there is such a column , You can add , No effect )
data.dropna(axis=1, how='all', inplace=True)
# Specifies the column name
data.columns = columns
return data
get_txt_data('./resources/ctd2020-09-27.txt')
2.2 Data processing
because Little yes There is no indication of what data processing will take place ( In addition to the above data reading obstacles ), So here we simply delete the duplicate columns , Share next time excel to sqlserver Some data processing will be involved ( Such as : Remove duplicates according to the specified fields 、 Date format conversion, etc )
# Data processing
def process_data(data):
# Contains no columns to process , Then simply remove the weight 、 Store in database
data.drop_duplicates(inplace=True)
return data
2.3 data storage
Because we want to store data in mysql, So before storage , We need to connect to the database first , What I'm using here is sqlalchemy+pymysql link mysql database , The code is as follows :
# Linked database
def link_mysql(user, password, database):
# create_engine(" Database type + Database driven :// Database user name : Database password @IP Address : port / database ", The other parameters )
engine = create_engine(f'mysql+pymysql://{user}:{password}@localhost:3306/{database}?charset=utf8')
return engine
And then use pandas Of to_sql The function can simply and quickly convert Dataframe Format data is stored in the database , If you are interested, please read what I wrote before Python Data storage read ,6 A thousand words to deal with all kinds of methods , There is a comparison in it. It can be used directly pymysql And use pandas Of to_sql The difference in the rate at which data is stored , The description may not be accurate , Welcome to read and correct .
# Store the data
def data_to_sql(data, user='root', password='Zjh!1997', database='sql_study', table='ctd'):
engine = link_mysql(user, password, database)
# call pandas Of to_sql Store the data
t1 = time.time() # Time stamp Unit second
print(' Data insertion start time :{0}'.format(t1))
# The first parameter : Table name
# The second parameter : Database connection engine
# The third parameter : Whether to store index
# Fourth parameter : If the table exists Just append data
data.to_sql(table, engine, index=False, if_exists='append')
t2 = time.time() # Time stamp Unit second
print(' Data insertion end time :{0}'.format(t2))
print(' Successfully inserted data %d strip ,'%len(data), ' Time consuming :%.5f second .'%(t2-t1))
Finally, we write a summary function , It is here that the above logic is connected :
# The text file is stored in mysql
def txt_to_sql(filepath, user='root', password='Zjh!1997', database='sql_study', table='ctd'):
# Read the file
data = get_txt_data(filepath)
# Data processing
data = process_data(data)
# data storage
data_to_sql(data, user, password, database, table)
2.4 Call function test
filepath = './resources/ctd2020-09-27.txt'
# Specify only the file path , Other parameters use default values Convenient test
txt_to_sql(filepath)
Then you can write a database reading function to further test whether the data is really stored in the database :
# Reading data from a database
def read_mysql(user='root', password='Zjh!1997', database='sql_study', table='ctd'):
engine = link_mysql(user, password, database)
# Read the sql sentence
sql = f'select * from {table} limit 3'
# The first parameter : Inquire about sql sentence
# The second parameter :engine, Database connection engine
pd_read_sql = pd.read_sql(sql, engine)
return pd_read_sql
Call function , Check storage , That's all right. .
Come here , The function part of the program is finished , The next step is to write gui Interface , Make it easier for users to use , such as : File path Database parameter input, etc .
2.5 Write a GUI
Here we are using PySimpleGUI
package , As the name says , It's really simple , The bottom layer is Python Self contained tkinter.
Import related packages
# Write a GUI
import PySimpleGUI as sg
# Call the data store function
from txt_to_sql import txt_to_sql
To write GUI Layout
# Set up GUI Layout
# Related parameters key: Receiving input data key target: The corresponding data is displayed to the specified target
# default_text: Set the default value of the input box
layout = [
[sg.Text(' Read the contents of the specified file , After processing, it is stored in the specified database table ~')],
[sg.FileBrowse(' Click to select a file ', key='filepath', target='file'), sg.Text(key='file')],
[sg.Text(' Login username '), sg.InputText(key='user', default_text='root', )],
[sg.Text(' The login password '), sg.InputText(key='password', default_text='Zjh!1997')],
[sg.Text(' Database name '), sg.InputText(key='database', default_text='sql_study')],
[sg.Text(' Stored table name '), sg.InputText(key='table', default_text='ctd')],
[sg.Button(' Start to deal with '), sg.Button(' sign out ')]
]
Create program window 、 Business logic
# Create a window program
window = sg.Window('Txt To MySQL', layout, default_element_size=(100,))
while True:
event, values = window.read() # get data
# print(event)
if event==' Start to deal with ':
# Pass input data into the data handler
txt_to_sql(values['filepath'], values['user'], values['password'], values['database'], values['table'])
else:
# event in (None, ' sign out '): # Click to exit Shut down the program
break
window.close()
According to the effect
In the layout section ,layout For a list data , Each element in the list is also a list , Means a column , The commonly used layout modules are :Text( This article shows that )、InputText( Input box )、Button( General button )、FileBrowse( Single file selection );
In the create window program section , It mainly sets the default window size default_element_size, Just set the width , Height adapts to layout controls , In addition, getting the input value is very simple , direct read that will do , The dictionary is returned , Data processing is more convenient .
Come here , We will complete the development of all programs , Then there is the method of using the program directly .
3、 ... and 、 Direct consumption method
Two code files , Remove whitespace and comments , also 51 Line code , Hey ~
3.1 function
GUI Interface , Support the selection of specified files 、 Enter the database user name password Database name Table name .
Read the specified file , After data processing , Stored in the specified database table , If the table does not exist, you can directly create a new table to store data ; Otherwise, add data directly to the data table .
3.2 Usage method
Download the project code :https://github.com/XksA-me/txt-to-mysql
Unpack and open the file :python-Jonny
, This document contains all python Code + Test data + Environmental Science +windows bat Run the file , Other documents are @ use Java Written txt to mysql Methods and related configuration files ,
Original project address :https://github.com/schatz0-0/txt-to-mysql Original project B Station video sharing address :https://www.bilibili.com/video/BV12b4y1J7pD
Continue with how to use python edition , First of all, we need to unzip the python Environment package , It can be extracted directly , No need for secondary installation . The relevant documents in the screenshot above explain :
├── Pipfile Virtual environment profile ( Never mind )
├── Pipfile.lock Virtual environments depend on package relationships ( Never mind )
├── __pycache__ ( Never mind )
│ └── txt_to_sql.cpython-310.pyc ( Never mind )
├── python-Jonny-tJ_VXFMA.7z ( Virtual environment zip , You need to decompress it directly )
├── requirements.txt ( The third... Required for this project Python package , Have been installed in the given virtual environment )
├── resources ( Test data )
│ └── ctd2020-09-27.txt
├── start.bat (windwos You can run the file directly under , Start project )
├── txt_to_sql.py (Python Code file , Contains data reading Handle Storage )
└── txt_to_sql_gui.py (Python Code file , contain gui Interface , Call in txt_to_sql.py file , So just run this file )
After the virtual environment is decompressed , According to our local directory , Modify the start.bat
file , The contents are as follows :
@echo off
C:
cd C:\Users\Administrator\Desktop\python-Jonny
C:\Users\Administrator\Desktop\python-Jonny\python-Jonny-tJ_VXFMA\Scripts\python txt_to_sql_gui.py
exit
I don't understand this very well , Sell now , The general meaning of the above is : Get into c Disk project directory , Then take advantage of the virtual environment python Executable file Run the code I can , Last exit Exit procedure .
What you need to modify is the file directory involved , Just be consistent with your local , I wrote it on the ECS and put it on c disc ( There's only one dish ), You can choose to put it on other plates , Easy to manage .
After modification , Just click start.bat
To run the project , A black box will pop up (cmd), And a gui Program interface , The log of program execution output will be displayed in the black box ( It's in the program print Or misinformation ),gui First, we need to click the button to select the stored file , Then enter the database information , Set the default value , And then click Start to deal with
Button to run the program 、 Store the data , Click the exit button to close the program .
Four 、 Can expand
Currently only supported txt, And the data format is the specified type ( Spaces or tabs \t separated ), Have the time , Everyone needs it , It can be extended to support multiple format files , Add a file suffix to identify
The interface is simple , I saw it in the morning [email protected] Is my _ Is my _ It's the video I sent , Just thought of using python Writing is also very convenient , Hasty time , The interface is quite general , But tools , It is important to realize the function at the beginning .
This project has many shortcomings and can be improved , Welcome to study and exchange ~
Record today + A simple clip , Tomorrow, I will give a video explanation , Welcome to my attention B standing : cousin Max.
Recent articles :
discarded Tkinter! A few lines of code are generated quickly and beautifully GUI!
How to build a nail robot that can reply automatically
finally “ make ” A tool that can be programmed anytime, anywhere
How to find me :
边栏推荐
- Collection集合与Map集合
- Capitalize the title of leetcode simple question
- Global and Chinese market of portable and handheld TVs 2022-2028: Research Report on technology, participants, trends, market size and share
- Oracle foundation and system table
- MySQL数据库(四)事务和函数
- 如何成为一个好的软件测试员?绝大多数人都不知道的秘密
- How to become a good software tester? A secret that most people don't know
- How to use Moment. JS to check whether the current time is between 2 times
- Want to change jobs? Do you know the seven skills you need to master in the interview software test
- The latest query tracks the express logistics and analyzes the method of delivery timeliness
猜你喜欢
Mysql database (I)
ucore lab2 物理内存管理 实验报告
What to do when programmers don't modify bugs? I teach you
Nest and merge new videos, and preset new video titles
What are the software testing methods? Show you something different
遇到程序员不修改bug时怎么办?我教你
Introduction to safety testing
線程及線程池
UCORE lab2 physical memory management experiment report
Future trend and planning of software testing industry
随机推荐
CSAPP shell lab experiment report
Brief description of compiler optimization level
Mysql database (III) advanced data query statement
Knowledge that you need to know when changing to software testing
What if software testing is too busy to study?
STC-B学习板蜂鸣器播放音乐
想跳槽?面试软件测试需要掌握的7个技能你知道吗
Differences between select, poll and epoll in i/o multiplexing
Global and Chinese markets of PIM analyzers 2022-2028: Research Report on technology, participants, trends, market size and share
MySQL数据库(三)高级数据查询语句
Face and eye recognition based on OpenCV's own model
Sorting odd and even subscripts respectively for leetcode simple problem
UCORE lab2 physical memory management experiment report
Brief introduction to libevent
What level do 18K test engineers want? Take a look at the interview experience of a 26 year old test engineer
[oiclass] maximum formula
Threads and thread pools
Global and Chinese market of pinhole glossmeter 2022-2028: Research Report on technology, participants, trends, market size and share
Stc-b learning board buzzer plays music
Opencv recognition of face in image