当前位置:网站首页>Sqoop I have everything you want
Sqoop I have everything you want
2022-07-06 17:40:00 【Bald Second Senior brother】
Catalog
Sqoop brief introduction :
sqoop yes apache One of its “Hadoop And the relational database server ” Tools for .
Data import :
MySQL,Oracle Import data to Hadoop Of HDFS、HIVE、HBASE And other data storage systems
Export data :
from Hadoop Export data to relational database from the file system of mysql etc.
The configuration file :
1. Environment variable configuration
export SQOOP_HOME=/opt/module/ha/sqoop export PATH=$PATH:$SQOOP_HOME/bin
2. To configure sqoop Medium
file
hadoop,hive,hbase You can configure it if you need it
Sqoop Use
codegen
effect :
Generate code to interact with database records , Map tables in a relational database to a Java class , There are fields corresponding to each column in this class .
Example :
bin/sqoop codegen \ --connect jdbc:mysql://ljx:3306/company \ --username root \ --password 123456 \ --table staff \ --bindir /opt/module/sqoop/staff \ ### Folder needs to exist --class-name Staff \ --fields-terminated-by "\t" --outdir /opt/module/sqoop/staff ### Folder needs to exist
create-hive-table
effect :
Import table definitions Hive, Generate the... Corresponding to the table structure of the relational database hive Table structure
Example :
sqoop create-hive-table --connect jdbc:mysql://ljx:3306/sample --username root --password 123456 --table student --hive-table hive-student
eval
effect :
assessment SQL Statement and display the results , Often used in import Before the data , Get to know SQL Is the statement correct , Is the data normal , And the results can be displayed on the console .
Example :
sqoop eval --connect jdbc:mysql://ljx:3306/mysql --username root --password 123456 --query "select * from user"
export( Very important )
effect :
from HDFS( Include Hive and HBase) Export data to a relational database .
Example :
sqoop export --connect jdbc:mysql://ljx:3306/sample --username root --password 123456 --table student --export-dir /user/hive/warehous/jjj --input-fields-terminated-by "\t" -m 1
import( important )
effect :
Import tables from the database HDFS,( Include Hive,HBase) in , If the import is Hive, So when Hive When there is no corresponding table in , Automatically create .
Example : take mysql in user Import table data to hdfs in
sqoop import --connect jdbc:mysql://ljx:3306/mysql --username root --password 123456 --table user
import-all-tables
effect :
Can be RDBMS All tables in are imported into HDFS in , Each table corresponds to a HDFS Catalog
Limiting conditions :
Each table must have a single column primary key , Or designate --autoreset-to-one-mapper Parameters .
Each table can only import All columns , That is, columns cannot be specified import.
Example :
sqoop import-all-tables --connect jdbc:mysql://ljx:3306/sample --username root --password 123456 --warehouse-dir /all_table
import-mainframe
effect :
direct import host , This order is too violent , and import-all-tables Not much difference , Wait until you need to study .
job
effect :
Used to generate a sqoop Mission , It will not be executed immediately after generation , It needs to be done manually .
Example :
establish job
qoop job --create myjob -- import-all-tables --connect jdbc:mysql://ljx:3006/mysql --username root --password 123456 --warehouse-dir /user/all_user
Be careful : In the use of job To perform sqoop The instruction of needs to be in -- Add a space after
list-databases
effect :
List the available databases on the server
Example :
sqoop list-databases --connect jdbc:mysql://ljx:3306 --username root --password 123456
list-tables
effect :
List the available tables in the database
Example :
sqoop list-tables --connect jdbc:mysql://ljx:3306/mysql --username root --password 123456
merge
effect :
take HDFS The data under different directories in is merged and put into the specified directory
Example :
[[email protected] sqoop-1.4.7]$ bin/sqoop merge \ --new-data /tdata/newdata/ \ --onto /tdata/olddata/ \ --target-dir /tdata/merged \ --jar-file /opt/module/sqoop/staff/Staff.jar \ --class-name Staff \ --merge-key id
metastore
effect :
Recorded Sqoop job Metadata information , If the service is not started , So default job The storage directory of metadata is ~/.sqoop, Can be found in sqoop-site.xml Revision in China .
Example :
start-up sqoop Of metastore service
sqoop metastore
version
effect :
Display version information
Example :
sqoop version
The same thing sqoop take mysql Data transmission hive
sqoop import --connect jdbc:mysql://ljx:3306/sample --username root --password 123456 --table student --hive-import -m 1
And use import Import to hdfs Only more than --hive-import , If you want to specify hive Database and tables of can be used --hive-database And --hive-table To specify the
The same thing sqoop take mysql Data transmission hbase
sqoop import --connect jdbc:mysql://ljx:3306/mysql --username root --password 123456 --table help_keyword --hbase-table new_help_keyword --column-family person --hbase-row-key help_keyword_id
Be careful : Import time hbase The specified table should exist in
边栏推荐
- The art of Engineering
- Redis快速入门
- Guidelines for preparing for the 2022 soft exam information security engineer exam
- How uipath determines that an object is null
- Akamai talking about risk control principles and Solutions
- Flink parsing (VII): time window
- Concept and basic knowledge of network layering
- 【ASM】字节码操作 ClassWriter 类介绍与使用
- 【MySQL入门】第四话 · 和kiko一起探索MySQL中的运算符
- Interpretation of Flink source code (II): Interpretation of jobgraph source code
猜你喜欢
Chrome prompts the solution of "your company management" (the startup page is bound to the company's official website and cannot be modified)
C# WinForm中DataGridView单元格显示图片
C version selenium operation chrome full screen mode display (F11)
[ASM] introduction and use of bytecode operation classwriter class
虚拟机启动提示Probing EDD (edd=off to disable)错误
基于LNMP部署flask项目
Selenium test of automatic answer runs directly in the browser, just like real users.
Flink parsing (VII): time window
Huawei certified cloud computing hica
Automatic operation and maintenance sharp weapon ansible Foundation
随机推荐
沉淀下来的数据库操作类-C#版(SQL Server)
Concept and basic knowledge of network layering
【逆向】脱壳后修复IAT并关闭ASLR
基于LNMP部署flask项目
Flink analysis (I): basic concept analysis
[introduction to MySQL] third, common data types in MySQL
EasyRE WriteUp
C# WinForm系列-Button简单使用
Solr appears write Lock, solrexception: could not get leader props in the log
The art of Engineering
Virtual machine startup prompt probing EDD (edd=off to disable) error
MySQL basic addition, deletion, modification and query of SQL statements
自动化运维利器-Ansible-Playbook
The problem of "syntax error" when uipath executes insert statement is solved
CTF reverse entry question - dice
PyTorch 提取中间层特征?
C# NanoFramework 点灯和按键 之 ESP32
The NTFS format converter (convert.exe) is missing from the current system
The art of Engineering (3): do not rely on each other between functions of code robustness
Pyspark operator processing spatial data full parsing (4): let's talk about spatial operations first