当前位置:网站首页>Animation scoring data analysis and visualization and it industry recruitment data analysis and visualization
Animation scoring data analysis and visualization and it industry recruitment data analysis and visualization
2022-07-05 05:24:00 【Data ape from zero】
Data visualization course design
1, Animation scoring data analysis and visualization
2,IT Industry recruitment data analysis and visualization
1, Animation scoring data analysis and visualization
1.1 Data capture
Upload the captured file to ${HIVE_HOME}/mydata Under the table of contents
1.2 Hive Table creation and import
1.2.1 establish cartoon_info Table and import the data
CREATE EXTERNAL TABLE Json( data string )
Load data to Json Spare in the table
load data local inpath 'mydata/infos_total.json' overwrite into table Json;
establish cartoon_info surface
drop table if exists cartoon_info; CREATE EXTERNAL TABLE cartoon_info( `ssid` string, `cartoon` string, `views` bigint, `coins` int, `follow` int, `series_follow` int, `danmakus` int, `likes` int, `favorite` int, `favorites` int, `reply` int, `share` int, `cover` string, `url` string, `episodes` int, `count` int, `is_finish` int, `pub_time` TIMESTAMP, `media_tags` string, `voice_actor` string, `score` float ) stored as parquet location '/warehouse/cartoon_info';
Use Json Parsing insert data , Details please see : Hive And Json analysis ( Ordinary Json and Json Array )
insert overwrite table cartoon_info select json_tuple(json,'ssid' ,'cartoon' ,'views' ,'coins' ,'follow' ,'series_follow' ,'danmakus' ,'likes' ,'favorite' ,'favorites' ,'reply' ,'share' ,'cover' ,'url','episodes' ,'count' ,'is_finish' ,'pub_time','media_tags','voice_actor','score') from ( select explode(split(regexp_replace(regexp_replace(data,'\\[|\\]',''),'\\}\\, \\{','\\}\\;\\{' ) ,'\\;')) as json from Json )a;
1.2.2 establish cartoon_comments surface
CREATE EXTERNAL TABLE Json2( data string );
Load data to Json2 Spare in the table
load data local inpath 'mydata/comments_total.json' overwrite into table Json2;
establish cartoon_comments Table and import the data
drop table if exists cartoon_comments; CREATE EXTERNAL TABLE cartoon_comments( `mid` string, `uname` string, `ssid` string, `message` string, `like` int, `dt` timestamp ) stored as parquet location '/warehouse/cartoon_comments';
Use Json Parsing insert data , Details please see : Hive And Json analysis ( Ordinary Json and Json Array )
insert overwrite table cartoon_comments select json_tuple(json,'mid' ,'uname' ,'ssid' ,'message' ,'like' ,'dt' ) from (select explode(split(regexp_replace(regexp_replace(data,'\\[|\\]',''),'\\}\\, \\{','\\}\\;\\;\\;\\{' ) ,'\\;\\;\\;')) as json from Json2)a;
Two IT Industry recruitment data analysis and visualization
1.1 Data capture
1, You need to log in to dragnet !! Please pay attention to replacing individuals Cookie And Cookie Don't have Chinese , Otherwise, an error will be reported ; If Cookie Don't take effect , Please open other pages of dragnet to get Cookie.
2, If an error is reported, please open dragnet to check whether it needs to be verified
Upload the captured file to ${HIVE_HOME}/mydata Under the table of contents
2.1 Hive Table creation and import
CREATE EXTERNAL TABLE Json3( data string )
Load data to Json3 Spare in the table
load data local inpath 'mydata/jobsInfo.json' overwrite into table Json3;
2.1.1 establish jobs_info Table and import the data
drop table if exists jobs_info; CREATE EXTERNAL TABLE jobs_info( `job` string, `keyword` string, `place` string, `requirement` string, `salary` string, `tags` string, `welfare` string, `pubtime` date ) stored as parquet location '/warehouse/jobs_info';
Use Json Parsing insert data , Details please see : Hive And Json analysis ( Ordinary Json and Json Array )
insert overwrite table jobs_info select json_tuple(json,'job' ,'keyword' ,'place' ,'requirement' ,'salary' ,'tags' ,'welfare' ,'pubtime') from ( select explode(split(regexp_replace(regexp_replace(data,'\\[|\\]',''),'\\}\\, \\{','\\}\\;\\{' ) ,'\\;')) as json from Json3 )a;
3, Data analysis and Visualization
3.1 Pyhive Connect Hive course :
Python install sasl,thrift,thrift-sasl And connect PyHive
Connection code : Pyhive
3.2 Data analysis and Visualization
Install the necessary packages
pip install pandas==0.23.4 pip install pyecharts==1.9.1 pip install matplotlib==3.5.1 pip install numpy==1.18.5 pip install jieba==0.42.1 pip install squarify==0.4.3
1, Animation scoring data analysis and visualization Data analysis code :bilibili
The code contains [" Rose chart "," Clouds of words "," Radar map "," Scatter plot "," Funnel diagram "," Ring graph "," Bar chart "," Tree diagram "," Matchstick "," Subgraphs "] common 10 Types of Graphs , Contains 4 individual matplotlib Figure and 6 individual pyecharts A simple analysis of the graph .
2,IT Industry recruitment data analysis and visualization Data analysis code :IT
The code contains [" Rose chart "," Clouds of words "," Pictogram "," Scatter plot "," Funnel diagram "," Ring graph "," Bar chart "," Tree diagram "," Matchstick "," Subgraphs "] common 10 Types of Graphs , Contains 4 individual matplotlib Figure and 6 individual pyecharts A simple analysis of the graph .
边栏推荐
- To the distance we have been looking for -- film review of "flying house journey"
- [merge array] 88 merge two ordered arrays
- [to be continued] [depth first search] 547 Number of provinces
- Introduction to tools in TF-A
- [turn to] MySQL operation practice (III): table connection
- Haut OJ 1347: addition of choice -- high progress addition
- Haut OJ 1245: large factorial of CDs --- high precision factorial
- High precision subtraction
- [转]: OSGI规范 深入浅出
- Under the national teacher qualification certificate in the first half of 2022
猜你喜欢
Yolov5 ajouter un mécanisme d'attention
Count sort
Learning notes of "hands on learning in depth"
服务熔断 Hystrix
[turn to] MySQL operation practice (III): table connection
Remote upgrade afraid of cutting beard? Explain FOTA safety upgrade in detail
Collapse of adjacent vertical outer margins
Applet Live + e - commerce, si vous voulez être un nouveau e - commerce de détail, utilisez - le!
To the distance we have been looking for -- film review of "flying house journey"
Gbase database helps the development of digital finance in the Bay Area
随机推荐
[to be continued] [UE4 notes] L2 interface introduction
PMP考试敏捷占比有多少?解疑
YOLOv5添加注意力机制
Haut OJ 1347: addition of choice -- high progress addition
远程升级怕截胡?详解FOTA安全升级
xftp7与xshell7下载(官网)
Remote upgrade afraid of cutting beard? Explain FOTA safety upgrade in detail
服务熔断 Hystrix
Haut OJ 1316: sister choice buys candy III
Fragment addition failed error lookup
YOLOv5添加注意力機制
被舆论盯上的蔚来,何时再次“起高楼”?
Romance of programmers on Valentine's Day
Pointnet++的改进
lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: meta line 6 and head, line 8, column 8
C language Essay 1
Quick sort summary
[turn to] MySQL operation practice (I): Keywords & functions
Bubble sort summary
Pointnet++学习