Meteorological data processing example - matlab string cutting matching and R language date matching (data splicing)

Done a long time ago,Think about or throw it to review the record.
任务：Verify the forecast temperature data accuracy,The weather forecast data compared with the observatory data
数据处理目标：To the observation data of meteorological sites and crawl forecast data matching and joining together,Facilitate the prediction accuracy of subsequent processing.

图1 Observations of meteorological site data

图2 The weather forecast on the crawl data
下面开始：

Extract the station

The weather forecast data is at hand38×3=114个,Named after the place of pinyin,如"akesu.csv",站点数据95个,Named after the station number,如"50136.xlsx",Another station and place correspondingxls文件,And not all sites have corresponding weather forecast data.
First screened with weather data and the location of the site data,And the site place names corresponding,这部分我是用matlab实现（也可以用R,Because the previous plans to usematlabAnd write the code hasn't written,这里使用了strsplit、strfindFunction to cut string matching：

rawDataNum,rawDataStr]=xlsread('D:\大三\The observatory data\The name and stand number for.xls','C1:D99');
str='D:\大三\data\3new\';
files=dir(strcat(str,'*.csv'));
n=length(files);
filename=cell(1,n);
for i=1:length(files)
    a=strsplit(files(i).name,'.');
    filename{
    i}=a{
    1};
end
std=cell(99,1);
for i=1:99
std{
    i}=num2str(rawDataNum(i));
end
data=[rawDataStr,std];
filename=filename';
t=1;
for i=1:n
  idx=find(cellfun(@(x) ~isempty(strfind(x,filename{
    i})), rawDataStr));
  if ~isempty(idx)
  d{
    t,1}=data{
    idx,1};
  d{
    t,2}=data{
    idx,2};
  t=t+1;
  end
end

Extract the station and station number：
*

日期时间的处理

由于R语言读取xls、xlsx文件需要java环境,For the convenience of usingmatlab将xlsx转为csv）
Station data import,由于excelThe data format reason,The date of import display for five digits,Abnormal date display format,After all kinds of testing to find,Use the following statements into：

 as.Date(x,origin="1899/12/30")

解决
Relax before long,又有新的问题,在RDates in the default format for"2011-01-01",And forecast data read format for"2011/1/1"与"2011年1月1日”,对于前者,使用 as.Date(x)即可,对于后者,as.Date(“2011年1月1日”, “%Y年%m月%d日”)
查阅资料,RLanguage commonly used time processing with “zoo"、"lubridate”,其中"lubridate"包unique(ymd_hms(date1),ymd_hms(date2))Statement can take the time variable overlap match.
ps:安装zooPackage version error,需要手动安装,具体步骤参考：https://blog.csdn.net/lym152898/article/details/77572163
lubridatePackage is not applicable to3.5、3.6版本R,而RThe use of language is the biggest characteristic package,Stalled again.
RLanguage is another feature of the data type more,Data type conversion is convenient,Then use the following small white（菜鸡）方法：
The time format to"character"类型→直接利用intersect取交集→for循环匹配

Date match with joining together

This part of an error a lot,Mainly the data itself some problem,In general it can use：

merge(a,d,by="V1",all=F)

进行拼接,But does not apply to the data,Therefore had to write cycle：

path1='D:/大三/data/'
path2='D:/大三/data/3new/'
m<-read.csv('D:/大三/The observatory data/match.csv',header=F)
m<-as.matrix(m)
library(stringr)
 m[,1]<- str_replace(m[,1], "'","")
 m[,2]<- str_replace(m[,2], "'","")
 m[,1]<- paste(m[,1],'.csv',sep='')
 m[,2]<- paste(m[,2],'.csv',sep='')
fileName1 = dir(path1)
fileName2 = dir(path2)
d3<-list()
t<-1
for (i in 16:length(fileName2))
  {
    
  if(setequal(which(m[,1]==fileName2[i]),integer(0)))
    next;
  a<-read.csv(file = paste(path2,fileName2[i],sep = ''),header = T)
  fileName<-m[which(m[,1]==fileName2[i]),2]
  if(length(fileName>1))
    fileName=fileName[1]
  d<-read.csv(file = paste(path1,fileName,sep=""),header=F)
  b<-as.Date(d$V1,origin="1899/12/30")
  c<-as.character(a$V1)
  c<-gsub('/','-',c)
  library('stringr')
  c<-as.Date(c)
  a$V1<-c
  d$V1<-b
  a<-as.matrix(a)
  d1<-as.matrix(d)
  d2<-matrix()
  b<-as.character(b)
  c<-as.character(c)
  p<-intersect(b,c)
  if (length(p)!=0)
  {
    
    d2<-matrix(0,nrow = length(p),ncol =30 )
    for (j in 1:length(p))
    {
    
      mm<-which(b==p[j])
      n<-which(c==p[j])
      if (length(mm)>1|length(n)>1)
      {
    
        mm=mm[1]
        n=n[1]
      }
      s<-c(a[n,1:13],d1[mm,2:18])
      d2[j,]<-as.vector(s)
    }
  }
  d3[[t]]<-d2
  write.csv(d3[t],file = paste(path3,fileName2[i],sep = ''))
  t<-t+1
}

Managed to spell：’

总结

This is two and a half years ago big three written code and tasks,Now is very childish,当时得出的结论是1、在学习像R、matlab这种语言,Be sure to find out whether there is a similar function to solve this
2、RThe processing of language data types,Involving a large number of data conversion details,Can only be more familiar with multi-purpose
Although the processing semi-serious,But looked from now,This type of meteorological data processing is the basic operation,After my data processing intonc数据/卫星数据/栅格数据,Site for meteorological data processing has not been doing basic,After direction should also pay attention to on the satellite data and model.But weather stations data processing was my introduction to programming,如今想来,There is some miss.