当前位置:网站首页>Spark judges that DF is empty

Spark judges that DF is empty

2022-07-07 20:23:00 The south wind knows what I mean


Performance test of different schemes

I have the same question , And tested 3 There are three main solutions :

df!= null df.count> 0
df.head(1).isEmpty()
df.rdd.isEmpty

 About need 9366ms
 About need 5607 millisecond 
 About need 1921ms

Of course. 3 Species effective , But in terms of performance , This is the same in my machine in terms of execution time DF Found when executing these methods on

Update another method

/** *  Judge DataFrame Is it empty  * @param df DataFrame * @return true  Said is empty  or false  It means not empty  */
  def isEmpty(df: DataFrame): Boolean ={
    

    try{
    
      df.head()
      false
    }catch {
    
      case e: NoSuchElementException =>
        println(e.getMessage)
        true
    }

  }
原网站

版权声明
本文为[The south wind knows what I mean]所创,转载请带上原文链接,感谢
https://yzsam.com/2022/188/202207071815030300.html