Spark - logging simple to use

One . introduction

Use Spark Running a task to log often encounters a problem that there are too many logs , Except for your own print Beyond the log , There are still a lot of it Executor、client Log , On the one hand, the task will occupy more machine storage during operation , Secondly, it is not convenient to query your own print journal . Here are some common log systems and usage methods .

Two . Common log systems

  A common logging system is Log4j and SLF4J, With Log4j For example , Set for a task logLevel Use the following syntax ：

    Logger.getLogger("org.apache.spark").setLevel(Level.ERROR)

among Level It includes the following types ：

LogLevel	Level	Use
OFF	2147483647	Turn off all logging
FATAL	50000	As translated , Fatal mistake
ERROR	40000	Error message prompt , Generally need Try Catch
WARN	30000	Potential error messages
INFO	20000	Normal log information
DEBUG	10000	Fine grained logging , For application debugging
TRACE	5000	Finer grained log information than debugging
ALL	-2147483648	Turn on all logging

    protected Level(int level, String levelStr, int syslogEquivalent) {
        super(level, levelStr, syslogEquivalent);
    }

    public static final Level OFF = new Level(2147483647, "OFF", 0);
    public static final Level FATAL = new Level(50000, "FATAL", 0);
    public static final Level ERROR = new Level(40000, "ERROR", 3);
    public static final Level WARN = new Level(30000, "WARN", 4);
    public static final Level INFO = new Level(20000, "INFO", 6);
    public static final Level DEBUG = new Level(10000, "DEBUG", 7);
    public static final Level TRACE = new Level(5000, "TRACE", 7);
    public static final Level ALL = new Level(-2147483648, "ALL", 7);

every last level Each class corresponds to a level Of int character , If you set logLevel stay DEBUG Level , Is lower than its corresponding level = 10000 Of TRACE、ALL Type logs are not displayed , So you can set it according to your own needs level Level .Log4j The general recommended type is ERROR、WARNING、INFO、DEBUG.

3、 ... and .Spark Logging

Use Log4j perhaps SLF4j Need to go through XML perhaps log4j.properties Configuration related information , It also needs to be introduced Corresponding log rely on , Often appear NoSuchMethod The conflict of , Very inconvenient ,spark since 2.4+ Introduced Logging, It is based on  SLF4j Defined a Trait, So that the log can be used only by inheriting the interface .

trait Logging {

  // Make the log field transient so that objects with Logging can
  // be serialized and used on another machine
  @transient private var log_ : Logger = null

   ...
}

You can see here Logger use @transient  Keyword decoration ensures that no error will be reported when serializing .

1. Output different types of logs

import org.apache.spark.internal.Logging

object LogUtil extends Logging {

  def main(args: Array[String]): Unit = {

    logInfo("LogInfo")
    logWarning("LogWarning")
    logError("LogError")

  }

}

Use the corresponding class to inherit Logging class , The corresponding function can output the corresponding level Log ,LogInfo The log will be output to Stdout in ,LogWarning,LogError The log will be output to Stderr in , You can control the output log according to your own needs level.

2. Set up logs Level

Besides using Logging The output is different level Out of type log ,spark It also supports setting LogLevel.

    val sc = spark.sparkContext
    sc.setLogLevel("error")

setLogLevel The function allows us to define spark The level of the relevant log , This method will override any user-defined log level , Configure through the type of string , The following logs are supported level：

ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN . 

  def setLogLevel(logLevel: String) {
    // let's allow lowercase or mixed case too
    val upperCased = logLevel.toUpperCase(Locale.ROOT)
    require(SparkContext.VALID_LOG_LEVELS.contains(upperCased),
      s"Supplied level $logLevel did not match one of:" +
        s" ${SparkContext.VALID_LOG_LEVELS.mkString(",")}")
    Utils.setLogLevel(org.apache.log4j.Level.toLevel(upperCased))
  }

Four . summary

1. Log output

Logging Only applicable to Spark Task log , The following methods are used alone log Function does not output any logs .

  def main(args: Array[String]): Unit = {

    logInfo("LogInfo")
    logWarning("LogWarning")
    logError("LogError")

  }

2. Log type

Logging Use SLF4J,extends Logging After the interface , After the task is started, you will be prompted ：

 

3. The level of logging

Spark setLevel Method overrides other user-defined logLevel, So we need to pay attention to the coverage relationship .