使用Prometheus监控Spark任务

BackGroud

目前spark支持Graphite打点，但是Graphite默认会为每个打点生成对应的文件，直接只有两个点数据，graphite生成的文件也有200多k，对Graphite造成很大的压力。所以考虑使用Prometheus来代替Graphite。

Prometheus的打点默认是pull模式，和Graphite不同，而Spark3.0 新加的打点方式也是K8S友好的。所以还是考虑 使用Prometheus结合Graphite Exporter 的方式。

Graphite Exporter 新的配置文件

mappings:
- match: '*.applicationMaster.*'
  name: spark_app_application_master
  labels:
    application: $1
    qty: $2
- match: '*.*.executor.filesystem.*.*'
  name: spark_app_filesystem_usage
  labels:
    application: $1
    executor_id: $2
    fs_type: $3
    qty: $4
- match: '*.*.jvm.*.*'
  name: spark_app_jvm_memory_usage
  labels:
    application: $1
    executor_id: $2
    mem_type: $3
    qty: $4
- match: '*.*.ExecutorMetrics.*'
  name: spark_app_executor_jvm_metrics
  labels:
    application: $1
    executor_id: $2
    mem_type: $3
- match: '*.*.executor.jvmGCTime.count'
  name: spark_app_jvm_gcTime_count
  labels:
    application: $1
    executor_id: $2
- match: '*.*.jvm.pools.*.*'
  name: spark_app_jvm_memory_pools
  labels:
    application: $1
    executor_id: $2
    mem_type: $3
    qty: $4
- match: '*.*.ExternalShuffle.shuffle-client.*'
  name: spark_app_shuffle_client
  labels:
    application: $1
    executor_id: $2
    qty: $3
- match: '*.*.executor.*.count'
  name: spark_app_executor_task_metrics
  labels:
    application: $1
    executor_id: $2
    qty: $3
- match: '*.*.executor.threadpool.*'
  name: spark_app_executor_tasks
  labels:
    application: $1
    executor_id: $2
    qty: $3
- match: '*.*.BlockManager.*.*'
  name: spark_app_block_manager
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4
- match: '*.*.DAGScheduler.*.*'
  name: spark_app_dag_scheduler
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4
- match: '*.*.CodeGenerator.*.*'
  name: spark_app_code_generator
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4
- match: '*.*.HiveExternalCatalog.*.*'
  name: spark_app_hive_external_catalog
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4
- match: '*.*.*.StreamingMetrics.*.*'
  name: spark_app_app_streaming_metrics
  labels:
    application: $1
    executor_id: $2
    app_name: $3
    type: $4
    qty: $5

Spark配置

Spark配置仍然采用Graphite打点配置，只需要将端口改为 9109即可

• 参考文档： https://blog.csdn.net/An1090239782/article/details/104945390