Using Prometheus to monitor spark tasks

BackGroud

at present spark Support Graphite Dot , however Graphite By default, the corresponding file will be generated for each dot , There are only two point data directly ,graphite The generated files also have 200 many k, Yes Graphite Cause a lot of pressure . So consider using Prometheus Instead of Graphite.

Prometheus The default is pull Pattern , and Graphite Different , and Spark3.0 The new management method is also K8S Amicable . So consider Use Prometheus combination Graphite Exporter The way .

Graphite Exporter New profile

mappings:
- match: '*.applicationMaster.*'
  name: spark_app_application_master
  labels:
    application: $1
    qty: $2
- match: '*.*.executor.filesystem.*.*'
  name: spark_app_filesystem_usage
  labels:
    application: $1
    executor_id: $2
    fs_type: $3
    qty: $4
- match: '*.*.jvm.*.*'
  name: spark_app_jvm_memory_usage
  labels:
    application: $1
    executor_id: $2
    mem_type: $3
    qty: $4
- match: '*.*.ExecutorMetrics.*'
  name: spark_app_executor_jvm_metrics
  labels:
    application: $1
    executor_id: $2
    mem_type: $3
- match: '*.*.executor.jvmGCTime.count'
  name: spark_app_jvm_gcTime_count
  labels:
    application: $1
    executor_id: $2
- match: '*.*.jvm.pools.*.*'
  name: spark_app_jvm_memory_pools
  labels:
    application: $1
    executor_id: $2
    mem_type: $3
    qty: $4
- match: '*.*.ExternalShuffle.shuffle-client.*'
  name: spark_app_shuffle_client
  labels:
    application: $1
    executor_id: $2
    qty: $3
- match: '*.*.executor.*.count'
  name: spark_app_executor_task_metrics
  labels:
    application: $1
    executor_id: $2
    qty: $3
- match: '*.*.executor.threadpool.*'
  name: spark_app_executor_tasks
  labels:
    application: $1
    executor_id: $2
    qty: $3
- match: '*.*.BlockManager.*.*'
  name: spark_app_block_manager
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4
- match: '*.*.DAGScheduler.*.*'
  name: spark_app_dag_scheduler
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4
- match: '*.*.CodeGenerator.*.*'
  name: spark_app_code_generator
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4
- match: '*.*.HiveExternalCatalog.*.*'
  name: spark_app_hive_external_catalog
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4
- match: '*.*.*.StreamingMetrics.*.*'
  name: spark_app_app_streaming_metrics
  labels:
    application: $1
    executor_id: $2
    app_name: $3
    type: $4
    qty: $5

Spark To configure

Spark Configuration is still used Graphite Management configuration , Just change the port to 9109 that will do

• Reference documents ： https://blog.csdn.net/An1090239782/article/details/104945390