Exporter for Storage Area Network (SAN)

Last update: Dec 16, 2022

Related tags

Overview

SAN Exporter

Prometheus exporter for Storage Area Network (SAN).

We all know that each SAN Storage vendor has their own glossary of terms, health/performance metrics and monitoring tool.

But from operator view,

We normally focus on some main metrics which are similar on different storage platform.
We are not only monitoring SAN storage but also other devices and services at multi-layer (application, virtual Machine, hypervisor, operating system and physical).

That's why we build this to have an unified monitoring/alerting solution with Prometheus and Alermanager.

Architecture overview

Features

There are some main features you might want to know, for others, please see example configuration.

Enable/disable optinal metrics for each backend
Enable/disable backend
Backend will automatically stop collecting data from SAN system after timeout seconds from last request of client. With this feature, we can deploy two instances as Active/Passive mode for high availability.

Note: Backend may not respond metrics in the first interval while collecting, calculating and caching metrics.

Quick start

Start a dummy driver with Docker

$ git clone [email protected]:vCloud-DFTBA/san_exporter.git
$ cd san_exporter/
$ cp examples/dummy_config.yml config.yml
# docker run --rm -p 8888:8888 -v $(pwd)/config.yml:/san-exporter/config.yml --name san-exporter daikk115/san-exporter:0.1.0

See the result at http://localhost:8888/dummy_backend

Start a dummy driver manually

$ git clone [email protected]:vCloud-DFTBA/san_exporter.git
$ cd san_exporter/
$ cp examples/dummy_config.yml config.yml
$ sudo apt-get install libxml2-dev libxslt1-dev python3.7-dev
$ pip3 install -r requirements.txt
$ python3.7 manage.py

See the result at http://localhost:8888/dummy_backend

Start with a HPE 3PAR api simulator

Deployment

Create configuration file

# mkdir /root/san-exporter
# cp /path/to/san_exporter/examples/config.yml.sample /root/san-exporter/config.yml

Update /root/san-exporter/config.yml for corresponding to SAN storage

Run new container

# docker volume create san-exporter
# docker run -d -p 8888:8888 -v san-exporter:/var/log/ -v /root/san-exporter/config.yml:/san-exporter/config.yml --name san-exporter daikk115/san-exporter:latest

Supported Drivers

Matrix of driver's generic metrics

	Capacity all	Capacity pool	IOPS/Throuhgput pool	Latency pool	IOPS/Throughput node	Latency node	CPU node	RAM node	IOPS/Throughput LUN	Latency LUN	IOPS/Throughput disk	Latency disk	IOPS/Throughput port	Latency port	Alert
HPMSA	X	X	X	X		X							X	X	X
DellUnity	X	X			X				X	X	X	X	X	X	X
HitachiG700	X	X													X
HPE3Par	X	X	X	X			X						X	X	X
NetApp	X	X	X	X							X	X
SC8000	X	X			X	X	X	X			X	X	X	X	X
V7k	X	X	X	X			X						X

Connection port requirements
- For some SAN system, we collect metrics over SP API but some others, we collect metrics dirrectly from controller API.
- In some special cases, we collect alerts over SSH.

SAN System	Service Processor	Connection Port
HPMSA	NO	443
Dell Unity	NO	443
Hitachi G700	YES	23451
IBM V7000	NO	#TODO
IBM V5000	NO	#TODO
HPE 3PAR	YES	#TODO
NetApp ONTAP	NO	443
SC8000	NO	3033

Metrics

All metrics are prefixed with "san_" and has at least 2 labels: backend_name and san_ip

Info metrics:

Metrics name	Type	Help
san_storage_info	gauge	Basic information: serial, version, ...

Controller metrics:

Metrics name	Type	Help
san_totalNodes	gauge	Total nodes
san_masterNodes	gauge	Master nodes
san_onlineNodes	gauge	Online nodes
san_compress_support	gauge	Compress support, 1 = Yes, 0 = No
san_thin_provision_support	gauge	Thin provision support, 1 = Yes, 0 = No
san_system_reporter_support	gauge	System reporter support, 1 = Yes, 0 = No
san_qos_support	gauge	QoS support, 1 = Yes, 0 = No
san_totalCapacityMiB	gauge	Total system capacity in MiB
san_allocatedCapacityMiB	gauge	Total allocated capacity in MiB
san_freeCapacityMiB	gauge	Total free capacity in MiB
san_cpu_system_utilization	gauge	The average percentage of time that the processors on nodes are busy doing system I/O tasks
san_cpu_compression_utilization	gauge	The approximate percentage of time that the processor core was busy with data compression tasks
san_cpu_total	gauge	The cpus spent in each mode

Pool metrics:

Metrics name	Type	Help
san_pool_totalLUNs	gauge	Total LUNs (or Volumes)
san_pool_total_capacity_mib	gauge	Total capacity of pool in MiB
san_pool_free_capacity_mib	gauge	Free of pool in MiB
san_pool_provisioned_capacity_mib	gauge	Provisioned of pool in MiB
san_pool_number_read_io	gauge	Read I/O Rate - ops/s
san_pool_number_write_io	gauge	Write I/O Rate - ops/s
san_pool_read_cache_hit	gauge	Read Cache Hits - %
san_pool_write_cache_hit	gauge	Write Cache Hits - %
san_pool_read_kb gauge	gauge	Read Data Rate - KiB/s
san_pool_write_kb	gauge	Write Data Rate - KiB/s
san_pool_read_service_time_ms	gauge	Read Response Time - ms/op
san_pool_write_service_time_ms	gauge	Write Response Time - ms/op
san_pool_read_IOSize_kb	gauge	Read Transfer Size - KiB/op
san_pool_write_IOSize_kb	gauge	Write Transfer Size - KiB/op
san_pool_queue_length	gauge	Queue length of pool

Port metrics:

Metrics name	Type	Help
san_port_number_read_io	gauge	Port Read I/O Rate - ops/s
san_port_number_write_io	gauge	Port Write I/O Rate - ops/s
san_port_write_kb	gauge	Port Write Data Rate - KiB/s
san_port_read_kb	gauge	Port Read Data Rate - KiB/s
san_port_write_IOSize_kb	gauge	Port Write Transfer Size - KiB/op
san_port_read_IOSize_kb	gauge	Port Read Transfer Size - KiB/op
san_port_queue_length	gauge	Queue length of port

For more information about specific metrics of SANs, see Specific SAN Metrics

Integrate with Prometheus, Alertmanager and Grafana

Some grafana images:

You might also like...

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

Pattern Pattern is a web mining module for Python. It has tools for: Data Mining: web services (Google, Twitter, Wikipedia), web crawler, HTML DOM par

Computational Linguistics Research Group

8.4k Jan 3, 2023

Neurolab is a simple and powerful Neural Network Library for Python

Neurolab Neurolab is a simple and powerful Neural Network Library for Python. Contains based neural networks, train algorithms and flexible framework

152 Dec 6, 2022

A scikit-learn compatible neural network library that wraps PyTorch

A scikit-learn compatible neural network library that wraps PyTorch. Resources Documentation Source Code Examples To see more elaborate examples, look

4.9k Dec 31, 2022

Visualizer for neural network, deep learning, and machine learning models

Netron is a viewer for neural network, deep learning and machine learning models. Netron supports ONNX (.onnx, .pb, .pbtxt), Keras (.h5, .keras), Tens

21k Jan 6, 2023

Graph neural network message passing reframed as a Transformer with local attention

Adjacent Attention Network An implementation of a simple transformer that is equivalent to graph neural network where the message passing is done with

49 Dec 28, 2022

data/code repository of "C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer"

C2F-FWN data/code repository of "C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer" (https://arxiv.org/abs/

46 Dec 14, 2022

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Deep Daze mist over green hills shattered plates on the grass cosmic love and attention a time traveler in the crowd life during the plague meditative

4.4k Jan 3, 2023

End-to-End Object Detection with Fully Convolutional Network

This project provides an implementation for "End-to-End Object Detection with Fully Convolutional Network" on PyTorch.

472 Dec 22, 2022

TensorFlow-based neural network library

Sonnet Documentation | Examples Sonnet is a library built on top of TensorFlow 2 designed to provide simple, composable abstractions for machine learn

9.5k Jan 7, 2023

Comments

Support purestorage please!

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like A clear and concise description of what you want to happen.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here. Can you support purestorage?

opened by wanbeepeto 0

Releases(v0.8.0)

v0.8.0(Aug 17, 2021)
Release notes:

Add Dell Unnity driver

Add Hitachi G700 driver

Add HPE 3PAR driver

Add HPMSA driver

Add NetApp ONTAP driver

Add Dell SC800 driver

Add IBM V7000 driver

Docker image: daikk115/san-exporter:0.8.0

Source code(tar.gz)
Source code(zip)
v0.1.0(Aug 15, 2021)
Release notes:

Add dummy driver

Docker image: daikk115/san-exporter:0.1.0

Source code(tar.gz)
Source code(zip)

Exporter for Storage Area Network (SAN)

Related tags

Overview

SAN Exporter

Architecture overview

Features

Quick start

Deployment

Supported Drivers

Metrics

Info metrics:

Controller metrics:

Pool metrics:

Port metrics:

Integrate with Prometheus, Alertmanager and Grafana

You might also like...

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

Neurolab is a simple and powerful Neural Network Library for Python

A scikit-learn compatible neural network library that wraps PyTorch

Visualizer for neural network, deep learning, and machine learning models

Graph neural network message passing reframed as a Transformer with local attention

data/code repository of "C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer"

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

End-to-End Object Detection with Fully Convolutional Network

TensorFlow-based neural network library

Comments

Support purestorage please!

Releases(v0.8.0)

v0.8.0(Aug 17, 2021)

v0.1.0(Aug 15, 2021)

Owner

vCloud

This is an unofficial implementation of the paper “Student-Teacher Feature Pyramid Matching for Unsupervised Anomaly Detection”.

Benchmark for Answering Existential First Order Queries with Single Free Variable

Deep Latent Force Models

Our implementation used for the MICCAI 2021 FLARE Challenge titled 'Efficient Multi-Organ Segmentation Using SpatialConfiguartion-Net with Low GPU Memory Requirements'.

🔎 Super-scale your images and run experiments with Residual Dense and Adversarial Networks.

Pytorch Implementation of rpautrat/SuperPoint

A blender add-on that automatically re-aligns wrong axis objects.

領域を指定し、キーを入力することで画像を保存するツールです。クラス分類用のデータセット作成を想定しています。

Degree-Quant: Quantization-Aware Training for Graph Neural Networks.

Collect super-resolution related papers, data, repositories

Gapmm2: gapped alignment using minimap2 (align transcripts to genome)

[ICLR 2021] "CPT: Efficient Deep Neural Network Training via Cyclic Precision" by Yonggan Fu, Han Guo, Meng Li, Xin Yang, Yining Ding, Vikas Chandra, Yingyan Lin

Official Code Implementation of the paper : XAI for Transformers: Better Explanations through Conservative Propagation

A PyTorch Implementation of Single Shot Scale-invariant Face Detector.

Semantic Segmentation with Pytorch-Lightning

PGPortfolio: Policy Gradient Portfolio, the source code of "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem"(https://arxiv.org/pdf/1706.10059.pdf).

ManipulaTHOR, a framework that facilitates visual manipulation of objects using a robotic arm

NeRD: Neural Reflectance Decomposition from Image Collections

A Python module for the generation and training of an entry-level feedforward neural network.

Resources complimenting the Machine Learning Course led in the Faculty of mathematics and informatics part of Sofia University.