The number of warehouse 】 data quality

   Today is 618, mid-year sale.In previous years, I will participate in 618 and Double 11 to buy something, but I have no desire to participate this year.I don’t know if it’s because I’ve been quarantined in Shanghai for too long. I feel like I’ve been isolated from winter to summer. I feel that clothes, shoes, etc. are not so necessary. It’s true to stock up on vegetables and food... I heard that JD.com’s summer interns are allI became a daily intern. After June 18, I started to lay off employees. I don’t know if it’s true or not. The economy is dying, and the Internet is also not good.

   The above are all digressions.When I was looking for an internship before, I was asked a question on the second side: Do you understand the quality of data?There was no answer at the time.Now that I have come into contact with the actual work, I found that there is a special data quality management platform, which is probably to monitor data and tasks from various angles. I will talk about it today.

1.Definition

Data quality management is the quality control over the entire data life cycle of data generation, processing and consumption. The specific dimensions include:

• Accuracy
• Integrity
• Consistency
• Timeliness
• Validity
• Unique

Data production stage: data is missing or data is inaccurate due to system abnormalities or system process problems.

Data processing and consumption stage: During the processing, can the integrity of the data extracted be consistent with the data generated by the system, and whether the dataWhether the output is timely and other quality issues.

2. Goal

A set of quality evaluation system is established for the tables in the data warehouse system, which evaluates the data integrity, accuracy, consistency, validity, timeliness, uniqueness and other dimensions to guide the construction of logarithmic tablesand a reasonable assessment of the accuracy of the logarithmic table.

3. Implementation

In simple terms, it is to monitor some indicators from the whole link and multiple perspectives through a series of rules, form a quality report, and evaluate the quality.Here are some examples of metrics to monitor:

• Table: primary key, data volume (number of rows, disk size);
• Field: Proportion of empty rows, repeated rows, fixed rows, enumeration, enumeration range, length;
• SLA: The latest output time promised to the outside world (alarm when the task is delayed);

Review should be carried out every week to record accidents, broken lines, number of alarms, alarm rate, and number of nights, analyze the reasons, and optimize the task.

Welcome to click here to follow the official account.