Shangsilicon Valley real-time data warehouse project (Alibaba cloud real-time data warehouse)

p01 Project brief introduction

p02 Project requirements analysis

p03 Alibaba cloud technology framework

p04 Technology selection

P05 System architecture design

p06 The business process

p07 E-commerce meter structure

p08 rds Buy

p09 rds Configuration and connection

Set the whitelist

Apply for an Internet address

Create account

p10 Create business databases and tables

perform sql Script

p11 establish DataHub And Project

establish dataHub

You can create topic, There is no need to create , We use it directly DTS Synchronous data , Automatically created topic

p12 Synchronize the fact table to datahub

Get into , The synchronization link must be configured first

Synchronization settings

next step





RDS Prepare the data in ,datahub There will also be data in

p13 Synchronize dimension tables to mysql

Create a dimension table Library

Continue to use DTS Sync

from rds Sync to another rds

After opening, there are 2 There are two examples

Synchronize link settings , This is the test, so it is the same instance server , The actual work is 2 platform


Waiting becomes Syncing , That's all right.

View , Reduce subsequent cascading


Created 2 Individual view View of provinces and regions Product category view

p14 Real time data warehouse layering

p15 Create a cluster and create a project

Create project

p16 Create workspaces and business development platforms

Create a workspace dataworks

p17 Establish tasks and flinkSQL Development Manual

You can use component drag and drop , But we don't use this method , We use sql Pattern


Help document
The data source table is ods layer , The data result table is dwd layer

p18 ODS To DWD Layer business process

Be careful 1： Because the data are DTS Push it over , The pushed fields are added in front of the original table fields
One. dts_ Prefix .
Be careful 2：dts_utc_timestamp、dts_operation_flag、dts_after_flag yes DataHub The words provided by the framework
paragraph .dts_operation_flag=‘U’ To modify 、 ‘I’ Indicates insertion 、‘D’ Said to delete ,dts_after_flag=‘Y’
Be careful 3：dts_order_status=1 Indicates placing an order ,dts_order_status=2 Means paid

order_info Data source table
order_detail Data source table

dwd_paid_order_detail Data table