Technical practice dry goods | from workflow to workflow

The author of this article ：

Baked Scallion Pancake , Guanyuan data front end Engineer , Landing team development specifications , Development quality and speed coexist , Committed to making it easier to use  ABI  product .

background

Let me give you a simple example , Because of work needs , You may have to extract data from the database every day , Then make a report , Finally, send it to relevant leaders in the form of email . But every leader may need to see different things , You need to filter and process the data before making the report , Then this repeated process every day , Can it be abstracted into a specific workflow , Each step is embodied as a function node , Then connect them in the form of tasks , adopt  DAG  The visual form of , Just run regularly every day ？ So , We will need a workflow to standardize and automate this process .

What is the workflow ？DAG  What is it again? ？ Now let's enter today's content .

Preface

This article will explain us  Universe（ One of the three product lines of Guanyuan , That is, the intelligent data development platform of Guanyuan data ） Workflow and  DAG  These two concepts , Then expand and introduce some other contents . The whole is divided into four parts ：

1.  Workflow in the development platform ;

2.  How to abstract implementation  DAG;

3.  Introduction to other workflows ;

4.  Based on workflow and  DAG  Summary and thinking of .

Let's start ～

One 、 workflow

Let's start with a brief introduction  Universe  Workflow in ：

Realize the dependencies of various tasks 、 Scheduling sequence design , Visualize the process 、 Low code design and management , Fast and highly available configuration of task nodes , To handle a series of data tasks ; And at the appointed time / Run after meeting the event dependency , Orderly call up all task nodes , Automatically complete the data processing process , It is simple and easy to use 、 High reliability, high scalability and other advantages .

According to this description , We can simply summarize two core capabilities of workflow ：

1.  Dispatch ;

2.  Configuration change （ node ）.

Here are the two core competencies .

1.1  Dispatch

Development platform support is based on  

Cron

  Expression timing scheduling and event scheduling based on input source data dependency , Among them, the timing scheduling adopts  

quartz

  Distributed scheduler .

It has the following features ：

• High availability

• adopt  DAG  Realize the visual arrangement of task nodes , No complex platform language learning costs , Task scheduling is out of the box ;

• Support configuration sequence scheduling 、 Successfully scheduled 、 Failure scheduling and other scheduling relationships , Flexibly adjust the scheduling strategy ;

• Support on time 、 By day 、 By week 、 Run the workflow on a monthly basis , The running results can be quickly pushed to nailing 、 Enterprise wechat and other platforms , A configuration , Continuously available .

• high reliability ： A lot of decentralization Master And many Worker Distributed architecture ,  Avoid single point of failure , Enhance system reliability .

• High scalability ： Can be based on  SDK  Develop custom task types and seamlessly connect processes .

1.1.1  Timing schedule

Support every day / Once a week / monthly / The form and interval length of each year and accurate to minutes （ when / branch ） To set the timing .

for instance ： I expect workflow every morning  7  When and at night  21  Time to run , Then I can choose   Every day  - 7 when /21 when  - 00  Fractional form , You can also set the minutes / Time interval to run .

1.1.2  Event scheduling

Generally, workflow has data source dependency , Like data sets / database , When all the dependent data sources are updated , Workflow can be automatically run once .

1.2  Configuration change

Based on a conventional configuration description , Produce an interactive  UI, Used to build the target object .

The purpose of scheduling is to run workflows , The operation of workflow depends on the configuration of different task nodes , Different configurations will inevitably have different  UI  Components , Then how can we use the known data structure to assemble a visual  UI  Well ？ The answer is configuration .

We are based on a configuration description （ object ） To read , Then render the corresponding components according to the configuration , At the same time, set the values of components into a general configuration object , Thus, from description to  UI  And then to the process of building the target object . Next, I will give three simple examples to illustrate the power and charm of configuration .

1.2.1  Basic ability

If we need to build a target object as follows ：

{
    name: '',
    description: '',
}


Then we will have the following configuration description ：

[
    {
        fieldName: 'name',
        label: ' name ',
        type: 'STRING',
        defaultValue: '',
    },
    {
        fieldName: 'description',
        label: ' describe ',
        type: 'TEXT',
        defaultValue: '',
    },
]


Generated  UI  as follows ：

1.2.2  Dynamic capability

Many times we need to dynamically implement a target object , What does that mean ？ Is to select different values of an attribute , Dynamically use an attribute to combine into a new target object , That corresponds to  UI  Above is to select different attribute values to display different components , That depends on our basic ability alone , Obviously can't do .

For example, I want to calculate the area of a graph , For example, a square needs the side length attribute , What the circle needs is the radius attribute , The target object and  UI  Will become ：

• When selecting a square


{
    shape: 'square',
    side: 8,
}


• When selecting a circle

{
    shape: 'circle',
    radius: 4,
}


You can see  

side

  and  

radius

  With  

shape

  And dynamic , Then we can simply modify the configuration description ：

[
    {
        fieldName: 'shape',
        label: ' graphics ',
        type: 'MODEL',
        model: {
            modelType: 'SELECT',
            labels: [ ' round ', ' Square ' ],
            values: [ 'circle', 'square' ],
        },
    },
    {
        fieldName: 'radius',
        label: ' radius ',
        type: 'NUMBER',
        dependsOnMap: {
            shape: [ 'circle' ],
        },
        defaultValue: 4,
    },
    {
        fieldName: 'side',
        label: ' Side length ',
        type: 'NUMBER',
        dependsOnMap: {
            shape: [ 'square' ],
        },
        defaultValue: 8,
    },
]


You can see , We just added  dependsOnMap  attribute , Then the internal rendering and the construction of objects are slightly adapted , You can choose different attributes to show the requirements of different components .

Here's a brief explanation  dependsOnMap  attribute , its  key  The value should be some  fieldName,value  Is an array , It is convenient to expand the case that multiple values are allowed , In this way, it can be based on  fieldName  To get  value, Compare with the value in the configuration , If so, show the component , The core logic is as follows ：

function isDependsOnMap (dependsOnMap, config) {
  const fieldNames = Object.keys(dependsOnMap || {})
  if (fieldNames.length === 0) return true
  return fieldNames.every(fieldName => {
    const values = dependsOnMap[fieldName] || []
    return values.indexOf(_get(config, fieldName)) > -1
  })
}

1.2.3  Complex ability

We may also have data transmission between components in our daily writing . Because of the object constraints described by the configuration , When we render each component, it is actually independent , There is no connection between components , Therefore, we only need to implement a data sharing layer at the top , Components 3 Put the data to be transferred on the data sharing layer , Components that need this data 1 Just get it directly .

The configuration is as follows ：

[
    {
        fieldName: 'fieldName1',
        label: ' Components 1',
        type: 'MODEL',
        model: {
            modelType: 'SELECT',
            labels: [ ' round ', ' Square ' ],
            values: [ 'circle', 'square' ],
            from: { fieldName: 'disabledFieldName' }, //  Component dependent 3 Settings in , Judge whether the current component needs  disabled
        },
    },
    {
        fieldName: 'fieldName2',
        label: ' Components 2',
        type: 'NUMBER',
    },
    {
        fieldName: 'fieldName3',
        label: ' Components 3',
        type: 'MODEL',
        model: {
            modelType: 'BOOLEAN',
            targetSharedFieldName: 'disabledFieldName', //  Set data fields to the data sharing layer
        },
    },
]


The key configuration attribute is the component 3 Inside  

model.targetSharedFieldName

  And the component 1 Inside  

model.from

, The two correspond to each other , The general implementation is as follows ：

const SharedContext = React.createContext({
  updateFieldValue: () => {}, //  Update fields  value
  getFieldValue: () => {}, //  Get field  value
})

function Comp1 ({ definition }) {
  const { targetSharedFieldName } = definition.model
  const { updateFieldValue } = useContext(SharedContext)

  useEffect(() => {
    updateFieldValue(targetSharedFieldName, value)
  }, [ deps ])
}

function Comp2 ({ definition }) {
  const { from } = definition.model
  const { getFieldValue } = useContext(SharedContext)
  const value = getFieldValue(from)
}


Finally, a complex configuration in the last development platform  UI  Moving graph , Feel the power and charm of configuration ：

1.2.4  Service capability,

When we need to build some array class target objects , The first thought must be to show it in the form of a list  UI, So we designed some service type components , Only responsible for rendering the list , But the components of each list depend on the type of array elements .

For example, we need such an array class target object ：

{
    list: [
        { name: 'a', age: 12 },
        { name: 'b', age: 18 },
    ],
}


The corresponding configuration description can be written like this ：

[
    {
        fieldName: 'list',
        label: ' list ',
        type: 'MODEL',
        model: {
            modelType: 'LIST',
            definitions: [
                {
                    fieldName: 'name',
                    label: ' name ',
                    type: 'STRING',
                },
                {
                    fieldName: 'age',
                    label: ' Age ',
                    type: 'NUMBER',
                },
            ],
        },
    },
]


And the corresponding  UI  as follows ：

This  

LIST

  A component is a component of a service type , Display the array object in the form of a list .

1.2.5  Registration ability

Built in components may not fully meet the needs of configuration , Because configuration is just a convention , But draw by building objects  UI  It belongs to liberalization , The presentation forms vary greatly , For this purpose, we provide a registration mechanism . Users can customize the type of registered components , To draw the corresponding target object .

1.3  summary

Based on such excellent configuration capability, it should be abstracted , So it is also used in  BI  On the custom chart . Based on this , We wrote a library called  

Lego

, Just as the name means , We expect to build some dedicated for configuration  UI  Time is as simple as building blocks , Agree on a description （ Interface ）, Just put it together .

After introducing the workflow , We also need a visual interface to describe this process , that  DAG  It is undoubtedly a good form of display .

Two 、DAG

DAG  Full name  

Directed Acyclic Graph

, Chinese is

Directed acyclic graph

. It consists of finite vertices and “ There is a directional side ” form , Starting from any vertex , Passing through several directed edges , Can't return to the vertex . An example is shown below ：

A simple understanding of  DAG  The concept of , How to abstract a simple and easy-to-use one according to the working flow field of the development platform  DAG  Well ？ First, organize and draw a  DAG  What information and status are needed ：

• Node information （nodes）

• Node location （location）

• Connection information （edges）

• Edit and read-only status

The first three points are easy to understand , It should be drawing  DAG  Three essential elements , Explain the fourth point , Because the workflow of the development platform has the concept of online and offline , Go online after the development , No modification allowed , As a specification in the development of data warehouse , Then our workflow exists offline editable , Online read-only distinction .

Start with editing and read-only , We can  DAG  It is divided into  Playground  and  Renderer  Two parts , And can be used independently .Playground  Corresponding editing status ,Renderer  Corresponding to read-only status .Playground  You should generate the drawing information in the editing state in real time , and  Renderer  Is responsible for rendering in real time according to the rendering information . Then let's sort out what abilities we should have in editing and read-only States ：

• Playground
• Node drag
• Connection addition and deletion
• newly added / Replication node
• Select the node in the box to copy in batch / Delete
• Automatic layout / Cancel the operation

• Renderer
• Zoom in and out
• Drag the canvas
• Click

Think about it again , our  DAG  What other abilities should you have ？ Here I briefly list the following points in combination with the use of the development platform ：

• Provide style configuration （ Such as node size / Line width, etc ）

• Support width height adaptation

• Custom draw nodes and lines

• Other rendering capabilities are enhanced （ Such as annotation function , Itself does not belong to  DAG  The function of , Instead, consider the implementation of extended functions ）

thus , our  DAG  There is probably a complete structure and implementation direction ：

|- ConfigContext              ---  Configuration layer
   |- Playground            ---  Editing layer
   |- ResponsiveProvider ---  Adaptive wide high-rise （ Optional ）
      |- Renderer        ---  Read-only layer , Show only
         |- Nodes        ---  node
      |- Edges        ---  attachment


The usage is like this ：

2.1  Read only use

<ConfigContext.Provider&nbsp;value={{&nbsp;node:&nbsp;{&nbsp;width:&nbsp;56,&nbsp;height:&nbsp;56&nbsp;}&nbsp;}}>
&nbsp;<ResponsiveProvider>
&nbsp;&nbsp;<Renderer&nbsp;nodes={nodes}&nbsp;location={location}&nbsp;edges={edges}&nbsp;/>
&nbsp;</ResponsiveProvider>
</ConfigContext.Provider>


2.2  Editors use

<ConfigContext.Provider&nbsp;value={{&nbsp;node:&nbsp;{&nbsp;width:&nbsp;60,&nbsp;height:&nbsp;60&nbsp;}&nbsp;}}>
&nbsp;<Playground&nbsp;nodes={nodes}&nbsp;location={location}&nbsp;edges={edges}&nbsp;/>
</ConfigContext.Provider>


2.3  Custom node and connection use

<ConfigContext.Provider&nbsp;value={{&nbsp;node:&nbsp;{&nbsp;width:&nbsp;56,&nbsp;height:&nbsp;56&nbsp;}&nbsp;}}>
&nbsp;<Renderer&nbsp;nodes={nodes}&nbsp;location={location}&nbsp;edges={edges}>
&nbsp;&nbsp;<Nodes>
&nbsp;&nbsp;&nbsp;{(props)&nbsp;=>&nbsp;<CustomNode&nbsp;/>}
&nbsp;&nbsp;</Nodes>
&nbsp;&nbsp;<Edges>
&nbsp;&nbsp;&nbsp;{(props)&nbsp;=>&nbsp;<CustomEdge&nbsp;/>}
&nbsp;&nbsp;</Edges>
&nbsp;</Renderer>
</ConfigContext.Provider>


2.4  Bottom layer drawing

Here we chose  svg, Because  svg  Strong enough in drawing , Support  css  To customize the style , It also facilitates the binding of events . With this direction , We can determine which tags correspond to the following elements in turn ：

• canvas ：svg

• node ：
  foreignObject

• attachment ：path

Draw a structure like this ：

The canvas is zoomed in and out and moved through  viewBox  Property settings

according to  html  structure , Wiring is how we need to care about generation , Here we mainly calculate a quadratic Bessel curve by the position of two nodes （Quadratic Curves） Get a perfect curve with inverse symmetry , as follows ：

Here we say that the next quadratic Bessel curve is  path  How to implement in Tags . First, draw the information that requires three points , The following diagram ：

Second, because our curve is inversely symmetric , In fact, you only need to draw half of it , This half is a quadratic Bessel curve , Then the position of the three points is easy to confirm , as follows ：

among  P0  As a starting point ,P4  End point , For ease of calculation ,P1  Corresponding  1/4  Horizontal spacing , The height is the same as the starting point ,P2  It is  1/2  Horizontal and vertical spacing , And then by calculating  path  The path of  d  attribute , Separate  3  One point can be substituted ：

d = M P0x P0y Q P1x P1y P2x P2y T P4x P4y

So we get a complete curve , It consists of two quadratic Bessel curves .

2.5  Layout

With nodes and connections , Layout is also a very important part , Manual dragging is obviously sometimes not neat enough , If there is an automatic layout algorithm , Then it will be much easier , Here we chose   

dagre

   As a calculation tool for automatic layout . There are three main algorithms ：

function rank(g) {
 switch(g.graph().ranker) {
  case "network-simplex": networkSimplexRanker(g); break;
  case "tight-tree": tightTreeRanker(g); break;
  case "longest-path": longestPathRanker(g); break;
  default: networkSimplexRanker(g);
 }
}


network-simplex

  and  

tight-tree

  Similar in layout , They all realize the layout in a compact way ,

longest-path

  The difference is that if there are multiple end nodes , Ensure that these nodes are aligned from top to bottom , Rather than layout nearby , Here's the picture ：

• network-simplex
    and  
  tight-tree

• longest-path

3、 ... and 、 Other workflows

How to use these workflows will not be described in detail here , I will just learn from their ideas on Workflow drawing and Application .

3.1 

n8n

The workflow automation platform that doesn't box you in, that you never outgrow.

n8n  Support event driven （ Generally used by third parties  hooks/ Local file modification and monitoring ） and   

cron

   Expression scheduling workflow , At the same time, the dependencies between nodes are determined in the order of data transmission . It is very similar to our workflow , It's just that our workflow depends on node task scheduling , Instead of relying on data .

3.1.1  application

What is it suitable for ？ Here's the picture ：

If you are an open source enthusiast , I hope to know my  Github Repo  By  star  Or remove  star  The first time I know the news , Then you can use it  github  Open  star hook, And then through  slack  Send yourself a message . Through the integration of third-party platforms , It can connect all kinds of unrelated applications , Develop convenient workflow .

3.1.2  summary

n8n  It's integrated now  200+  Application , It covers most mainstream applications . But some domestic applications are still missing , Like nails / Enterprise wechat, etc , So it successfully supports custom development nodes , If you are interested, click  

here

 . As a whole ,n8n  It is more like an integrated application workflow , Of course, it also supports some local functions , Such as reading and writing files / Use git Operation etc. . It can take some common operations that we need to write little by little in our daily work or development , Integrate into one workflow , Convenient daily life .

3.1.3  reference

From its workflow design , Maybe some points can be used for reference ：

• When configuring nodes, you can see what the output data of the previous node is , It is convenient to configure in the current step

• The node configuration can be executed immediately , See the corresponding output data

• There are some data visualization enhancements on the connection , For example, how many lines of output data

• You can directly click Add to select the post node , Omit some connection operations

3.1.4  other

Later, I tried whether the lower node can form a ring , It turns out that you can , Trapped in an infinite loop , The application is stuck , as follows ：

Unlimited data growth , Run infinite loop .

3.2 

Orange

Open source machine learning and data visualization. Build data analysis workflows visually, with a large, diverse toolbox.

3.2.1  application

Orange  It's more suitable for  ML  Related work , It's a bit like our  AI Flow, But the data flow / Data exploration / Chart analysis and other functions are integrated in it , You don't need to go to other pages to configure, process and view separately , View the data in the form of workflow 、 Processing and analysis . Simple last figure ：

There is an interesting little dot , Its connection supports the transmission of full data or selected data , Here's the picture ：

Then the way of data transmission will be reflected in the connection . On the connection , It shows the endpoint in the form of an arc （ I guess the arc is used to increase the connection area of nodes , At the same time, it is also suitable for circular nodes ）, If there is a connection, it is a solid line , If there is no connection, it is a dotted line , Very friendly for the display of status .

3.2.2  summary

Orange  The function integration is very powerful , In addition to basic data conversion , And charts / Model / Evaluation and other functions , It's very suitable  AI  Direction of data analysis .

Four 、 Summary and reflection

The concept of workflow has been put forward for a long time , It is the abstraction of business rules between processes and their operation steps 、 Summarize and describe . The emergence of workflow makes our process standardized , The steps become clear . The workflow above data development avoids a series of repeated operations , At the same time  DAG  To show in the form of , Make the process more intuitive . Of course  DAG  It is not necessarily used in scheduling systems with sequence restrictions , It can also be used in other forms , For example, on the display of causality such as data kinship , It can also be used to display the family map , One more level , It can even be used in data processing networks , Data flows from one point to another , It does not necessarily need to be displayed in the form of visualization , Just need this concept .

4.1  possibility

In fact, our workflow has powerful scheduling and configuration functions , But limited by limited function nodes , If we can support custom configuration nodes , It allows users to have more imagination at the level of data development , Instead of being limited to these existing nodes to do workflow development .

Reference material

[1] 

https://dolphinscheduler.apache.org/zh-cn/docs/latest/user_doc/about/introduction.html

[2]

 

https://en.wikipedia.org/wiki/Workflow

[3]

 https://en.wikipedia.org/wiki/Directed_acyclic_graph

[4] 

https://github.com/biolab/orange3

[5] 

https://github.com/n8n-io/n8n