Argo workflows source code analysis

via：https://goframe.org/pages/viewpage.action?pageId=44462821

This paper mainly focuses on Argo Workflow At the heart of Feature And the source code implementation of the core execution process ,Feature Please check the implementation details of Argo Workflow Have a deeper understanding of the source code .

One 、 Knowledge comb

because Argo There are many concepts and contents in itself , I will sort out the key knowledge points through mind mapping , As a preliminary knowledge ：

https://whimsical.com/[email protected]YocBL

Some basic concepts and functions will not be introduced here , You can refer to the previous article Argo Introduction article ：Argo Workflow Introduce

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-JlT6FGDe-1657099905843)(https://goframe.org/download/attachments/44462821/Kubernetes%20Argo%20Framework%20%282%29.png?version=1&modificationDate=1650791794971&api=v2)]

Two 、 Full of curiosity

In order to learn better Argo Workflow, Here are a few questions , We explore with questions Argo The effect may be better ：

1. Workflow What are the core components , What are their roles ？
2. Workflow How to realize context transfer of process data ？
3. Workflow How to realize the process management logic of ？
4. Workflow Where are the templates and status data stored ？

Next, let's sort it out Argo Workflow And some key logic , Then we come back to answer these questions .

3、 ... and 、 Engineering structure

Argo Workflow The whole project uses classic kubebuilder Built , So most directory structures and kubebuilder bring into correspondence with . About kubebuilder Please refer to ：https://cloudnative.to/kubebuilder/

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-kiHSt5x8-1657099905844)(https://goframe.org/download/attachments/44462821/image2021-7-3_15-15-13.png?version=1&modificationDate=1650791815357&api=v2)]

Directory name	Responsibilities and instructions
api	Swagger API Definition Json File storage directory , Mainly for Argo Server UI Use .
cmd	Import source code file
- argo	argo CLI
- argoexec	argoexec container image command
- workflow-controller	Kubernetes CRD Controller
community	Introduction to open source community , For now, there's just one README.MD
config	Argo Workflow Controller Configuration objects and related methods
docs	Argo Workflow Related introduction documents of , Consistent with the official website documents
errors	Encapsulate third parties github.com/pkg/errors Components ,argo Workflow Error management component used internally
examples	Rich usage examples , Mainly yaml file
hack	Scripts and tool files used in the project
manifests	Argo Installation configuration file , They are all yaml file , Use kustomize Tool management , About kustomize Please refer to ：https://kubernetes.io/zh/docs/tasks/manage-kubernetes-objects/kustomization/
persist	Argo Database persistence encapsulation component , Support MySQL/PostgreSQL Two databases . Persistence is mainly aimed at Archived Workflow Object storage , contain Workflow Definition of and status data .
pkg	Argo Workflow The foreign API Definition 、 Structure definition 、 Client definition , Mainly for external services 、 Client side usage .
- apiclient	Argo Server foreign API Related definitions 、 Client component .
- workflow	Argo Workflow Controller Definition of related structures .
- client	Argo Workflow Controller And Kubernetes The interaction of Client/Informer/Lister Definition .
server	Argo Server modular .
test	Unit test file .
ui	Argo Server The front end of the UI NodeJS The source code file , Use Yarn Package management .
util	Toolkit module encapsulated by the project
workflow	Argo Workflow Logical encapsulation of core functions

Four 、Workflow Controller

Argo The core and the most complex is Workflow Controller The implementation of the .Argo Workflow Controller The main responsibilities of CRD The implementation of the , as well as Pod Creation of . because Argo It's using Kubernetes CRD Design , Therefore, the overall architecture and process control are Kubernetes Informer Realization , For relevant background knowledge, please refer to the previous two articles ：Kubernetes Informer And client-go Information 、Kubernetes CRD, Controller, Operator.

1、 Basic framework

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-tMtDozAA-1657099905845)(https://goframe.org/download/attachments/44462821/architecture.jpeg?version=1&modificationDate=1650791846273&api=v2)]

2、 Important design

Argo Workflow Controller There are some components , I personally think the more important design to share with you .

1） Definition is separated from state

This is actually Kubernetes The standard design of , namely CRD Realization The object should contain Spec And Status Property object , among Spec Corresponding CR The definition of , and Status Corresponding CR Business status information .Spec Created and modified by the business client , Generally, it will not be updated after creation , stay Informer Controller Only... Can be read in the processing flow . and Status yes Informer Controller Fields that change constantly according to the needs of business scenarios .

2） Separation of definition and data

Argo Workflow Template Only process and variable definitions should be included , Variable data is generated by the runtime , For example, through Template Run time generation to the terminal or Artifact, Re pass Outputs Is defined by others Template quote . One Node After successful execution , Its output data will be saved to Template.Status Field （Kubernetes etcd） perhaps Artifact in , The return execution will not generate repeatedly . One Node After the execution fails , If re executed, the dependent data will be pulled again . This design of separation of definition and data makes Workflow Template It can be designed in advance , Even through UI Generated by dragging .

3） Global and local variables

stay Argo Workflow Controller There are two kinds of variables in the interior : One is Workflow Globally effective variables (globalParams), One is the current Template Effective local variables (localParams). The global variables also include the input customized by the developer / Output variables 、Workflow Annotations&Labels, These variables can also be Workflow Access in the global . The two variables are accessed in different ways , Therefore, they will not conflict with each other .

4） Templated variable design

Argo Workflow Controller In fact, the variables of are mainly used in template parsing . stay Controller In the process , You will see it many times json.Marshal/json.Unmarshal operation ： adopt json.Marhsal take Template Object to string , Then the variables in the string are replaced with the real contents through template parsing , Then string json.Unmarshal Overwrite the original attribute value on the object . This design also makes Workflow Template The content corresponding to the variable in must be a specific value （ character string / Basic types such as numbers ）, It cannot be a complex object , Otherwise, the template resolution and replacement cannot be completed .

5） Multi template fusion design

stay Argo Workflow There are three places to set Template Run template , In order of priority ：Default Template、Workflow Template and Node Template.

**Default Template**: overall situation Template Definition , All created Workflow Will automatically use this Template Definition .

**Workflow Template**: Workflow All in the process Node Will be used Template Definition .

**Node Template**: Use Steps/DAG Each step of process scheduling / Mission Node Used Template.

High priority Template Low priority will be overwritten at run time Template, Finally, it is generated by fusion Template And then use it to Pod The creation of .

6） Simplified scheduling control

Argo Workflow At present, only two scheduling control modes are used ：Steps and DAG.

**Steps:** Pass the sequence of steps 、 parallel / Serial control to schedule execution tasks .

**DAG:** Through directed acyclic graph , Scheduling tasks based on the dependencies between tasks .

And these two methods can be mixed , bring Argo Workflow It can basically meet most of the task scheduling business scenarios .

3、 Core structure

Whole Controller The core data structure involved in the logic is as follows .

data structure	Structure is introduced
WorkflowController	[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-HaFUJmSj-1657099905845)(https://goframe.org/download/attachments/44462821/image2021-7-1_11-14-52.png?version=1&modificationDate=1650791949773&api=v2)] be used for Workflow Controller The core data structure object of process control , Encapsulates the main Controller Processing logic 、 Maintaining the core related business logic objects 、 Data queue 、KubeClient object 、Informer Objects, etc. . This structure has only one object instance , Created by the main process .
Workflow	[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-oM0zaYN4-1657099905846)(https://goframe.org/download/attachments/44462821/image2021-7-1_11-18-1.png?version=1&modificationDate=1650791972026&api=v2)]Workflow Content management objects , be used for Workflow Logical processing of .
WorkflowSpec	[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-HOfc0Yz4-1657099905846)(https://goframe.org/download/attachments/44462821/image2021-7-1_11-19-10.png?version=1&modificationDate=1650791999290&api=v2)]Workflow Content definition mapping object , With developers yaml The file structure corresponds one by one . We need to pay attention to WorkflowStatus The difference between ：WorkflowSpec yes Workflow The definition of , originate Workflow Yaml Configuration and object initialization . The modification operation will not be performed when running after initialization , In runtime operation, only Spec Object to perform a read operation .WorkflowStatus yes Workflow Runtime state information management object , Because the status information will change constantly , Therefore, the internal properties will be constantly modified .
WorkflowStatus	[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-vqF9Tp8Q-1657099905846)(https://goframe.org/download/attachments/44462821/image2021-7-1_11-21-30.png?version=1&modificationDate=1650792018995&api=v2)]Workflow The runtime state information management object in the logic processing flow . The structure is similar to Kubernetes Pod Operation related resource structure . Some important notes ：``1、StoredTemplates The attribute is a Map type , Store the current Workflow be-all Template object , For global access . The key name is generated TemplateID, The generation rule is ：Scope/MetaName/TemplateName[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-eGmqJNzD-1657099905847)(https://goframe.org/download/attachments/44462821/image2021-7-1_17-42-37.png?version=1&modificationDate=1650792075580&api=v2)]
WorkflowStep	Yes , You didn't guess wrong. , This is the object used to manage and execute each operation step of process control . The step object must be bound with a Template object .Workflow The initialization execution step of is through woc.execWf.Spec.Entrypoint As an entrance Template.
wfOperationCtx	Workflow Business logic encapsulates objects . Some important notes ：1、wf/orig/execWf1）wf This object is developed by yaml Created Workflow Deep copy of object . Official comments suggest that... Should be used in runtime logic processing execWf instead of wf object ,wf Objects may be discarded in the future .2）orig This object is developed by yaml Created Workflow object , Developers should not modify it at any time , This object is mainly used for the subsequent processing of Workflow Of patch Update judgment .3）execWf This object is modified in runtime logical processing Workflow object , because Workflow Objects are constantly modified and updated in logical processing , especially execWf Are multiple templates (Wf/WfDefault/WfTemplate) Merge structure of . About TemplateDefault Please refer to the official documents for the introduction of ：https://argoproj.github.io/argo-workflows/template-defaults/WfTemplate originate templateRef To configure , Please refer to official documents for details ：https://argoproj.github.io/argo-workflows/workflow-templates/#referencing-other-workflowtemplates2、globalParams Global variables , The type is map[string]string, The Workflow All in template Share this variable , The name of this variable can also be used template Template variables in .3、update This attribute is used to identify the current Workflow Whether the object has been updated , In order to judge whether to synchronize to Kubernetes in .4、node stay woc The source code of the processing flow will appear node The concept of , there node yes Steps/DAG in The execution node of , Each node will run one Pod To execute . Pay attention to it and Template It's not a concept .
templateresolution.``Context	[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-rFsdUB76-1657099905848)(https://goframe.org/download/attachments/44462821/image2021-7-1_16-56-26.png?version=1&modificationDate=1650792180937&api=v2)] As the notes show , be used for Workflow Medium template retrieval .

4、 The core processes

Flow chart of main nodes ：https://whimsical.com/[email protected]V1vz75

because Argo Workflow Controller There are a lot of details 、 The process is very long , The process is simplified here , Only relatively important execution nodes are reserved , In order to focus on the introduction .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-j0eLELmf-1657099905849)(https://goframe.org/download/attachments/44462821/Kubernetes%20Argo%20Controller%20%282%29.png?version=1&modificationDate=1650792215206&api=v2)]

1）WorkflowController

• ![img](https://goframe.org/download/attachments/44462821/image2021-7-2_10-6-9.png?version=1&modificationDate=1650792237734&api=v2)

Controller It's started by Cobra Command line component management , adopt workflow-controller Command execution starts . Create after startup WorkflowController object , And execute the Run Method gives the control of the process to the object maintenance . At the same time, a HTTP Serever:``6060/healthz, be used for Controller Containers Health check . however , From the execution results ,6060 The health check service of the port is not used , Instead, it is opened later Metrics Http Server As the address of health examination .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-i205v2Bd-1657099905849)(https://goframe.org/download/attachments/44462821/image2021-7-2_10-23-20.png?version=1&modificationDate=1650792257141&api=v2)]

• Initializing WorkflowController Will automatically create an internal Informer object Watch ConfigMap The change of , When argo Correlation ConfigMap After the update , It will update automatically wfc Related configuration of , Including database connection Session.

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-0JNMhU73-1657099905849)(https://goframe.org/download/attachments/44462821/image2021-7-2_15-42-36.png?version=2&modificationDate=1650792303496&api=v2)]

2）wfController.Run

WorkflowCotroller First, a lot of initialization operations will be carried out , Mainly as follows ：

• establish wfc.wfInformer/wfc.wftmplInformer/wfc.podInformer/wfc.cwftmplInformer And bind related Event Handler, According to their respective settings cache.ListWatch The rules are right Event To filter （ Only listen argo Create related resources ）. for example ：

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-cwlwZBu0-1657099905850)(https://goframe.org/download/attachments/44462821/image2021-7-2_15-51-15.png?version=1&modificationDate=1650792536921&api=v2)]

• establish Metrics Http Server:9090, be used for Prometheus Indicator reporting of , There are a lot of internal indicators , You can create a topic to study , I don't want to go into it .

• classical Kubernetes Client Leader Election logic , When elected Leader when , stay Leader Node passing OnStartedLeading Callback into wfc.startLeading Logic .

• wfc.startLeading Start the opening of the queue in 、 Creation of asynchronous tasks , It's used here wait.Until Method , This method will create an asynchronous co process execution every other period of time .

• This involves 3 A queue of

  worker
  

  establish ：

  wfc.wfQueue/wfc.podQueue/wfc.podCleanupQueue
  

  ：

  • wfc.wfQueue For the core Workflow Object creation / Modify process control .
  • wfc.podQueue be used for Pod Update , In fact, when Pod If there is an update Pod There is still , Then go again wfc.wfQueue Add a piece of data to go through again Workflow The process is right Pod Execute modification .
  • wfc.podCleanupQueue be used for Pod The marking of is complete . close ： Shut down first main container, To shut down wait container（ Send first when closing syscall.SIGTERM Send again syscall.SIGKILL The signal ）. Delete ： Directly from Kubernetes in Delete The Pod.
  • The relationship between several queues can also be seen in the official architecture diagram .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-pYJ8VRse-1657099905850)(https://goframe.org/download/attachments/44462821/image2021-7-2_16-19-31.png?version=1&modificationDate=1650792558087&api=v2)]

3）wrc.wfQueue

wfc.wfQueue Is the core message queue , Next, we will mainly study the business logic processing of the queue .

4）util.FromUnstructured

Because of our wfc.wfInformer It uses dynamicInterface Filter type , So all event objects are unstructured.Unstructured object （ It's actually a map[string]interface{}）, Cannot convert directly to by assertion Workflow object . So here we use util.FromUnstructured Methods will unstructured.Unstructured Object to Workflow object .

5）newWorkflowOperationCtx

This method will create the core wfOperationCtx object , The object is in Workflow The core of processing is context flow and variable management object , Next wfc(WorkflowController) Will transfer the process control of business logic to woc(wfOperationCtx) To manage . We can understand ,wfc It's a Kubernetes Controller, be used for CRD The implementation of the , Responsible for working with Kubernetes Event Dealing with .woc Responsible for internal business logic 、 technological process 、 Variable management , therefore woc yes Workflow The core business logic in processing encapsulates objects .

6）woc.operate

There is no doubt that , The next control passed to woc(wfOperationCtx), adopt woc.operate Enter the business logic processing process .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-KmW45uFB-1657099905851)(https://goframe.org/download/attachments/44462821/image2021-7-2_16-37-11.png?version=1&modificationDate=1650792577887&api=v2)]

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-zEFHCZaI-1657099905851)(https://goframe.org/download/attachments/44462821/image2021-7-2_16-38-49.png?version=1&modificationDate=1650792611456&api=v2)]

7）woc.setExecWorkflow

• adopt woc.execWf Property object settings woc Of volumes Disk mount .
• adopt woc.setGlobalParameters Set up woc Of globalParams Global variables .
• adopt woc.substituteGlobalVariables analysis woc.execWf.Spec Template variables in .

8）woc.createTemplateContext

adopt woc.CreateTemplateContext establish templateresolution.Context, This object is used for Workflow Medium template retrieval .

9）woc.substituteParamsInVolumes

adopt woc.substituteParamsInVolumes Method parsing replacement Volume Variable content in configuration .

10）woc.createPVCs

adopt woc.createPVCs Methods according to the woc.execWf.Spec.VolumeClaimTemplates Configuration creation PVC.

11）woc.executeTemplate

• adopt woc.executeTemplate Method start execution Workflow Medium Template, The entrance is woc.execWf.Spec.Entrypoint.

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-0j1ffyCM-1657099905851)(https://goframe.org/download/attachments/44462821/image2021-7-1_17-28-40.png?version=1&modificationDate=1650792629990&api=v2)]

• The internal will be based on the given Entrypoint Go first StoredTemplates Retrieve the corresponding Template object , After finding it, you should Template Object makes a deep copy and returns the copied object . If you can't find it, go to Workflow Object , And cache 、 Return to found Template object .

12）woc.mergedTemplateDefaultsInto

About what is TemplateDefaults Please refer to chapter introduction ：https://argoproj.github.io/argo-workflows/template-defaults/

adopt woc.mergedTemplateDefaultsInto Method to set the user configured TemplateDefaults Merge into the current operation Template On the object .

13）common.ProcessArgs

common.ProcessArgs The method is mainly used for Template Template variable parsing for .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-fxy4rvFB-1657099905852)(https://goframe.org/download/attachments/44462821/image2021-7-2_19-43-57.png?version=1&modificationDate=1650792688988&api=v2)]

Be careful ：argo There are two kinds of variables in the interior , One is Workflow Globally effective variables (globalParams), One is the current Template Effective local variables (localParams). The global variables also include the input customized by the developer / Output variables 、Workflow Annotations&Labels, These variables can also be Workflow Access in the global .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-MImPOHF7-1657099905852)(https://goframe.org/download/attachments/44462821/image2021-7-2_19-30-46.png?version=1&modificationDate=1650792735446&api=v2)]

In template variable parsing , There is another key point .Argo Template variables of support expressions , Expression parsing uses github.com/antonmedv/expr Components .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-XKq7xPDm-1657099905853)(https://goframe.org/download/attachments/44462821/image2021-7-3_14-38-26.png?version=1&modificationDate=1650792754050&api=v2)]

14）processedTmpl.Memoize

processdTmpl.Memoize Configuration is used for developers to customize whether to cache the current Template Execution results , Please refer to chapter ：https://argoproj.github.io/argo-workflows/memoization/#using-memoization

15）processedTmpl.GetType

Next is Template The key to implementation , According to Template type , Execute different operation logic . As you can see from the flowchart , The key is Container type , as well as Steps&DAG type . among Container The type is all Template The end of execution , in other words Template Execution ultimately requires a container to implement . and Steps&DAG Type is used to control the Template technological process , Through loop execution , It will eventually fall to Container Type .

• Suspend

Suspend Type of Template adopt woc.executeSuspend Method realization , The interior is just the current Template Mark the update time and Suspend And throw it back in the queue for the next judgment .

• **Script**

Script Type of Template adopt woc.executeScript Method realization , Internal judgment of current Script Is there anything else Template In the use of , Then call woc.createWorkflowPod establish Pod To Kubernetes in .

• **Resource**

Resource Type of Template adopt woc.executeResource Method realization ,Resource Content by creating a argoexec Containers , And use argoexec resource Command parsing parameters , Containers are created by calling woc.createWorkflowPod establish Pod To Kubernetes in .

• **Data**

Data Type of Template adopt woc.executeData Method realization ,data Content by creating a argoexec Containers , And use argoexec data Command parsing parameters , Containers are created by calling woc.createWorkflowPod establish Pod To Kubernetes in .

• **ContainerSet**

ContainerSet Type of Template adopt woc.executeContainerSet Method realization , Multiple containers are created by calling woc.createWorkflowPod establish Pod To Kubernetes in . About ContainerSet Type of Template Please refer to ：https://argoproj.github.io/argo-workflows/container-set-template/

• **Steps & DAG**

Steps&DAG Type of Template adopt woc.executeSteps、woc.executeDAG Method realization , Internally, there will be multiple Template Control the process , Cycle call woc.executeTemplate Methods execute each Template.

• **Container**

This part is the whole Workflow Controller The key to scheduling , Is to create Pod Core logic of .Container Type of Template adopt woc.executeTemplate Method realization . In this method , There are several important points involved Pod Set up ：

a） Create... Based on conditions Init/Wait Containers, The interior is all through woc.newExecContainer Create a container , When the container is created, set the general environment variables and Volume mount .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-nEkkf3tR-1657099905854)(https://goframe.org/download/attachments/44462821/image2021-7-3_10-58-40.png?version=1&modificationDate=1650793138477&api=v2)]

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-uBFyPasb-1657099905854)(https://goframe.org/download/attachments/44462821/image2021-7-3_10-52-34.png?version=1&modificationDate=1650793159784&api=v2)]

b）addVolumeReferences According to the customized Volume, Attach to by name Pod Of Init/Wait/Main Containers in .

c）addSchedulingConstraints Methods according to the WorkflowSpec The configuration of Pod Some scheduling strategies for scheduling , Include ：NodeSelector/Affinity/Tolerations/SchedulerName/PriorityClassName/Priority/HostAliases/SecurityContext.

d）woc.addInputArtifactsVolumes about artifacts Feature is a very important method , take Artifacts dependent Volume Mount to Pod in , these Volume Include ：/argo/inputs/artifacts 、 /mainctrfs And what the developer set in the configuration Volume Address .

If Template The type is Script, Then one will be added /argo/staging Of emptyDir Type of Volume, be used for Init/Wait/``Main Containers To share Resource Content . Let's take an official example (scripts-bash.yaml)：

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-8uAWY9sc-1657099905854)(https://goframe.org/download/attachments/44462821/image2021-7-5_19-53-5.png?version=1&modificationDate=1650793183278&api=v2)]

In the use of artifacts When it comes to configuration , It will create a file named inputs-artifacts Of emptyDir type volume for Init/Wait/Main Containers share artifacts data . Let's take an official example (artifacts-passing.yaml)：

![img](https://goframe.org/download/attachments/44462821/image2021-7-5_19-48-52.png?version=1&modificationDate=1650793206909&api=v2)

e）addInitContainers & addSidecars & ``addOutputArtifactsVolumes take Main Containers Medium Volume Synchronously mount to Init/Wait Containers in , In order to share data . You can see from an example ,Main Containers Medium Volume stay Init/Wait Containers There are .

f） Some fixed environment variable settings , Pay attention to the Template Environment variable Settings , Will the whole Template Object to Json Then plug it into the environment variable , To facilitate subsequent container reading ：

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-Umlr9py6-1657099905855)(https://goframe.org/download/attachments/44462821/image2021-7-3_10-23-46.png?version=1&modificationDate=1650793264646&api=v2)]

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-1AAyE8l0-1657099905856)(https://goframe.org/download/attachments/44462821/image2021-7-3_10-24-12.png?version=1&modificationDate=1650793488449&api=v2)]

g）substituePodParams Last variable substitution , Especially from Workflow ConfigMap perhaps Volume Property .

h）kubeclientset.CoreV1.Pods.Create Will be created before Pod Submitted to the Kubernetes Execution creation .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-jYVIWbxT-1657099905856)(https://goframe.org/download/attachments/44462821/image2021-7-3_10-37-11.png?version=2&modificationDate=1650793634886&api=v2)]

5、 ... and 、ArgoExec Container

1、 Core structure

Whole agoexec The core data structure involved in the logic is as follows .

data structure	Brief introduction
WorkflowExecutor	be used for Init/Wait Containers The core object of operation management .
ContainerRuntimeExecutor	As the notes show , Used with Docker Container interactive API Interface .
Artifact	[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-A26YKQha-1657099905857)(https://goframe.org/download/attachments/44462821/image2021-7-3_11-32-4.png?version=1&modificationDate=1650800709280&api=v2)]Artifact Resource management objects .
ArtifactDriver	[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-CYQNGU65-1657099905858)(https://goframe.org/download/attachments/44462821/image2021-7-3_11-31-15.png?version=1&modificationDate=1650800767563&api=v2)] be used for Artifacts Drive management .Argo By default, it supports multiple Artifacts drive .
ArchiveStrategy	ArchiveStrategy Used to identify the Artifact The compression strategy .

2、ArgoExec Init

Only in Template The type is Script Or with Artifacts When the function ,Argo Workflow Controller For Pod establish Init Container, The Container It uses argoexec Mirror image , adopt argoexec init Command to start running .Init Container The main responsibility is to Script Of Resource Read or will depend on Artifacts Content pull , Save to a locally mounted share Volume On , Convenient for subsequent startup Main Container Use .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-D9jAw1SZ-1657099905859)(https://goframe.org/download/attachments/44462821/image2021-7-3_11-41-5.png?version=1&modificationDate=1650800884583&api=v2)]

because Init Container The implementation process of is relatively simple , Here is a brief introduction .

1）iniExecutor & wfExecutor.Init

First create WorkflowExecutor object , This object is used for Init/Wait Containers Core business logic encapsulation 、 Process control execution .

stay WorkflowExecutor Objects are created at the same time ContainerRuntimeExecutor object , be used for Docker Container Interaction , Include Docker Terminal output read 、 Obtain the result file and other important operations . By default ,WorkflowExecutor Will create a DockerExecutor object .

Besides , You may wonder why you can be with Pod Inside Container Interaction , And how to get Docker I feel curious about the output of . Then we describe One Pod You may understand by looking at it ：

![img](https://goframe.org/download/attachments/44462821/image2021-7-3_11-56-38.png?version=1&modificationDate=1650800965124&api=v2)

You can see , The container is attached docker.dock File to local , So that the local can pass docker Command and docker Interact . Of course Init Container Not directly with Docker Interaction , Often only Wait Container Will , therefore Init Container This... Is not mounted in docker.sock file .

2）wfExecutor.StageFiles

wfExecutor.StageFiles Method is used to Script/Resource（ If there is ） Save and write locally mounted in the form of a file Volume Location , these Volume yes Container Share subsequent operations , follow-up Main Container Through sharing Volume Access these files . It should be noted that , Different Template type , The source of content and the location of the written disk will be different ：

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-zh965CY6-1657099905859)(https://goframe.org/download/attachments/44462821/image2021-7-3_11-59-59.png?version=1&modificationDate=1650801036613&api=v2)]

3）wfExecutor.LoadArtifacts

This method is only used Artifacts Effective in functional scenarios . Responsible for configuring Artifact Pull to local , And decompress according to the compression strategy , Modify the permissions , For the next step Main Container visit . For easy expansion ,Artifacts Used ArtifactDrive Interface design , Different types of Artifact Can be implemented separately , And introduce according to the type , Use through the interface .

3、ArgoExec Wait

be-all Argo Workflow Template One will be created during execution Wait Container, This is a very critical Container. The Container Responsible for monitoring Main Container Life cycle of , stay Main Container After the main logic operation in , Be responsible for reading the output part 、 Persistence , such Main Container There is no need to worry about how to transmit the results of this step to the next step .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-BTu3aG68-1657099905860)(https://goframe.org/download/attachments/44462821/image2021-7-3_13-54-35.png?version=1&modificationDate=1650801059768&api=v2)]

because Wait Container The implementation process of is relatively simple , Here is a brief introduction .

1）wfxecutor.Wait

This method is used to wait Main Container complete , Let's look at the default DockerExecutor How is the underlying implementation done ：

2）wfExecutor.CaptureScriptResult

By capturing Main Container The terminal output of , And save the output . Special attention required Is the size of the execution result , If exceeded 256KB Will be forcibly truncated .

2）wfExecutor.SaveLogs

Save log , By default, it will be saved to argo Self contained minio service （ Use S3 Communication protocol ） in , The log can also be Argo Server Visit the exhibition .

Argo default ArtifactRepository：

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-KHtOhUhX-1657099905861)(https://goframe.org/download/attachments/44462821/image2021-7-5_17-48-35.png?version=1&modificationDate=1650801163453&api=v2)]

3）wfxecutor.SaveParameters

Only in Template in Outputs This logic will only be executed during configuration , This method saves the result of container execution to the current Template.Outputs.Parameters in .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-B78xQxRF-1657099905862)(https://goframe.org/download/attachments/44462821/image2021-7-3_13-59-11.png?version=1&modificationDate=1650801201199&api=v2)]

3）wfxecutor.SaveArtifacts

If Template There is Artifacts In operation , This method is used to read Main Container Medium Artifacts Save to /mainctrfs Catalog , And unpack （untar/unzip） Save the temporary directory after /tmp/argo/outputs/artifacts Next , Then... In the temporary directory Artifacts The file will be uploaded to Artifact Repository in . It is worth noting that ：

• /mainctrfs Directory is Wait Container And Main Container The share of Volume, So direct file Copy that will do . This is inside Volume Interaction , Files are compressed （tgz） Later , No need to decompress .
• Temporary directory /tmp/argo/outputs/artifacts Under the Artifacts The file is only for subsequent ArtifactDriver Upload to Artifact Repository in , And the uploaded file content needs to be decompressed （untar/unzip）, Because the compression mechanism is just argo Internal file interaction , Not external ArtifactDriver Universal .
• default ArtifactRepository yes minio, Therefore, the execution results will also be saved to minio In service .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-e5JU67UX-1657099905862)(https://goframe.org/download/attachments/44462821/image2021-7-3_14-17-9.png?version=1&modificationDate=1650801224141&api=v2)]

4）wfExecutor.AnnotateOutputs

Wait Container This last step is very interesting . But it may make Metadata Medium Annotation It will become bigger . Attention should be paid when using ,Annotation It's limited in size ,Kubernetes The default size limit for this item is 256KB.

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-0A8k2fYp-1657099905863)(https://goframe.org/download/attachments/44462821/image2021-7-3_14-20-25.png?version=1&modificationDate=1650801243905&api=v2)]

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-ihEE6Kll-1657099905863)(https://goframe.org/download/attachments/44462821/image2021-7-3_14-25-18.png?version=2&modificationDate=1650801281878&api=v2)]

This Annotations Will be in Workflow Controller It is automatically read out and set to Template Of Outputs Properties of the , Such a Template The output of the execution can be associated with other Template Quote to ：

In the final analysis , In terms of the underlying implementation , Multiple Template The way of transferring process data mainly depends on Annotations、Artifacts And share Volume.

4、ArgoExec Other commands

ArgoExec Other orders for （data/resource/emissary） It is mainly used for content analysis in the process of process scheduling , Relatively simple , No more introduction here , Interested can see the source code .

6、 ... and 、 common problem

Argo Workflow The process and main logic of are sorted out , Next, let's answer the first few questions .

Because of the long space , We moved the Q & a content here ：Argo Workflow common problem