Add a live-processing example ClusterWorkflowTemplate #1433
Conversation
There was a problem hiding this comment.
@hazdl this was a good learning exercise to learn how to write Argo Workflows Templates. But I think that what is included in this PR goes beyond the scope of https://jira.diamond.ac.uk/browse/AP-1097. The point of the ClusterWorkflowTemplates in the examples directory is to provide minimal examples that demonstrate how to do something. In this case, our aim is simply to show the users how to run live-processing. I think that the CT analysis is distracting from the point of the template, to demo the "workflows.diamond.ac.uk/type: "live"" annotation.
Let's discuss a more minimal example that satisfies the requirements JIRA ticket.
|
This example should be merged into the examples folder. It demonstrates the usage of a ClusterWorkflowTemplate with priority-based processing and serves as a reference implementation for users. |
hazdl
left a comment
There was a problem hiding this comment.
This example need to be merged in the examples folder, it is an example od a clusterworkflowtemplate with priority processing. It is refered to in the docs under the Enhance Workflow Types and Priorities documentation.
This PR adds a new example Argo Workflows ClusterWorkflowTemplate named surface-analysis to the workflows repository to demonstrate how users can submit jobs intended for live processing.
The example is primarily intended as a learning exercise for a new team member and provides a simple end-to-end scientific workflow that demonstrates:
Creating and registering a ClusterWorkflowTemplate
Submitting workflows from a cluster-scoped template
DAG execution and task dependencies
Artifact passing between workflow stages
Parallel execution of independent tasks
Generating output reports from upstream processing results
The example also demonstrates how workflows intended for live processing can be identified using workflow metadata annotations and labels, allowing the platform to distinguish between live and offline processing workloads.
The workflow itself performs a synthetic surface analysis pipeline consisting of:
3D volume reconstruction
Surface segmentation
Crack detection
Surface roughness analysis
HTML report generation
Testing verified by:
Creating the ClusterWorkflowTemplate
Submitting workflows from the template
Executing the DAG successfully
Verifying artifact passing between tasks
Producing the final report artifact
Confirming the workflow is correctly identified as a live-processing example via metadata