Skip to content

Add a live-processing example ClusterWorkflowTemplate #1433

Open
hazdl wants to merge 1 commit into
mainfrom
hz/sample-cluster-wftemplate
Open

Add a live-processing example ClusterWorkflowTemplate #1433
hazdl wants to merge 1 commit into
mainfrom
hz/sample-cluster-wftemplate

Conversation

@hazdl

@hazdl hazdl commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

This PR adds a new example Argo Workflows ClusterWorkflowTemplate named surface-analysis to the workflows repository to demonstrate how users can submit jobs intended for live processing.

The example is primarily intended as a learning exercise for a new team member and provides a simple end-to-end scientific workflow that demonstrates:

Creating and registering a ClusterWorkflowTemplate
Submitting workflows from a cluster-scoped template
DAG execution and task dependencies
Artifact passing between workflow stages
Parallel execution of independent tasks
Generating output reports from upstream processing results

The example also demonstrates how workflows intended for live processing can be identified using workflow metadata annotations and labels, allowing the platform to distinguish between live and offline processing workloads.

The workflow itself performs a synthetic surface analysis pipeline consisting of:
3D volume reconstruction
Surface segmentation
Crack detection
Surface roughness analysis
HTML report generation

Testing verified by:
Creating the ClusterWorkflowTemplate
Submitting workflows from the template
Executing the DAG successfully
Verifying artifact passing between tasks
Producing the final report artifact
Confirming the workflow is correctly identified as a live-processing example via metadata

@davehadley davehadley left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hazdl this was a good learning exercise to learn how to write Argo Workflows Templates. But I think that what is included in this PR goes beyond the scope of https://jira.diamond.ac.uk/browse/AP-1097. The point of the ClusterWorkflowTemplates in the examples directory is to provide minimal examples that demonstrate how to do something. In this case, our aim is simply to show the users how to run live-processing. I think that the CT analysis is distracting from the point of the template, to demo the "workflows.diamond.ac.uk/type: "live"" annotation.

Let's discuss a more minimal example that satisfies the requirements JIRA ticket.

@hazdl

hazdl commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

This example should be merged into the examples folder. It demonstrates the usage of a ClusterWorkflowTemplate with priority-based processing and serves as a reference implementation for users.
The example is referenced in the documentation added as part of the Enhance Workflow Types and Priorities Documentation changes and should be included to ensure the documentation links to a complete, working example.

@hazdl hazdl left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example need to be merged in the examples folder, it is an example od a clusterworkflowtemplate with priority processing. It is refered to in the docs under the Enhance Workflow Types and Priorities documentation.

@hazdl
hazdl requested a review from davehadley July 17, 2026 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants