srun on a single node

[soaringxmc]

Hi, @m-kurz @b-fg

Currently, smartflow can run on cpu and gpu clusters. However, we can only use local + mpirun to run on a single node. It is highly desirable if we can also use slurm+srun on a single node of a cluster, where database and cfd runs are located on the same node.

In the code, if the database is created, echo experiment would not be able to start. However, if the database is not created by commenting the database lines, echo experiment would not be able to start. Do you have any ideas for solving the issue?

import smartsim
import os
from smartsim import Experiment
from smartsim.database.orchestrator import Orchestrator

exp = Experiment('envs', launcher='slurm')

db = exp.create_database(
    interface='lo',
    # Set the database to run on the current allocation
    db_nodes=1,
    batch=False,  # Important: This tells SmartSim to use the current allocation
)
exp.start(db)

models = []
for i in range(1):
    # Define run arguments
    run_args = {
        # 'cpus-per-task': 8,
        # 'gpus-per-task': 1,
    }
        
    run_settings = exp.create_run_settings(
        exe="echo",
        exe_args="Hello World",
        run_command="srun",
        run_args=run_args,
    )
    run_settings.set_tasks(1)

    model = exp.create_model(f"env_{i}", run_settings)
    exp.start(model, block=False, summary=False)
    models.append(model)

[b-fg]

In the code, if the database is created, echo experiment would not be able to start. However, if the database is not created by commenting the database lines, echo experiment would not be able to start

I am not sure I fully understand...

It is highly desirable if we can also use slurm+srun on a single node of a cluster

Also, what is the problem with this? What error do you get?

[soaringxmc]

@b-fg Sorry for being unclear.

SmartSim might only support local+mpirun on a single node, even when the node is allocated by slurm.
If we use slurm+srun, the orchestrater database must be hosted on a separate node, so at least two nodes should be allocated.

Anyway, smartflow can fully support a single gpu node (local+mpirun) and multiple gpu nodes (slurm+srun).

[b-fg]

I see. Maybe that a question which @ashao could provide some insights.

[soaringxmc]

Thank you for your advice. @b-fg
Hi @ashao
Is it possible to use slurm+srun to put orchestrator database and simulations on a single node allocated by slurm? Currently, we can only use local+mpirun for this purpose, even if the single node is allocated by slurm on a cluster.