用户定义的作业类型在通过SLURM进行计算时不可用

Question

我正在尝试设置pyiron计算(0.3.6版)。我想通过SLURM在计算机集群上执行一个非python脚本。我编写了自己的OwnProgramJob类，它继承了GenericJob类。在我的本地电脑上一切都很顺利。但是，在集群上运行时，我自己的类在pyiron中不可用：

...    
File "/beegfs-home/users/fufl/.local/project/lib/python3.8/site-packages/pyiron_base/generic/hdfio.py", line 1251, in import_class
        return getattr(
AttributeError: module '__main__' has no attribute 'OwnProgramJob'

如何使我自己的类对集群上的pyiron可用？

我认为一种方法是按照JOB_CLASS_DICT的建议，将自己的类直接添加到pyiron源代码中，并修改https://github.com/pyiron/pyiron/issues/973#issuecomment-694347111。还有另一种方法不修改pyiron源代码吗？

我的源代码可以在下面找到作为参考。

非常感谢,

佛里安

木星笔记本：

import pyiron
from pathlib import Path

pr = pyiron.Project(path=f"{str(Path.home())}/pyiron/projects/example")

from pyiron_base import GenericJob
import os

class OwnProgramJob(GenericJob):
    def __init__(self, project, job_name):
        super().__init__(project, job_name)
        self.input = OwnProgramInput()
        self.executable = "cat input.in > output.out"
    
    def write_input(self):
        with open(os.path.join(self.working_directory, "input.in"), 'w') as infile:
            infile.write("asd 100")
    
    def collect_output(self):
        file = os.path.join(self.working_directory, "output.out")
        with open(file) as f:
            line = f.readlines()[0]
            energy = float(line.split()[1])
        with self.project_hdf5.open("output/generic") as h5out:
            h5out["energy_tot"] = energy
    
    
class OwnProgramInput(GenericParameters):
    def __init__(self, input_file_name=None):
        super(OwnProgramInput, self).__init__(
            input_file_name=input_file_name,
            table_name="input")
        
    def load_default(self):
        self.load_string("input_energy 100")

job = pr.create_job(job_type=OwnProgramJob, job_name="test", delete_existing_job=True)

job.server.queue = 'cpu'

job.run()

pr.job_table()

SLURM作业文件：

#SBATCH --workdir={{working_directory}}
#SBATCH --get-user-env=L
#SBATCH --partition=cpu                                                                                                                                                                                                                  
{%- if run_time_max %}
#SBATCH --time={{run_time_max // 60}}
{%- endif %}
{%- if memory_max %}
#SBATCH --mem={{memory_max}}
{%- endif %}
#SBATCH --cpus-per-task={{cores}} 
    
{{command}}

Answer 1

要使作业类在提交到队列系统时可用，就必须将其包含在python路径中。因此，我建议将类定义拆分到一个名为ownprogramjob.py的独立python模块中。

import os
from pyiron_base import GenericJob, GenericParameters


class OwnProgramJob(GenericJob):
    def __init__(self, project, job_name):
        super().__init__(project, job_name)
        self.input = OwnProgramInput()
        self.executable = "cat input.in > output.out"
    
    def write_input(self):
        with open(os.path.join(self.working_directory, "input.in"), 'w') as infile:
            infile.write("asd 100")
    
    def collect_output(self):
        file = os.path.join(self.working_directory, "output.out")
        with open(file) as f:
            line = f.readlines()[0]
            energy = float(line.split()[1])
        with self.project_hdf5.open("output/generic") as h5out:
            h5out["energy_tot"] = energy
    
    
class OwnProgramInput(GenericParameters):
    def __init__(self, input_file_name=None):
        super(OwnProgramInput, self).__init__(
            input_file_name=input_file_name,
            table_name="input")
        
    def load_default(self):
        self.load_string("input_energy 100")

然后，您可以使用以下方法提交：

from pyiron import Project
from ownprogramjob import OwnProgramJob


pr = Project("test")
job = pr.create_job(job_type=OwnProgramJob, job_name="test", delete_existing_job=True)

job.server.queue = 'cpu'

job.run()

pr.job_table()

最好的

1月