Snakemake：--use-conda with --cluster

Question

当我使用--cluster和--use-conda运行时，在提交到集群之前，Snakemake似乎没有设置conda环境，因此我的作业失败。在集群提交前设置conda环境有没有什么窍门？

编辑：

我在conda环境中得到了snakemake，如下所示：

channels:
  - bioconda
  - conda-forge
dependencies:
  - snakemake-minimal=5.19.3
  - xrootd=4.12.2

复制者：

我使用Snakefile、dothing.py和environment.yml创建了一个目录

Snakefile

shell.prefix('unset PYTHONPATH; unset LD_LIBRARY_PATH; unset PYTHONHOME; ')

rule dothing:
    conda: 'environment.yml'
    output: 'completed.out'
    log: 'thing.log'
    shell: 'python dothing.py &> {log} && touch {output}'

dothing.py

import uncertainties

print('it worked!')

environment.yml

name: testsnakeconda
channels:
  - conda-forge
dependencies:
  - uncertainties=3.1.4

如果我像这样在本地运行

snakemake --cores all --use-conda

它运行起来没有任何问题：

Building DAG of jobs...
Creating conda environment environment.yml...
Downloading and installing remote packages.
Environment for environment.yml created (location: .snakemake/conda/e0fff47f)
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       dothing
        1

[Tue Jun 30 16:19:38 2020]
rule dothing:
    output: completed.out
    log: thing.log
    jobid: 0

Activating conda environment: /path/to/environment.yml
[Tue Jun 30 16:19:39 2020]
Finished job 0.
1 of 1 steps (100%) done
Complete log: /path/to/.snakemake/log/2020-06-30T161824.906217.snakemake.log

如果我尝试使用--cluster提交，比如

snakemake --cores all --use-conda --cluster 'condor_qsub -V -l procs={threads}' --latency-wait 30 --max-jobs-per-second 100 --jobs 50

没有关于设置conda环境的消息，作业失败并出现错误：

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cluster nodes: 50
Job counts:
        count   jobs
        1       dothing
        1

[Tue Jun 30 16:20:49 2020]
rule dothing:
    output: completed.out
    log: thing.log
    jobid: 0

Submitted job 0 with external jobid 'Your job 9246856 ("snakejob.dothing.0.sh") has been submitted'.
[Tue Jun 30 16:26:00 2020]
Error in rule dothing:
    jobid: 0
    output: completed.out
    log: thing.log (check log file(s) for error message)
    conda-env: /path/to/.snakemake/conda/e0fff47f
    shell:
        python dothing.py &> thing.log && touch completed.out
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    cluster_jobid: Your job 9246856 ("snakejob.dothing.0.sh") has been submitted

Error executing rule dothing on cluster (jobid: 0, external: Your job 9246856 ("snakejob.dothing.0.sh") has been submitted, jobscript: /path/to/.snakemake/tmp.a7fpixla/snakejob.dothing.0.sh). For error details see the cluster log and the log files of the involved rule(s).
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /path/to/.snakemake/log/2020-06-30T162049.793041.snakemake.log

我可以看到问题在于uncertainties包不可用：

$ cat thing.log
Traceback (most recent call last):
  File "dothing.py", line 1, in <module>
    import uncertainties
ImportError: No module named uncertainties

编辑:不带--cluster的详细输出

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       dothing
        1
Resources before job selection: {'_cores': 10, '_nodes': 9223372036854775807}
Ready jobs (1):
        dothing
Selected jobs (1):
        dothing
Resources after job selection: {'_cores': 9, '_nodes': 9223372036854775806}

[Thu Jul  2 21:51:18 2020]
rule dothing:
    output: completed.out
    log: thing.log
    jobid: 0

Activating conda environment: /path/to/workingdir/.snakemake/conda/e0fff47f
[Thu Jul  2 21:51:33 2020]
Finished job 0.
1 of 1 steps (100%) done
Complete log: /path/to/workingdir/.snakemake/log/2020-07-02T215117.964474.snakemake.log
unlocking
removing lock
removing lock
removed all locks

使用--cluster的详细输出

Building DAG of jobs...
Checking status of 0 jobs.
Using shell: /usr/bin/bash
Provided cluster nodes: 50
Job counts:
    count   jobs
    1   dothing
    1
Resources before job selection: {'_cores': 9223372036854775807, '_nodes': 50}
Ready jobs (1):
    dothing
Selected jobs (1):
    dothing
Resources after job selection: {'_cores': 9223372036854775806, '_nodes': 49}

[Thu Jul  2 21:40:23 2020]
rule dothing:
    output: completed.out
    log: thing.log
    jobid: 0

Jobscript:
#!/bin/sh
# properties = {"type": "single", "rule": "dothing", "local": false, "input": [], "output": ["completed.out"], "wildcards": {}, "params": {}, "log": ["thing.log"], "threads": 1, "resources": {}, "jobid": 0, "cluster": {}}
 cd /path/to/workingdir && \
/path/to/miniconda/envs/envname/bin/python3.8 \
-m snakemake dothing --snakefile /path/to/workingdir/Snakefile \
--force -j --keep-target-files --keep-remote \
--wait-for-files /path/to/workingdir/.snakemake/tmp.5n32749i /path/to/workingdir/.snakemake/conda/e0fff47f --latency-wait 30 \
 --attempt 1 --force-use-threads \
--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ \
   --allowed-rules dothing --nocolor --notemp --no-hooks --nolock \
--mode 2  --use-conda  && touch /path/to/workingdir/.snakemake/tmp.5n32749i/0.jobfinished || (touch /path/to/workingdir/.snakemake/tmp.5n32749i/0.jobfailed; exit 1)

Submitted job 0 with external jobid 'Your job 9253728 ("snakejob.dothing.0.sh") has been submitted'.
Checking status of 1 jobs.
...
Checking status of 1 jobs.
[Thu Jul  2 21:46:23 2020]
Error in rule dothing:
    jobid: 0
    output: completed.out
    log: thing.log (check log file(s) for error message)
    conda-env: /path/to/workingdir/.snakemake/conda/e0fff47f
    shell:
        python dothing.py &> thing.log && touch completed.out
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    cluster_jobid: Your job 9253728 ("snakejob.dothing.0.sh") has been submitted

Error executing rule dothing on cluster (jobid: 0, external: Your job 9253728 ("snakejob.dothing.0.sh") has been submitted, jobscript: /path/to/workingdir/.snakemake/tmp.5n32749i/snakejob.dothing.0.sh). For error details see the cluster log and the log files of the involved rule(s).
Cleanup job metadata.
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /path/to/workingdir/.snakemake/log/2020-07-02T214022.614691.snakemake.log
unlocking
removing lock
removing lock
removed all locks

Answer 1

对我来说起作用的是在规则内部设置python解释器的完整路径...

rule dothing:
    conda: 'environment.yml'
    output: 'completed.out'
    log: 'thing.log'
    shell: '/full_path_to_your_environment/bin/python dothing.py &> {log} && touch {output}'

以及python脚本的完整路径(如果它是安装在特定环境中的包的话)(就是我的例子)。

rule dothing:
    conda: 'environment.yml'
    output: 'completed.out'
    log: 'thing.log'
    shell: '/full_path_to_your_environment/bin/python /full_path_to_your_environment/package_dir/dothing.py &> {log} && touch {output}'

我所说的/full_path_to_your_environment/指的是conda和snakemake在第一次安装时为您的环境提供的散列名称(例如/path/to/workingdir/.snakemake/conda/e0fff47f)

这有点令人难以置信，但还是成功了。

希望这能有所帮助