调用snakefile中的另一个管道会导致输出错误

Question

我在我的snakemake流水线中使用了一个名为Canu的组装流水线，但是当涉及到调用Canu的规则时，snakemake退出时会出现MissingOutputException错误，因为流水线向集群本身提交了多个作业，所以看起来snakemake期望在第一个作业完成后输出。有没有办法避免这种情况？我知道我可以使用很长的延迟等待选项，但这并不是最优的。

snakefile代码：

#!/miniconda/bin/python

workdir: config["path_to_files"]
wildcard_constraints:
    separator = config["separator"],
    sample = '|' .join(config["samples"]),

rule all:
    input:
        expand("assembly-stats/{sample}_stats.txt", sample = config["samples"])

rule short_reads_QC:
    input:
        f"short_reads/{{sample}}_short{config['separator']}*.fq.gz"

    output:
        "fastQC-reports/{sample}.html"

    conda:
        "/home/lamma/env-export/hybrid_assembly.yaml"

    shell:
        """
        mkdir fastqc-reports
        fastqc -o fastqc-reports {input}
        """

rule quallity_trimming:
    input:
        forward = f"short_reads/{{sample}}_short{config['separator']}1.fq.gz",
        reverse = f"short_reads/{{sample}}_short{config['separator']}2.fq.gz",

    output:
        forward = "cleaned_short-reads/{sample}_short_1-clean.fastq",
        reverse = "cleaned_short-reads/{sample}_short_2-clean.fastq"

    conda:
        "/home/lamma/env-export/hybrid_assembly.yaml"

    shell:
        "bbduk.sh -Xmx1g in1={input.forward} in2={input.reverse} out1={output.forward} out2={output.reverse}  qtrim=rl trimq=10"

rule long_read_assembly:
    input:
        "long_reads/{sample}_long.fastq.gz"

    output:
        "canu-outputs/{sample}.subreads.contigs.fasta"

    conda:
        "/home/lamma/env-export/hybrid_assembly.yaml"

    shell:
        "canu -p {wildcards.sample} -d canu-outputs genomeSize=8m -pacbio-raw {input}"
rule short_read_alignment:
    input:
        short_read_fwd = "cleaned_short-reads/{sample}_short_1-clean.fastq",
        short_read_rvs = "cleaned_short-reads/{sample}_short_2-clean.fastq",
        reference = "canu-outputs/{sample}.subreads.contigs.fasta"

    output:
        "bwa-output/{sample}_short.bam"

    conda:
        "/home/lamma/env-export/hybrid_assembly.yaml"

    shell:
        "bwa mem {input.reference} {input.short_read_fwd} {input.short_read_rvs} |  samtools view -S -b > {output}"


rule indexing_and_sorting:
    input:
        "bwa-output/{sample}_short.bam"
    output:
        "bwa-output/{sample}_short_sorted.bam"

    conda:
        "/home/lamma/env-export/hybrid_assembly.yaml"

    shell:
        "samtools sort {input} > {output}"

rule polishing:
    input:
        bam_files = "bwa-output/{sample}_short_sorted.bam",
        long_assembly = "canu-outputs/{sample}.subreads.contigs.fasta"

    output:
        "pilon-output/{sample}-improved.fasta"

    conda:
        "/home/lamma/env-export/hybrid_assembly.yaml"

    shell:
        "pilon --genome {input.long_assembly} --frags {input.bam_files} --output {output} --outdir pilon-output"

rule assembly_stats:
    input:
        "pilon-output/{sample}-improved.fasta"
    output:
        "assembly-stats/{sample}_stats.txt"

    conda:
        "/home/lamma/env-export/hybrid_assembly.yaml"

    shell:
        "stats.sh in={input} gc=assembly-stats/{wildcards.sample}/{wildcards.sample}_gc.csv gchist=assembly-stats/{wildcards.sample}/{wildcards.sample}_gchist.csv shist=assembly-stats/{wildcards.sample}/{wildcards.sample}_shist.csv > assembly-stats/{wildcards.sample}/{wildcards.sample}_stats.txt"

确切的错误是：

Waiting at most 60 seconds for missing files.
MissingOutputException in line 43 of /faststorage/home/lamma/scripts/hybrid_assembly/bacterial-hybrid-assembly.smk:
Missing files after 60 seconds:
canu-outputs/F19FTSEUHT1027.PSU4_ISF1A.subreads.contigs.fasta
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

正在使用的snakemake命令：

snakemake --latency-wait 60 --rerun-incomplete --keep-going --jobs 99 --cluster-status 'python /home/lamma/faststorage/scripts/slurm-status.py' --cluster 'sbatch  -t {cluster.time} --mem={cluster.mem} --cpus-per-task={cluster.c} --error={cluster.error}  --job-name={cluster.name} --output={cluster.output}' --cluster-config bacterial-hybrid-assembly-config.json --configfile yaml-config-files/test_experiment3.yaml --use-conda --snakefile bacterial-hybrid-assembly.smk

Answer 1

我猜想canu给你的是canu-outputs/{sample}.contigs.fasta而不是canu-outputs/{sample}.subreads.contigs.fasta。如果是，则将canu commad编辑为

canu -p {wildcards.sample}.subreads ...

(顺便说一句，我不认为#!/miniconda/bin/python是必要的)。