在进入下一个规则之前，我如何让Snakemake将所有样本应用到单个规则？

Question

在具有j核的机器上，给定依赖于RuleB的RuleA，我希望Snakemake执行我的工作流程如下：

RuleA Sample1 using j threads
RuleA Sample2 using j threads
...
RuleA SampleN using j threads
RuleB Sample1 using 1 thread
RuleB Sample2 using 1 thread
...
RuleB SampleN using 1 thread

同时在RuleB样本上执行j。

相反，工作流按以下方式执行：

RuleA Sample1 using j threads
RuleB Sample1 using 1 thread
RuleA Sample2 using j threads
RuleB Sample2 using 1 thread
...

一次在一个样本上执行ruleB。

按照这个顺序执行，ruleB不能被并行化，而且工作流的运行速度比它慢得多。

更具体地说，我想用星号对读到基因组，并使用RNASeQC对它们进行量化。工具RNASEQC是单线程的，而STAR可以在单个示例上处理多个线程。

这将导致sample1中的Snakemake对齐读取，然后使用rnaseqc对它们进行量化，然后在sample2中进行同样的操作。我希望它首先读取所有的示例，然后对它们进行量化(这样，它将能够运行几个单线程rnaseqc工具的实例)。

Snakemake文件的相关摘录如下：

sample_basename = ["RNA-seq_L{}_S{}".format(x, y) for x,y in zip(range(1,41), range(1,41))]
sample_lane = [seq + "_L00{}".format(x) for x in [1, 2] for seq in sample_basename]

rule all:
    input:
        expand("rnaseqc/{s_l}/{s_l}.gene_tpm.gct", s_l=sample_lane)

rule run_star:
    input: 
        index_dir=rules.star_index.output.index_dir, 
        fq1 = "data/fastq/{sample}_R1_001.fastq.gz",
        fq2 = "data/fastq/{sample}_R2_001.fastq.gz",
    output:
        "star/{sample}/{sample}Aligned.sortedByCoord.out.bam",
        "star/{sample}/{sample}Aligned.toTranscriptome.out.bam",
        "star/{sample}/{sample}ReadsPerGene.out.tab",
        "star/{sample}/{sample}Log.final.out"
    log:
        "logs/star/{sample}.log"
    params:
        extra="--quantMode GeneCounts TranscriptomeSAM --chimSegmentMin 20 --outSAMtype BAM SortedByCoordinate",
        sample_name = "{sample}"
    threads: 18 
    script:
        "scripts/star_align.py"

rule rnaseqc:
    input: 
        bam="star/{sample}/{sample}Aligned.sortedByCoord.out.bam",
        gtf="data/gencode.v19.annotation.patched.collapsed.gtf"
    output:
        "rnaseqc/{sample}/{sample}.exon_reads.gct",
        "rnaseqc/{sample}/{sample}.gene_fragments.gct",
        "rnaseqc/{sample}/{sample}.gene_reads.gct",
        "rnaseqc/{sample}/{sample}.gene_tpm.gct",
        "rnaseqc/{sample}/{sample}.metrics.tsv"
    params:
        extra="-s {sample} --legacy",
        output_dir="rnaseqc/{sample}"
    log:
        "logs/rnaseqc/{sample}"
    shell:
        "rnaseqc.v2.3.4.linux {params.extra} {input.gtf} {input.bam} {params.output_dir} 2> {log}"

奇怪的是，使用snakemake -np -j进行一次模拟运行是正确的：

[Mon Oct 21 13:08:11 2019]
rule run_star:
    input: data/STAR/, data/fastq/RNA-seq_L182_S16_L002_R1_001.fastq.gz, data/fastq/RNA-seq_L182_S16_L002_R2_001.fastq.gz
    output: star/RNA-seq_L182_S16_L002/RNA-seq_L182_S16_L002Aligned.sortedByCoord.out.bam, star/RNA-seq_L182_S16_L002/RNA-seq_L182_S16_L002Aligned.toTranscriptome.out.bam, star/RNA-seq_L182_S16_L002/RNA-seq_L182_S16_L002ReadsPerGene.out.tab, star/RNA-seq_L182_S16_L002/RNA-seq_L182_S16_L002Log.final.out
    log: logs/star/RNA-seq_L182_S16_L002.log
    jobid: 1026
    wildcards: sample=RNA-seq_L182_S16_L002
    threads: 18

[Mon Oct 21 13:08:11 2019]
rule run_star:
    input: data/STAR/, data/fastq/RNA-seq_L173_S7_L001_R1_001.fastq.gz, data/fastq/RNA-seq_L173_S7_L001_R2_001.fastq.gz
    output: star/RNA-seq_L173_S7_L001/RNA-seq_L173_S7_L001Aligned.sortedByCoord.out.bam, star/RNA-seq_L173_S7_L001/RNA-seq_L173_S7_L001Aligned.toTranscriptome.out.bam, star/RNA-seq_L173_S7_L001/RNA-seq_L173_S7_L001ReadsPerGene.out.tab, star/RNA-seq_L173_S7_L001/RNA-seq_L173_S7_L001Log.final.out
    log: logs/star/RNA-seq_L173_S7_L001.log
    jobid: 737
    wildcards: sample=RNA-seq_L173_S7_L001
    threads: 18
...
[Mon Oct 21 13:10:50 2019]
rule rnaseqc:
    input: star/RNA-seq_L221_S15_L001/RNA-seq_L221_S15_L001Aligned.sortedByCoord.out.bam, data/gencode.v19.annotation.patched.collapsed.gtf
    output: rnaseqc/RNA-seq_L221_S15_L001/RNA-seq_L221_S15_L001.exon_reads.gct, rnaseqc/RNA-seq_L221_S15_L001/RNA-seq_L221_S15_L001.gene_fragments.gct, rnaseqc/RNA-seq_L221_S15_L001/RNA-seq_L221_S15_L001.gene_reads.gct, rnaseqc/RNA-seq_L221_S15_L001/RNA-seq_L221_S15_L001.gene_tpm.gct, rnaseqc/RNA-seq_L221_S15_L001/RNA-seq_L221_S15_L001.metrics.tsv
    log: logs/rnaseqc/RNA-seq_L221_S15_L001
    jobid: 215
    wildcards: sample=RNA-seq_L221_S15_L001

rnaseqc.v2.3.4.linux -s RNA-seq_L221_S15_L001 --legacy data/gencode.v19.annotation.patched.collapsed.gtf star/RNA-seq_L221_S15_L001/RNA-seq_L221_S15_L001Aligned.sortedByCoord.out.bam rnaseqc/RNA-seq_L221_S15_L001 2> logs/rnaseqc/RNA-seq_L221_S15_L001

[Mon Oct 21 13:10:50 2019]
rule rnaseqc:
    input: star/RNA-seq_L284_S38_L001/RNA-seq_L284_S38_L001Aligned.sortedByCoord.out.bam, data/gencode.v19.annotation.patched.collapsed.gtf
    output: rnaseqc/RNA-seq_L284_S38_L001/RNA-seq_L284_S38_L001.exon_reads.gct, rnaseqc/RNA-seq_L284_S38_L001/RNA-seq_L284_S38_L001.gene_fragments.gct, rnaseqc/RNA-seq_L284_S38_L001/RNA-seq_L284_S38_L001.gene_reads.gct, rnaseqc/RNA-seq_L284_S38_L001/RNA-seq_L284_S38_L001.gene_tpm.gct, rnaseqc/RNA-seq_L284_S38_L001/RNA-seq_L284_S38_L001.metrics.tsv
    log: logs/rnaseqc/RNA-seq_L284_S38_L001
    jobid: 278
    wildcards: sample=RNA-seq_L284_S38_L001

但是，如果不使用snakemake -j标志，则不会执行-np。

[Mon Oct 21 13:13:49 2019]
rule run_star:
    input: data/STAR/, data/fastq/RNA-seq_L249_S3_L001_R1_001.fastq.gz, data/fastq/RNA-seq_L249_S3_L001_R2_001.fastq.gz
    output: star/RNA-seq_L249_S3_L001/RNA-seq_L249_S3_L001Aligned.sortedByCoord.out.bam, star/RNA-seq_L249_S3_L001/RNA-seq_L249_S3_L001Aligned.toTranscriptome.out.bam, star/RNA-seq_L249_S3_L001/RNA-seq_L249_S3_L001ReadsPerGene.out.tab, star/RNA-seq_L249_S3_L001/RNA-seq_L249_S3_L001Log.final.out
    log: logs/star/RNA-seq_L249_S3_L001.log
    jobid: 813
    wildcards: sample=RNA-seq_L249_S3_L001
    threads: 18

Aligning RNA-seq_L249_S3_L001
[Mon Oct 21 13:21:33 2019]
Finished job 813.
2 of 478 steps (0.42%) done

[Mon Oct 21 13:21:33 2019]
rule rnaseqc:
    input: star/RNA-seq_L249_S3_L001/RNA-seq_L249_S3_L001Aligned.sortedByCoord.out.bam, data/gencode.v19.annotation.patched.collapsed.gtf
    output: rnaseqc/RNA-seq_L249_S3_L001/RNA-seq_L249_S3_L001.exon_reads.gct, rnaseqc/RNA-seq_L249_S3_L001/RNA-seq_L249_S3_L001.gene_fragments.gct, rnaseqc/RNA-seq_L249_S3_L001/RNA-seq_L249_S3_L001.gene_reads.gct, rnaseqc/RNA-seq_L249_S3_L001/RNA-seq_L249_S3_L001.gene_tpm.gct, rnaseqc/RNA-seq_L249_S3_L001/RNA-seq_L249_S3_L001.metrics.tsv
    log: logs/rnaseqc/RNA-seq_L249_S3_L001
    jobid: 243
    wildcards: sample=RNA-seq_L249_S3_L001

我正在使用通过Conda: 5.5.2提供的Snakemake的最新版本

Answer 1

也许您要寻找的是，与运行rnaseqc的规则相比，给予规则运行星星更高的优先级。如果是这样，请查看优先事项指令，例如：

rule star:
    priority: 50
    ...

rule rnaseqc:
    priority: 0
    ...

(未测试)这应该首先运行所有的明星作业，一次一个，因为他们需要18个核心，然后所有的rnaseqc作业并行运行。