Please explain me why i am getting this error in snakemake? I´ve been strugling for days please advise me on whats going wrong
Question:
I wrote this pipeline in snakemake to process my fastq files and get the raw counts but for some reason that I don’t understand in the last rule (featurecounts) I get this error:
WildcardError in line 175 of /mnt/c/Users/manso/Desktop/hel/pe.py:
Wildcards in input files cannot be determined from output files: ‘sample’
Other rules use the same input as featureCounts rule so I don’t understand why it returns this error for that specific rule.
I’d really appreciate your help.
Here is my snakefile:
(SAMPLE,FRR) = glob_wildcards("rawReads/{sample}_{frr}.fastq.gz")
rule all:
input:
#raw_FASTQC
expand("rawQC/fastqc/{sample}_{frr}_fastqc.html", sample=SAMPLE, frr=FRR),
expand("rawQC/fastqc/{sample}_{frr}_fastqc.zip", sample=SAMPLE, frr=FRR),
#raw_MultiQC
"rawQC/multiqc_report.html",
#FASTP
expand("trimmedReads/{sample}_1.fastq.gz", sample=SAMPLE),
expand("trimmedReads/{sample}_2.fastq.gz", sample=SAMPLE),
expand("trimmedReads/{sample}_fastp_report.html", sample=SAMPLE),
#trimmed_FASTQC
expand("trimmedQC/fastqc/{sample}_{frr}_fastqc.html", sample=SAMPLE, frr=FRR),
expand("trimmedQC/fastqc/{sample}_{frr}_fastqc.zip", sample=SAMPLE, frr=FRR),
#trimmed_MultiQC
"trimmedQC/multiqc_report.html",
#get fa and gtf files
"genome/Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa",
"genome/Homo_sapiens.GRCh38.106.gtf.gz",
#HISAT2_index
["index." + str(i) + ".ht2" for i in range(1,9)],
#HISAT_align
expand("aligned/{sample}.bam", sample=SAMPLE),
#samtools
expand("aligned/{sample}.sorted.bam", sample=SAMPLE),
expand("samtools_stats/{sample}.stats.txt", sample=SAMPLE),
expand("samtools_stats/{sample}.flagstat.txt", sample=SAMPLE),
#rawCounts
"raw_Counts"
rule raw_FASTQC:
input:
"rawReads/{sample}_{frr}.fastq.gz",
output:
html="rawQC/fastqc/{sample}_{frr}_fastqc.html",
zip= "rawQC/fastqc/{sample}_{frr}_fastqc.zip", # the suffix _fastqc.zip is necessary for multiqc to find the file. If not using multiqc, you are free to choose an arbitrary filename
params: "--quiet"
log:
"logs/fastqc/{sample}_{frr}.log"
threads: 16
wrapper:
"v1.7.0/bio/fastqc"
rule raw_MultiQC:
input:
expand("rawQC/fastqc/{sample}_{frr}_fastqc.zip", sample=SAMPLE, frr=FRR),
params:
path="rawQC/fastqc"
output:
"rawQC/multiqc_report.html"
shell:
"multiqc --force -n {output} {params.path}"
rule FASTP:
input:
read1="rawReads/{sample}_1.fastq.gz",
read2="rawReads/{sample}_2.fastq.gz",
output:
trimmed1="trimmedReads/{sample}_1.fastq.gz",
trimmed2="trimmedReads/{sample}_2.fastq.gz",
report_html= "trimmedReads/{sample}_fastp_report.html",
threads: 16
shell:
" fastp --thread {threads} -i {input.read1} -I {input.read2} -o {output.trimmed1} -O {output.trimmed2} -h {output.report_html} "
rule trimmed_FASTQC:
input:
"trimmedReads/{sample}_{frr}.fastq.gz"
output:
html="trimmedQC/fastqc/{sample}_{frr}_fastqc.html",
zip="trimmedQC/fastqc/{sample}_{frr}_fastqc.zip", # the suffix _fastqc.zip is necessary for multiqc to find the file. If not using multiqc, you are free to choose an arbitrary filename
params: "--quiet"
log:
"logs/fastqc/{sample}_{frr}.log"
threads: 16
wrapper:
"v1.7.0/bio/fastqc"
rule trimmed_MultiQC:
input:
expand("trimmedQC/fastqc/{sample}_{frr}_fastqc.zip", sample=SAMPLE, frr=FRR),
params:
path="trimmedQC/fastqc"
output:
"trimmedQC/multiqc_report.html"
shell:
"multiqc --force -n {output} {params.path} "
#Get annotation GTF
rule get_genome_gtf:
"Downloading Genome annotation file from Ensemble, Homo sapiens primary assembly (GRCh38)"
output:
gtf = "genome/Homo_sapiens.GRCh38.106.gtf.gz"
shell:
"cd genome"
" && wget ftp://ftp.ensembl.org/pub/release-106/gtf/homo_sapiens/Homo_sapiens.GRCh38.106.gtf.gz"
" && gunzip -k Homo_sapiens.GRCh38.106.gtf.gz "
# Get genome fa
rule get_genome_fa:
"Downloading Genome sequence, Homo sapiens primary assembly (GRCh38)"
output:
fa = "genome/Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa"
shell:
"cd genome"
" && wget ftp://ftp.ensembl.org/pub/release-106/fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa.gz"
" && gunzip -k Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa "
rule HISAT2_index:
input:
fa = rules.get_genome_fa.output.fa
output:
["index." + str(i) + ".ht2" for i in range(1,9)],
message:
"indexing genome"
threads: 16
shell:
" hisat2-build -p {threads} {input.fa} index --quiet"
rule HISAT2_align:
input:
read1=rules.FASTP.output.trimmed1,
read2=rules.FASTP.output.trimmed2,
index=rules.HISAT2_index.output
output:
bam="aligned/{sample}.bam",
metrics="logs/{sample}_HISATmetrics.txt"
threads: 16
shell:
" hisat2 --threads {threads} -x index -1 {input.read1} -2 {input.read2} 2> {output.metrics}"
" | samtools view -Sbh -o {output.bam} "
rule samtools_sort:
input:
aligned=rules.HISAT2_align.output.bam
#"aligned/{sample}.bam"
output:
"aligned/{sample}.sorted.bam"
threads: 8
shell:
"samtools sort {input.aligned} -o {output}"
rule samtools_stats:
input:
"aligned/{sample}.sorted.bam",
output:
"samtools_stats/{sample}.stats.txt",
shell:
"samtools stats {input} > {output} "
rule samtools_flagstat:
input:
"aligned/{sample}.sorted.bam",
output:
"samtools_stats/{sample}.flagstat.txt",
shell:
"samtools flagstat {input} > {output} "
rule featureCounts:
input:
samples="aligned/{sample}.sorted.bam",
gtf=rules.get_genome_gtf.output.gtf
output:
"raw_Counts"
threads:
16
shell:
"featureCounts -T {threads} -a {input.gtf} -o {output} {input.samples}"
ยดยดยด
Answers:
Snakemake uses pattern in the output to infer which inputs to use. In the last rule, output is raw_Counts, which gives no indication of what to use for {sample} wildcard. Changing it to something like this might work for your use case:
rule featureCounts:
input:
samples="aligned/{sample}.sorted.bam",
gtf=rules.get_genome_gtf.output.gtf
output:
"raw_Counts_{sample}.txt"
This will require adding the expanded version to rule all:
# add this target to rule all
expand("raw_Counts_{sample}.txt", sample=SAMPLE),
Edit: if this rule is intended as an aggregate, then in the input directive you will want to remove wildcard search by substituting all the values.
rule featureCounts:
input:
samples=expand("aligned/{sample}.sorted.bam", sample=SAMPLE),
gtf=rules.get_genome_gtf.output.gtf
output:
"raw_Counts"
Edit 2: note that glob_wildcards does not return unique values for each wildcard, but rather the wildcards associated with each globbed file. If you want unique values, then one easy way to achieve that is to convert SAMPLE to a set (specifically for this rule).
rule featureCounts:
input:
samples=expand("aligned/{sample}.sorted.bam", sample=set(SAMPLE)),
gtf=rules.get_genome_gtf.output.gtf
output:
"raw_Counts"
I wrote this pipeline in snakemake to process my fastq files and get the raw counts but for some reason that I don’t understand in the last rule (featurecounts) I get this error:
WildcardError in line 175 of /mnt/c/Users/manso/Desktop/hel/pe.py:
Wildcards in input files cannot be determined from output files: ‘sample’
Other rules use the same input as featureCounts rule so I don’t understand why it returns this error for that specific rule.
I’d really appreciate your help.
Here is my snakefile:
(SAMPLE,FRR) = glob_wildcards("rawReads/{sample}_{frr}.fastq.gz")
rule all:
input:
#raw_FASTQC
expand("rawQC/fastqc/{sample}_{frr}_fastqc.html", sample=SAMPLE, frr=FRR),
expand("rawQC/fastqc/{sample}_{frr}_fastqc.zip", sample=SAMPLE, frr=FRR),
#raw_MultiQC
"rawQC/multiqc_report.html",
#FASTP
expand("trimmedReads/{sample}_1.fastq.gz", sample=SAMPLE),
expand("trimmedReads/{sample}_2.fastq.gz", sample=SAMPLE),
expand("trimmedReads/{sample}_fastp_report.html", sample=SAMPLE),
#trimmed_FASTQC
expand("trimmedQC/fastqc/{sample}_{frr}_fastqc.html", sample=SAMPLE, frr=FRR),
expand("trimmedQC/fastqc/{sample}_{frr}_fastqc.zip", sample=SAMPLE, frr=FRR),
#trimmed_MultiQC
"trimmedQC/multiqc_report.html",
#get fa and gtf files
"genome/Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa",
"genome/Homo_sapiens.GRCh38.106.gtf.gz",
#HISAT2_index
["index." + str(i) + ".ht2" for i in range(1,9)],
#HISAT_align
expand("aligned/{sample}.bam", sample=SAMPLE),
#samtools
expand("aligned/{sample}.sorted.bam", sample=SAMPLE),
expand("samtools_stats/{sample}.stats.txt", sample=SAMPLE),
expand("samtools_stats/{sample}.flagstat.txt", sample=SAMPLE),
#rawCounts
"raw_Counts"
rule raw_FASTQC:
input:
"rawReads/{sample}_{frr}.fastq.gz",
output:
html="rawQC/fastqc/{sample}_{frr}_fastqc.html",
zip= "rawQC/fastqc/{sample}_{frr}_fastqc.zip", # the suffix _fastqc.zip is necessary for multiqc to find the file. If not using multiqc, you are free to choose an arbitrary filename
params: "--quiet"
log:
"logs/fastqc/{sample}_{frr}.log"
threads: 16
wrapper:
"v1.7.0/bio/fastqc"
rule raw_MultiQC:
input:
expand("rawQC/fastqc/{sample}_{frr}_fastqc.zip", sample=SAMPLE, frr=FRR),
params:
path="rawQC/fastqc"
output:
"rawQC/multiqc_report.html"
shell:
"multiqc --force -n {output} {params.path}"
rule FASTP:
input:
read1="rawReads/{sample}_1.fastq.gz",
read2="rawReads/{sample}_2.fastq.gz",
output:
trimmed1="trimmedReads/{sample}_1.fastq.gz",
trimmed2="trimmedReads/{sample}_2.fastq.gz",
report_html= "trimmedReads/{sample}_fastp_report.html",
threads: 16
shell:
" fastp --thread {threads} -i {input.read1} -I {input.read2} -o {output.trimmed1} -O {output.trimmed2} -h {output.report_html} "
rule trimmed_FASTQC:
input:
"trimmedReads/{sample}_{frr}.fastq.gz"
output:
html="trimmedQC/fastqc/{sample}_{frr}_fastqc.html",
zip="trimmedQC/fastqc/{sample}_{frr}_fastqc.zip", # the suffix _fastqc.zip is necessary for multiqc to find the file. If not using multiqc, you are free to choose an arbitrary filename
params: "--quiet"
log:
"logs/fastqc/{sample}_{frr}.log"
threads: 16
wrapper:
"v1.7.0/bio/fastqc"
rule trimmed_MultiQC:
input:
expand("trimmedQC/fastqc/{sample}_{frr}_fastqc.zip", sample=SAMPLE, frr=FRR),
params:
path="trimmedQC/fastqc"
output:
"trimmedQC/multiqc_report.html"
shell:
"multiqc --force -n {output} {params.path} "
#Get annotation GTF
rule get_genome_gtf:
"Downloading Genome annotation file from Ensemble, Homo sapiens primary assembly (GRCh38)"
output:
gtf = "genome/Homo_sapiens.GRCh38.106.gtf.gz"
shell:
"cd genome"
" && wget ftp://ftp.ensembl.org/pub/release-106/gtf/homo_sapiens/Homo_sapiens.GRCh38.106.gtf.gz"
" && gunzip -k Homo_sapiens.GRCh38.106.gtf.gz "
# Get genome fa
rule get_genome_fa:
"Downloading Genome sequence, Homo sapiens primary assembly (GRCh38)"
output:
fa = "genome/Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa"
shell:
"cd genome"
" && wget ftp://ftp.ensembl.org/pub/release-106/fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa.gz"
" && gunzip -k Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa "
rule HISAT2_index:
input:
fa = rules.get_genome_fa.output.fa
output:
["index." + str(i) + ".ht2" for i in range(1,9)],
message:
"indexing genome"
threads: 16
shell:
" hisat2-build -p {threads} {input.fa} index --quiet"
rule HISAT2_align:
input:
read1=rules.FASTP.output.trimmed1,
read2=rules.FASTP.output.trimmed2,
index=rules.HISAT2_index.output
output:
bam="aligned/{sample}.bam",
metrics="logs/{sample}_HISATmetrics.txt"
threads: 16
shell:
" hisat2 --threads {threads} -x index -1 {input.read1} -2 {input.read2} 2> {output.metrics}"
" | samtools view -Sbh -o {output.bam} "
rule samtools_sort:
input:
aligned=rules.HISAT2_align.output.bam
#"aligned/{sample}.bam"
output:
"aligned/{sample}.sorted.bam"
threads: 8
shell:
"samtools sort {input.aligned} -o {output}"
rule samtools_stats:
input:
"aligned/{sample}.sorted.bam",
output:
"samtools_stats/{sample}.stats.txt",
shell:
"samtools stats {input} > {output} "
rule samtools_flagstat:
input:
"aligned/{sample}.sorted.bam",
output:
"samtools_stats/{sample}.flagstat.txt",
shell:
"samtools flagstat {input} > {output} "
rule featureCounts:
input:
samples="aligned/{sample}.sorted.bam",
gtf=rules.get_genome_gtf.output.gtf
output:
"raw_Counts"
threads:
16
shell:
"featureCounts -T {threads} -a {input.gtf} -o {output} {input.samples}"
ยดยดยด
Snakemake uses pattern in the output to infer which inputs to use. In the last rule, output is raw_Counts, which gives no indication of what to use for {sample} wildcard. Changing it to something like this might work for your use case:
rule featureCounts:
input:
samples="aligned/{sample}.sorted.bam",
gtf=rules.get_genome_gtf.output.gtf
output:
"raw_Counts_{sample}.txt"
This will require adding the expanded version to rule all:
# add this target to rule all
expand("raw_Counts_{sample}.txt", sample=SAMPLE),
Edit: if this rule is intended as an aggregate, then in the input directive you will want to remove wildcard search by substituting all the values.
rule featureCounts:
input:
samples=expand("aligned/{sample}.sorted.bam", sample=SAMPLE),
gtf=rules.get_genome_gtf.output.gtf
output:
"raw_Counts"
Edit 2: note that glob_wildcards does not return unique values for each wildcard, but rather the wildcards associated with each globbed file. If you want unique values, then one easy way to achieve that is to convert SAMPLE to a set (specifically for this rule).
rule featureCounts:
input:
samples=expand("aligned/{sample}.sorted.bam", sample=set(SAMPLE)),
gtf=rules.get_genome_gtf.output.gtf
output:
"raw_Counts"