Day 3¶
Full Cow Dataset Pipeline
Today is the all-cow day: a guided, hands-on run-through of every step you learned across Days 1–2, applied end-to-end to the cow dataset. You will not see new concepts here, instead you'll consolidate the workflow by driving it yourself from raw reads through diversity and differential abundance, with two instructor-led check-ins along the way.
How to use this page
Each Part below corresponds to a Part in the cow_site_workflow.txt where you will complete the commands before running them on Alpine. If you get stuck on the why behind a step, jump back to the corresponding Day 1 or Day 2 page (linked in each phase). The ⏸ Check-in sections are deliberate pause points, wait for the group before proceeding.
Setup¶
Request an interactive session and load QIIME2:
sinteractive --reservation=microbiome --time=04:00:00 --partition=amilan --nodes=1 --ntasks=6 --qos=normal
Make a fresh working directory for the cow run:
Move into the scratch directory first
Cow data location
The cow_site_workflow.txt file can be dowloaded below, it contains all the commands you will need to edit to work through the tutorial. The reads are currently in a zipped directory, you will need to download the zipped directory upload it into alpine and then unzip the directory. The metadata and barcodes file can be downloaded below, then you can upload them into alpine into their respective directory.
Gather the files you will need for this activity¶
1. cow_site_workflow.txt: we have provided the outline of the whole workflow for you to work with. Some lines will have blank values for you to input the correct values/file names in order to make the commands work.
Download the cow_site_workflow.txt file:
Download the cow tutorial workflow text file
You can upload this file into alpine and used the alpine file editor to edit the file as you work through the tutorial. Or you can open it up in VS code (or any other plain text editor) and edit it that way as you move throught the tutorial
2. Get the raw reads for this dataset
Copy the raw_reads to your cow directory:
3. Download the cow_metadata.txt, cow_metadata_nocontrols.txt, and the barcodes file cow_barcodes.txt (so that you can demultiplex the reads)
Download metadata files and then follow instructions in the cow_site_workflow.txt:
Download the barcode file here:
Download the cow tutorial barcode file
Download the metadata here:
Download the cow tutorial metadata file
Download the metadata without controls here (you will need this for ANCOM-BC2):
Download the cow tutorial metadata file with no controls
Once you have submitted your tree script you can use this command in the tree directory to get the finished tree and tree placements .qza files
Tree
Tree placements
Background: Cow Dataset Overview¶
20 adult dairy cattle were sampled (via swabbing) to determine the microbial community composition between 5 different body sites (nasal/skin/udder/fecal/oral). We want you to determine if and how the microbial composition changes between sites.
We will first begin by downloading the raw sequencing data from the workshop website and then upload those files onto Alpine into a new directory (that you create).
You can open the metadata file and the manifest in a text editor and skim them so you know what you're working with.
Part 1: import, demux, and denoising¶
On your own using the cow_site_workflow.txt file, begin the workflow for the cow dataset by importing the raw reads, demultiplexing the reads, and denoising:
Reference: Day 1: Importing Sequences, Reference Day 1: Denoising
⏸ Check-in #1¶
Group regroup. Bring your demux_cow.qzv, dada2_stats_cow.qzv, and table_cow.qzv open.
Stop and Think
1. What is the mean reads per sample?
2. How long are the reads?
3. What is the maximum length of all your sequences?
4. Which sample (not including extraction controls starting with EC) lost the highest % of reads?
5. Why did you chose to trim or truncate where you did?
This is also the right time to flag any errors from Phase 1 before they propagate.
Part 1b: removing long amplicons¶
Here we have a new step that we didn't need to use in the decomp tutorial
"Filter long amplicons before classification" After DADA2, filter representative sequences to your expected V4 amplicon length range before taxonomy:
qiime feature-table filter-seqs \
--i-data cow_seqs_dada2.qza \
--m-metadata-file seqs_cow.qza \
--p-where 'length(sequence) < 300' \
--o-filtered-data cow_seqs_dada2_filtered300.qza
Then you will use this filtered sequences table for subsequent analyses
Part 2: Taxonomy & generating a pylogenetic tree¶
On your own using the cow_site_workflow.txt file, continue the workflow for the cow dataset to assign taxonomy to your ASVs and build a phylogenetic tree.
Day 1: Taxonomic Classification
⏸ Check-in #2: Taxonomy, Phylogenetic Tree¶
Bringtaxa_barplot_nomitochloro_gg2_filtered300.qzvopen.
Stop and Think
1. Look at your taxa bar plot, organize it by cow sampling location (body_site) at the level 7 taxonomic level. What general trends do you notice?
2. What highly abundant ASV is shared between both the udder and skin samples?
3 Go to the class level, what are the top 2 most abundant bacterial classes in the fecal samples?
4 Which samples (still sorted by body_site) have higher alpha diversity in terms of observed features just by looking at the taxa plots?
5 Do all samples contain archaea as well?
6 Why do we filter out sp004296775?
7. Do the positive controls look the same as each other? Yes or No?
8. Do the negative/extraction controls (Samples labeled as EC), look like the positive controls? Yes or no?
9. Do the negative/extraction controls (Samples labeled as EC), look like the real samples? Yes or no?
Part 3: Rarefaction & Alpha and Beta Diversity.¶
On your own using the cow_site_workflow.txt file, continue the workflow for the cow dataset to generate the core metrics using rarefaction.
Reference: Day 2: Rarefaction, Day 2: Alpha Diversity, Day 2: Beta Diversity
⏸ Check-in #3: Rarefaction, Alpha and Beta Diversity¶
Bringobserved_features_statistics.qzv, faiths_pd_statistics.qzv, and emperor.qzv
Stop and Think
1. What is the name of the file you needed to use to figure out what min and max depths to use to generate the alpha rarefaction plot? (Hint: which file contains the sequencing depths for each sample)
2. What did you chose for the rarefaction depth (the input for core metrics -p-sampling-depth flag)? why?
3. Which cow body location had more observed features?
4. Which has the lowest?
5. What is the main difference between Faiths PD and Shannons alpha diversity metrics?
6. Which two body sites have the highest Faiths PD alpha diversity? Are the groups significantly different?
7. Which diversity metrics produced by the core-metrics pipeline require phylogenetic information?
8. Does it seem like there are any groupings in the beta diversity? What are the groupings?
9. Why do you think these samples are grouping together?
10. What test can you run to determine if the groups are significantly different?
11. What command would you use to run that test?
Part 4: ANCOM-BC2 & Machine Learning.¶
On your own using the cow_site_workflow.txt file, continue the workflow for the cow dataset to run ANCOM-BC2 and build a machine learning model.
Reference: Day 2: ANCOM-BC1, Reference: Day 2: Machine Learning,
⏸ Check-in #4: ANCOM-BC2 & Machine Learning¶
Bringancombc2_barplot_bodysite_genus.qzv and machine learning results
Stop and Think
1. When generating the filtered table for ANCOM-BC2, what value did you choose for --p-min-frequency? Which core metrics parameter should this match, and why do these values need to be the same? (Report your core metrics value here: ___)
2. Why do we filter out samples with low frequency and low abundance ASVs?
3. What was the most enriched genus in skin compared to fecal, and what was the most depleted genus in skin compared to fecal (make sure adjusted p is set to less than 0.05)?
4. Why might removing controls be important before downstream analysis?
5. What 2 features that are high in fecal samples?
6. What are 2 features that are low in nasal?
7. What is the accuracy of your model, and if the accuracy of the classifier is high, what does that suggest about the microbial compositions of each site?
Optional part 5: use R to run stats and make pretty figures¶
R Section
There is a section at the end of the cow_site_workflow.txt with blanks to fill in that correspond to an R markdown that you can work through using the cow body site data on your own after the workshop. The cow R tutorial materials are below. The material in the cow R tutrial are not compatible with the material for the decomp R tutorial in the resources tab, these are two seperate R tutorials. The decomp R tutorial goes more in depht, you can run through the cow body site R tutorial after the decomp R tutorial for extra practice. To get started you will need to download the cow_r directory, move it to where you want to work on it, then unzip the directory using unzip cow_r.zip. Then the R markdown you will work through is in the code directory in 04_code directory. The directory set up is the same as the decomp R tutorial.
Download the cow R tutorial directory
✰ Wrap-up: Results Review & Troubleshooting¶
Group regroup. Bring all your .qzv files open in view.qiime2.org. We'll work through:
- What does the cow data actually show? Compare across tables.
- Where did people make different parameter choices, and how did that change the result?
- Common pitfalls hit during the day — what would you do differently next time?
Open Q&A & Workshop Wrap-up¶
Free time with all instructors. Bring your own data questions, debugging issues, or anything from earlier in the workshop you want to revisit.
This completes the workshop. See Resources for downloads, the glossary, and the Publication Plots in R self-study reference.
Before you head out, please take five minutes for the Post-Workshop Survey — we'll share the password to open it at the end of the workshop.