Benchmark
XspecT is a tool designed for fast and accurate species classification of genome assemblies and simulated reads. To evaluate its classification accuracy, we conducted a benchmark using a set of Acinetobacter genomes.
The benchmark was performed by first download all available Acinetobacter genomes from Genbank, filtered on a passed ("OK") taxonomy check status. Genomes assigned to strain IDs were remapped to their respective species IDs, after which genomes with species IDs not contained in XspecT's Acinetobacter model were removed. The remaining genomes were then used to classify both assemblies and simulated reads generated from them. Simulated reads were generated by first filtering on genomes that were not part of the training data and that were categorized as "complete" by NCBI. The reads were then simulated from the longest contig of each genome (assumed to be the chromosome) using a custom Python script. Up to three genomes were selected per species. 100 000 reads were simulated for each genome, with a read length of 100 bp and no simulated sequencing errors. The reads were then classified using XspecT with predictions based on the maximum-scoring species.
Benchmark Results
The benchmark results show that XspecT achieves high classification accuracy, with an overall accuracy of 99.94% for whole genomes and 87.11% for simulated reads.
| Category | Total | Matches | Mismatches | Match Rate | Mismatch Rate |
|---|---|---|---|---|---|
| Assemblies | 44,905 | 44,879 | 26 | 99.94% | 0.06% |
| Simulated reads | 9,000,000 | 7,839,877 | 1,160,123 | 87.11% | 12.89% |
Running the benchmark yourself
To benchmark XspecT performance yourself, you can use the Nextflow workflow provided in the scripts/benchmark directory. This workflow allows you to run XspecT on a set of samples and measure species classification accuracy on both genome assemblies, as well as on simulated reads.
Before you run the benchmark, you first need to download benchmarking data to the data directory, for example from NCBI. To do so, you can use the bash script in scripts/benchmark-data to download the data using the NCBI Datasets CLI, which needs to be installed first. The script will download all available Acinetobacter genomes, as well as taxonomic data.
To run the benchmark, install Nextflow and run the following command:
nextflow run scripts/benchmark
This will execute the benchmark workflow, which will classify the samples, as well as reads generated from them, using XspecT. The results will be saved in the results directory:
results/classifications.tsvfor the classifications of the assembliesresults/read_classifications.tsvfor the classifications of the simulated readsresults/confusion_matrix.pngfor the confusion matrix of genome assembly classificationsresults/mismatches_confusion_matrix.pngfor a confusion matrix filtered on mismatches of genome assembly classificationsresults/stats.txtfor the statistics of the benchmark run