Finch: a tool adding dynamic abundance filtering to genomic MinHashing

Abstract

MinHash (Broder 1997) is a document similarity estimation technique that has been applied to problems in genomics including sequence search, phylogenetic reconstruction (Ondov et al. 2016; Brown and Irber 2016), and evaluating outbreaks of hospital ac-quired infections (HAIs) (Sim et al. 2017). We developed the finch-rs library (https: //github.com/onecodex/finch-rs) and finch command line tool for creating, filtering, and manipulating MinHash sketches from genomics data, including both FASTA sequence files and FASTQ raw read data from next-generation sequencing (NGS) instruments. We extend existing MinHash schemes for genomics data with two major additions: (1) calcu-lation of abundances (i.e., minmer counts) during the generation of the MinHash sketches; and (2) adaptive correction of biases introduced due to variable sequencing depths. These features greatly improve the utility of MinHashing when applied directly to raw read data (i.e., FASTQ files) and allows more robust estimation between both isolates and complex metagenomic samples. Finch and similar genomic MinHashing software works by breaking sequence data up into k-length nucleotide or amino acid subsequences (" k-mers "), computing a hash of each k-mer, and then taking the n lowest hash values. Collectively these n smallest values (" minmers ") comprise a " sketch " of the input sample. By default, previous MinHash implementations for genomics data work by creating sketches from all k-mers from an input genomic dataset (though the original Mash tool does enable filtering out k-mers that appear only once using a Bloom filter (Ondov et al. 2016)). While this works well for high-quality sequences such as genome assemblies (i.e., FASTA files), it quickly becomes problematic when working with raw FASTQ data where errors from NGS instruments can lead to a far larger number of unique observed k-mers than are truly present biologically. Similarly, this also leads to the inclusion of sequencing errors and k-mers from minor community members when comparing complex, mixed genomic samples (i.e., microbiome samples). In both cases, non-representative k-mers (either direct products of sequencing error or low abundance organisms) come to dominate sketches and confound inter-sample distance estimates.