Triplet-based species tree estimation: Sensitivity to gene tree rooting (or lack thereof)

Species tree estimation often involves phylogenomic methodologies that incorporate multiple genes sampled across the genome. However, gene tree heterogeneity (discordance), often attributed to Incomplete Lineage Sorting (ILS), presents a significant challenge to accurate species tree inference. Triplet and quartet-based species tree estimation methods have attracted considerable attention owing to their provable statistical consistency in the presence of ILS. Yet, the adoption of rooted triplet-based approaches in the systematics community has been restricted, largely due to their limitations in handling unrooted gene trees—an issue not faced by quartet-based methods. Since the root placement directly influences the distribution of induced triplets in a gene tree, the accuracy of triplet-based methods is dependent on gene tree rooting accuracy. While there is extensive research on approaches for rooting unrooted gene trees, the choice of rooting techniques and their implications for triplet-based species tree estimation on realistic model conditions are greatly understudied. In this study, we carry out an extensive empirical analysis of different gene tree rooting strategies to assess the effects of rooting on triplet-based species tree inference. Across simulated and empirical datasets, triplet-based estimation using STELAR is generally robust to gene tree rooting, with several algorithmic rooting strategies yielding performance comparable to, and in some model conditions exceeding, that of STELAR on outgroup-rooted gene trees and the widely used quartet-based method ASTRAL. Our results highlight different model conditions in which rooting choices positively influence triplet-based species tree inference, thereby providing new insights into their potential utility in phylogenomic analyses.Author summary: Reconstructing the evolutionary history of species from genomic data is complicated by the fact that different genes can tell different evolutionary stories – a problem known as gene tree discordance. To address this, researchers use methods that identify consistent patterns across many gene trees, comparing either sets of four (quartets) or three species (triplets). Triplet-based methods have seen less adoption partly because they require gene trees to be “rooted” – meaning the direction of evolution must be specified – while quartet-based methods do not. Rooting is typically done using an outgroup, but a reliable one isn’t always available. In this study, we conducted an extensive empirical evaluation of how different rooting strategies affect STELAR – a triplet-based species tree estimation method – across various simulated and biological datasets, spanning diverse model conditions and including varying levels of gene tree discordance, gene counts, and sequence lengths. We found that STELAR is generally robust to the choice of rooting method – algorithmic rooting approaches often performed comparably to, and sometimes better than, outgroup rooting or the popular quartet-based method ASTRAL. These findings suggest that the lack of a reliable outgroup need not be a barrier to using triplet-based methods and open new avenues for improving their performance in phylogenomic studies.