Interaction-Driven Topological Phase Diagram of Twisted Bilayer MoTe2

Abstract

Twisted bilayer MoTe2 is a promising platform to investigate the interplay between band topology and many-body interactions. We present a theoretical study of its interaction-driven quantum phase diagrams based on a three-orbital model, which can be viewed as a generalization of the Kane-Mele-Hubbard model with one additional orbital and long-range Coulomb repulsion. We predict a cascade of phase transitions tuned by the twist angle θ. At the hole-filling factor ν=1 (one hole per moiré unit cell), the ground state can be in the multiferroic phase, with coexisting spontaneous layer polarization and magnetism; the quantum anomalous Hall phase; and finally, the topologically trivial magnetic phases, as θ increases from 1.5° to 5°. At ν=2, the ground state can have a second-order phase transition between an antiferromagnetic phase and the quantum spin Hall phase as θ passes through a critical value. The dependence of the phase boundaries on model parameters, such as the gate-to-sample distance, the dielectric constant, and the moiré potential amplitude, is examined. The predicted phase diagrams can guide the search for topological phases in twisted transition metal dichalcogenide homobilayers.