Molecular Genetic Approaches to the Study of Primate Behavior, Social Organization, and Reproduction Anthony Di Fiore Department of Anthropology, New York University and New York Consortium in Evolutionary Primatology, New York, New York 10003 KEY WORDS molecular ecology population structure PCR DNA sequencing microsatellite noninvasive sampling dispersal mating system reproductive strategy behavior genetics ABSTRACT In the past several decades, the develop- ment of novel molecular techniques and the advent of noninvasive DNA sampling, coupled with the ease and speed with which molecular analyses can now be per- formed, have made it possible for primatologists to di- rectly examine the fitness effects of individual behavior and to explore how variation in behavior and social sys- tems influences primate population genetic structure. This review describes the theoretical connections between individual behavior and primate social systems on the one hand and population genetic structure on the other, dis- cusses the kinds of molecular markers typically employed in genetic studies of primates, and summarizes what pri- matologists have learned from molecular studies over the past few decades about dispersal patterns, mating sys- tems, reproductive strategies, and the influence of kinship on social behavior. Several important conclusions can be drawn from this overview. First, genetic data confirm that, in many species, male dominance rank and fitness are positively related, at least over the short term, though this relationship need not simply be a reflection of male- male contest competition over mates. More importantly, genetic research reveals the significance of female choice in determining male reproductive success, and documents the efficacy of alternative mating tactics among males. Second, genetic data suggest that the presumed impor- tance of kinship in structuring primate social relation- ships needs to be evaluated further, at least for some taxa such as chimpanzees in which demographic factors may be more important than relatedness. I conclude this paper by offering several suggestions of additional ways in which molecular techniques might be employed in behavioral and ecological studies of primates (e.g., for conducting ���molecular censuses��� of unhabituated populations, for studying disease and host-parasite interactions, or for tracking seed fate in studies of seed dispersal) and by providing a brief introduction to the burgeoning field of nonhuman primate behavioral genetics. Yrbk Phys An- thropol 46:62���99, 2003. �� 2003 Wiley-Liss, Inc. INTRODUCTION Observational studies of the behavior, ecology, and social organization of primates in their natural environments have contributed substantially to our understanding of mammalian social systems and their evolution. Nonetheless, even in the most com- plete long-term studies of wild primate populations, it is difficult to fully elucidate certain features of social systems such as dispersal patterns, patterns of within-group relatedness, and the effective ge- netic mating system. Nor is it possible through ob- servational studies alone to fully evaluate the effect of kinship on shaping patterns of social behavior or to examine the link between individual behavior (e.g., dominance interactions, alternative mating tactics) and reproductive success. All of these topics, however, have direct relevance to understanding the evolution of primate social systems, especially since some of the fundamental models forwarded to ex- plain the evolution of primate sociality take either male-male or female-female kinship as a point of departure for considering the evolutionary conse- quences of cooperative and competitive behaviors. In the past several decades, the development of novel molecular techniques (e.g., DNA fingerprint- ing, PCR-based microsatellite genotyping, and auto- mated DNA sequencing) and the advent of noninva- sive DNA sampling, coupled with the new ease and speed with which molecular analyses can be per- formed, have made it possible for primatologists to investigate some of these issues in greater detail. In fact, the pace with which molecular techniques are being applied as a complement to field observational studies of behavior has quickened substantially in the last 10 years, and researchers now routinely incorporate a molecular component into field re- search programs. The purpose of this paper is to broadly review the application of molecular techniques, particularly DOI 10.1002/ajpa.10382 Published online in Wiley InterScience (www.interscience.wiley. com). YEARBOOK OF PHYSICAL ANTHROPOLOGY 46:62���99 (2003) �� 2003 WILEY-LISS, INC.

polymerase chain reaction (PCR)-based microsatel- lite genotyping and mitchondrial DNA sequencing, to examining and understanding the social behavior and social systems of primates. Four major topics will be covered in this review. The first is a theoret- ical discussion that considers how primate popula- tion genetic structure (i.e., the patterning of genetic variation within and between social groups at the local and regional scale and within and between various classes of individuals within social groups) is influenced by individual-level behaviors and by aspects of primate social structure such as dispersal patterns, dominance hierarchies, mating patterns, and group formation processes. Understanding the links between behavior and social structure on the one hand, and population genetic structure on the other, is fundamental to making inferences about primate social organization and behavior from em- pirical patterns of genetic variation seen in wild primate populations. Secondly, this paper discusses some of the molecular markers and analytical tech- niques that have been used to examine primate ge- netic structure and the influence of individual be- havior on that structure. Third, I present an overview of studies that have used these markers and methods to investigate specific aspects of pri- mate social organization in wild and select captive populations. Finally, some avenues for future work are discussed, including the emerging discipline of nonhuman behavioral genetics, which promises to become a major area of research and may prove fundamental for understanding the evolutionary significance of the rich behavioral variation we see characterizing nonhuman primates. An online Appendix (http://www.nyu.edu/projects/difiore/ yearbook2003/appendix.html) accompanies this article and provides a comprehensive list of mic- rosatellite markers that have been used in pri- mate studies and the taxa in which they have been used. LINKS BETWEEN BEHAVIOR, SOCIAL STRUCTURE, AND POPULATION GENETIC STRUCTURE Geneticists have long recognized that the genetic variation present within natural biological popula- tions can be partitioned hierarchically into compo- nents that reflect underlying population structure (Wright, 1951, 1965 Crow and Kimura, 1970 Nei, 1973). The basic model of population genetic struc- ture by Wright (1943, 1951, 1965) envisions three hierarchical levels of organization: a large total pop- ulation (T), which can be divided into a set of dis- crete subpopulations (S), each of which contains a number of individuals (I). In this model, mating takes place within subpopulations, and subpopula- tions are connected with one another by some degree of gene flow. Practically speaking, additional hierar- chical levels of organization are also possible. T might encompass the set of social groups found in a particular geographic area, with each S represent- ing one of those constituent social groups alterna- tively, T might be taken to constitute the entire set of individuals belonging to a particular species, with several hierarchical levels of population organiza- tion (e.g., social groups, local populations, or re- gional populations) between the individual and spe- cies levels. In either case, classical population genetics theory describes the genetic consequences of population subdivision and of nonrandom mating within various subpopulations by using Wright���s F- statistics, which summarize how the total genetic variation present in a large population is partitioned among different hierarchical levels. Briefly, for the simplest case with three levels of organization, Wright���s FIS summarizes the effects of nonrandom mating within subpopulations on average individual heterozygosity. FST characterizes the reduction in individual heterozygosity expected within subpopu- lations relative to a total population as a result of genetic drift, effectively measuring the extent of population subdivision and the counteracting evolu- tionary processes of drift on the one hand and gene flow on the other. Finally, FIT summarizes the ex- tent to which average individual heterozygosity de- viates from Hardy-Weinberg expectations due to both nonrandom mating within subpopulations and population subdivision (Hartl and Clark, 1997). Wright���s F-statistics and other similar indices that describe the partitioning of genetic variance at dif- ferent hierarchical levels can be estimated for nat- ural populations using a variety of molecular marker data (Nei, 1973 Weir and Cockerham, 1984 Slatkin, 1985). Although this classical model for describing the partitioning of population genetic variation does not explicitly link genetic structure to elements of social organization or individual-level behavior (Sugg et al., 1996), it does make a number of implicit connec- tions that interest behavioral ecologists. For exam- ple, positive FIS values reflect inbreeding within subpopulations, as mating among close kin results in an increase in homozygosity relative to what would be expected if mating within the subpopula- tion were random. Negative FIS values, on the other hand, suggest that behavioral mechanisms for avoiding inbreeding may be at play. Additionally, because FST values reflect the relative importance of gene flow and genetic drift in homogenizing vs. di- versifying allele frequencies among subpopulations, they can be used to indirectly infer the minimum number of individuals dispersing between subpopu- lations in each generation (Wright, 1943 Takahata and Nei, 1984 Slatkin, 1995 Cockerham and Weir, 1993). To summarize, the classical model views ge- netic differentiation among subpopulations as depend- ing primarily on population-level rates of inbreeding within and migration between subpopulations and on demographic factors, such as effective subpopulation size, that influence the rate of subpopulation diversi- fication through drift. But although rates of sub- population divergence are ultimately dependent on PRIMATE MOLECULAR ECOLOGY 63

individual-level behavioral processes (such as mat- ing patterns within social groups or individual deci- sions over dispersal), those processes are incorpo- rated into the classical model only indirectly, through a population-level lens that ignores individ- ual decisions and variation in behavior between in- dividuals. In the last 25 years, prompted in part by observa- tional studies of wild animals, behavioral ecologists and population geneticists have begun to consider more explicitly how individual-level behaviors and other features of animal social systems (e.g., sex- biased dispersal patterns, dominance hierarchies, strong reproductive skew, and processes of new group formation) influence population genetic struc- ture, both theoretically and empirically (Chepko- Sade and Halpin, 1987 Melnick, 1987 Chesser, 1991a,b Sugg et al., 1996 Storz, 1999 Ross, 2001). Next, I review some of the features of social struc- ture and individual behavior that have particularly important influence on the population genetic struc- ture of natural populations of primates and other social mammals. Understanding how social struc- ture and individual behavior influence the partition- ing of genetic variation in natural populations is critical for designing effective conservation pro- grams to manage and conserve that variation. Ad- ditionally, such knowledge is essential for behav- ioral ecologists to make accurate inferences about the social systems and behavior of difficult-to-ob- serve taxa based on population genetic surveys. Dispersal patterns and genetic structure Where classical population genetics theory treats gene flow as a deterministic process with no account- ing for social structure or variation in behavior among individuals, a behavioral ecological perspec- tive on gene flow highlights several features of dis- persal that are likely to influence population genetic structure. First of all, in most species of vertebrates, one sex typically disperses while the other remains philopatric (Greenwood, 1980 Waser and Jones, 1986 Johnson and Gaines, 1990), a pattern that can have marked implications for the structuring of ge- netic variation within and between populations. Specifically, when dispersal is sex-biased, contrast- ing patterns of genetic structure are expected for the nuclear genome (which is inherited though both the maternal and paternal lines) vs. the genome that is passed strictly through the philopatric sex (the mi- tochondrial genome for females, the Y chromosome for males). Avise (1995, 2000) neatly summarized some of the expected patterns. For example, species characterized by high levels of female philopatry are expected to show strong evidence of population ge- netic substructuring to their mitochondrial genes. In contrast, little to no genetic structure is expected for autosomal markers or Y-linked genes in these spe- cies, since males are effectively distributing these as they move out of their natal social groups and begin breeding. Cercopithecine primates typify this pattern of fe- male philopatry and near-universal male dispersal (Melnick and Pearl, 1987 Pusey and Packer, 1987). For these species, mitochondrial genes are not shuf- fled among social groups within a local or regional population nearly to the extent seen for nuclear genes, which should theoretically result in contrast- ing patterns of nuclear vs. mitochondrial genetic structure (Melnick and Hoelzer, 1992, 1996). More- over, restricted mitochondrial gene flow, combined with the stochastic processes of mutation and ge- netic drift (by which populations come to diverge from one another genetically) and lineage sorting (the process by which maternally inherited mito- chondrial lines are lost from a population due to the fact that some females, by chance, leave no female descendants), should result in relatively low diver- sity in mitochondrial DNA among females within social groups and within local populations but much greater interpopulational differences, even in the absence of major geographic barriers to gene flow (Melnick and Hoelzer, 1996 Wallman et al., 1996). Thus, most cercopithecine primates should be char- acterized by very low levels of mitochondrial diver- sity within groups. In contrast, for species characterized by female dispersal (whether or not males are philopatric), there is no expectation of low mitochondrial DNA diversity within social groups or of greater geo- graphic substructuring to mitochondrial vs. nuclear diversity. Instead, all else being equal, comparable levels of substructuring are expected for both mito- chondrial and autosomal genes, since female-medi- ated gene flow effectively homogenizes both of these genomes across the landscape. Whether a marked population genetic structure is seen in Y-linked genes depends on the extent of male philopatry and on whether females disperse before breeding or carry with them offspring fertilized by males from their natal groups (Avise, 2000). Thus, for primates in which female dispersal and male philopatry are the norm���spider monkeys (Ateles), muriquis (Brachyteles), some red colobus (Procolobus badius), hamadryas baboons (Papio hamadryas hamadryas), and chimpanzees and bonobos (Pan)���we would ex- pect to see comparable evidence of structure in the mitochondrial or autosomal genomes, a greater de- gree of population genetic structure in Y-linked genes, and high mitochondrial DNA diversity within social groups. Finally, for taxa in which both sexes disperse to an appreciable degree, such as in many pair-living primates and highly folivorous taxa such as howler monkeys (Alouatta), gorillas (Gorilla), and many colobines, comparable levels of population structure are expected in all of these genomes (Avise, 2000). Sex-biased dispersal patterns also have obvious theoretical implications for patterns of within-group relatedness. In the extreme case, members of one sex are predominantly recruited as new breeding individuals in their natal populations. As a result, 64 A. DI FIORE

the average genetic relatedness among adults of the nondispersing sex is predicted to be greater than among those of the dispersing sex. Thus, for most cercopithecine primates, we would expect mean pairwise relatedness among females within a social group or among females within a local population to be greater than among males, since almost all males transfer into groups, while female breeders are re- cruited from within their natal groups. In contrast, we would expect males to show greater average lev- els of relatedness with one another in species show- ing male philopatry and a marked female bias in dispersal (e.g., chimpanzees, spider monkeys). Finally, individual-level decisions over dispersal can also influence patterns of population genetic structure, especially since dispersal between pri- mate social groups within a local population is not a random process, as classical population structure models assume. In some cases, individuals from the same natal social group may transfer together (rhe- sus macaques: Drickamer and Vessey, 1973 Meikle and Vessey, 1981 Japanese macaques: Sugiyama, 1976 baboons: Cheney and Seyfarth, 1977), or join social groups to which other members of the dispers- er���s previous group have already migrated (Cheney and Seyfarth, 1983). For example, Cheney and Sey- farth (1983) reported that 14 of 16 social group transfers by natal or not yet fully grown male vervet monkeys (Chlorocebus aethiops) were to groups con- taining former members of the disperser���s previous group. A biased transfer process results in a nonran- dom redistribution of genetic variation among social groups at the population level if transferring ani- mals then breed successfully in their new groups, the result should be greater differentiation among social groups than predicted by classical population genetic theory (Melnick, 1987). Mating behavior, reproductive skew, and genetic structure Classical population genetic models assume ran- dom mating within subpopulations, but this as- sumption is clearly violated in many natural popu- lations. In many primates and other social mammals, mating behavior is strongly skewed within social groups, sometimes to the point where only a single member of one or both sexes is seen to mate. In general, skew in reproductive behavior is greater among males and appears to correlate well with male dominance rank (Cowlishaw and Dunbar, 1991), which reflects a male���s ability to consistently win in agonistic encounters with other males. Field observations of this relationship between male rank and male mating success led Altmann (1962) to sug- gest the priority-of-access model of male dominance, which predicts that the top-ranking male in a social group will monopolize both mating and paternity by guarding females at those times during their estrus cycles when conception is most likely. When multi- ple females are in estrus, the model predicts that paternity should be shared among males in order of dominance rank. The priority-of-access model thus predicts a skew within-group paternity towards dominant males, the degree of which is determined by the average number of females simultaneously in estrus, which in turn depends on the number of cycling females and the length of the estrus cycle (Dunbar, 1988). The existence of such a relationship between male rank and paternity success has implications for ge- netic structure within and between social groups (Melnick, 1987 Pope, 1990). If a single male or a small set of males is responsible for most of the paternity within a social group over some period of time, then members of cohorts born during those males��� tenure are predicted to be more closely re- lated to one another (at least to the level of paternal half siblings, assuming complete monopolization of reproduction by a single male) than they would be to members of the larger social group or to individuals from cohorts sired under a different male���s tenure. Depending on male tenure length, this process could dramatically reduce effective social group size and thus increase the likely rate of genetic differentia- tion between social groups in a local population. For animals that live in extended family groups, including those such as some callitrichine primates that practice cooperative breeding (Goldizen, 1987 Sussman and Garber, 1987 Tardif et al., 1993), reproductive skew theory suggests a more nuanced relationship between individual-level mating behav- ior and the structuring of genetic variation (Keller and Reeve, 1994 Emlen, 1995, 1997 Clutton-Brock, 1998). Among cooperative breeders, reproduction within each sex is often strongly biased toward a single, dominant individual who actively suppresses the reproduction of subordinate, same-sex competi- tors (French, 1997). These subordinate animals nonetheless contribute to the reproductive success of dominants either directly (thorough helping behav- ior such as infant carrying or provisioning) or indi- rectly (through group size effects in reducing the likelihood of predation or enhancing the group���s ability to compete with other groups) (Koenig, 1995 Tardif, 1997). In some cooperatively breeding cal- litrichines, subordinate individuals sometimes do reproduce (Digby and Ferrari, 1994 Goldizen et al., 1996). Models of optimal reproductive skew predict that the degree to which a dominant animal con- cedes reproduction or mating opportunities to sub- ordinates should be inversely related to the degree of relatedness between those individuals, i.e., dom- inants are predicted to concede more reproduction to individuals who are less closely related to them (Keller and Reeve, 1994 Emlen, 1995, 1997 Clut- ton-Brock, 1998). This somewhat counterintuitive proposition is based on the fact that more closely related subordinate animals gain greater inclusive fitness benefits from their effect on a dominant���s reproduction than do less closely related animals and thus require less of a ���staying incentive��� in PRIMATE MOLECULAR ECOLOGY 65

terms of personal reproduction to keep them from dispersing At the population level, the rate and extent of genetic differentiation between social groups are ex- pected to be proportional to the degree of reproduc- tive skew seen in the population, which in turn depends on population density. For callitrichines liv- ing at high population density, where territories and available breeding positions are limited, offspring are likely to delay dispersal and become reproduc- tively suppressed adult ���helpers��� within their natal groups, which will lead to extensive genetic differ- entiation among groups. In contrast, genetic differ- entiation between groups should be lower at low population density, as maturing offpring are more likely to be able to disperse successfully and begin breeding themselves. Group formation processes and genetic structure Classical models of population genetic structure do not typically consider how the subpopulational composition of a larger population may change over time. However, studies of wild primate populations reveal that new social group formation is not a rare occurrence, and the method of new group formation has direct consequences for population genetic structure (Duggleby, 1977 Cheverud et al., 1978 Melnick and Kidd, 1983 Melnick, 1987). Among a number of species of cercopithecine primates, new groups typically form from the fissioning of existing groups along matrilineal lines (free-ranging rhesus macaques, Macaca mulatta: Chepko-Sade and Sade, 1979 wild rhesus macaques: Southwick et al., 1965 Japanese macaques, Macaca fuscata: Furuya, 1968, 1969 toque macaques, Macaca sinica: Dittus, 1988 baboons, Papio hamadryas: Nash, 1976). Theoreti- cally, the result of matrilineal fission is that each daughter group will be characterized by a higher average level of within-group relatedness than the parent group at the same time, the average degree of genetic differentiation among groups in the pop- ulation should also increase (Melnick and Kidd, 1983 Wade and McCauley, 1988 Whitlock and Mc- Cauley, 1990). The extent to which this expectation is met in wild cercopithecine populations, however, depends on the number and size of different matri- lines within a fissioning group and on degree of genetic differentiation between matrilines (Melnick and Kidd, 1983 Melnick, 1987). Additionally, be- cause cercopithecine groups are characterized by low mitochondrial DNA diversity and because new females do not immigrate into these groups, geo- graphical population expansion through group fis- sioning and colonization of new areas can lead to large areas being characterized by very similar mi- tochondrial haplotypes (Melnick and Hoelzer, 1996). This pattern of new group formation, combined with the stochastic process of lineage sorting at play in groups from across a species��� range, can also result in a clear geographic population structure in mito- chondrial DNA in the absence of any kind of physical barrier to dispersal or of similar structuring to nu- clear genetic variation (Hoelzer et al., 1994 Melnick and Hoelzer, 1996). In other species of primates, new group formation proceeds not through a process of fissioning from existed social groups but rather from the union of dispersing individuals of various source groups. For example, dispersing male and female Venezuelan red howler monkeys (Alouatta seniculus) join to- gether in coalitions against existing groups to estab- lish new home ranges and to begin breeding (Pope, 2000). Under this model of new group formation, within-group relatedness is expected to be low ini- tially. The rate and extent to which within-group relatedness increases in these groups over time and to which new groups become further differentiated genetically from existing groups depend on dispersal patterns and mating behavior, as discussed above. Importance of exploring these links There are two major reasons why primate behav- ioral ecologists should be concerned with under- standing the links between individual-level behav- ior and social structure on the one hand and population genetic structure on the other. First, knowledge of population genetic structure in differ- ent taxa is crucial for evaluating models of the evo- lution of social behavior and, indeed, of sociality. Many models of primate social evolution take as a fundamental assumption the importance of kin se- lection, i.e., the idea that behaviors and patterns of social affiliation can be selected for because of their effects not just on an individual���s direct fitness but on the survival and reproduction of relatives as well. As an example, Wrangham���s (1980) model for the evolution of ���female-bonded��� social groups in pri- mates suggests that females should refrain from dispersing from their natal ranges and should form groups preferentially with kin when larger groups of females can more effectively defend access to neces- sary resources than smaller groups or individuals. Moreover, many affiliative social behaviors are pre- dicted to be manifest more often among relatives than among nonrelatives because of kin selection (Gouzoules and Gouzoules, 1987 Silk, 2002). For example, grooming behavior in primates is hypoth- esized to be more commonly directed toward kin, and individuals are expected to form coalitions more often with relatives. However, these are, in effect, predictions that in many cases have yet to be tested in natural populations using genetic data. Additionally, understanding the links between be- havior, social structure, and population genetic structure is crucial for using molecular data to infer something about individual-level behaviors and the features of primate social organization that may have given rise to them. For many primate species, it is difficult to conduct the long-term investigations of mating behavior or dispersal patterns that are of such great interest to behavioral ecologists, but mo- 66 A. DI FIORE