Abstract Background Extracurricular sport participation and exercise (ESPE) refers to regular exercise/sport participation in addition to the physical education in school among a school-aged population. Rather than general physical activity, ESPE is typically deliberately initiated and presents an efficient target for interventions. However, compared to physical activity, relatively few studies have investigated sex differences in the development of and factors associated with ESPE using a person-centered approach. This study aimed to examine the latent trajectories of ESPE from childhood to emerging adulthood across sexes, and to identify the associated sex-specific individual (i.e., body mass index, body dissatisfaction, stress, and screen behavior) and parental (i.e., parental exercise and parental screen behavior) factors. Methods This study used data from part of the Child and Adolescent Behavior in Long-term Evolution (CABLE) project, which comprised 2072 fourth graders (aged 9 years) in Northern Taiwan followed annually from 2001 to 2013 (13 waves). Repeated-measures latent class analysis was used to identify the trajectories of ESPE for males and females, respectively. Multinomial logistic regression was further used to identify sex-specific factors related to ESPE. Results Four trajectories of ESPE were identified for males and females. For males, these trajectories were Rarely-to-Never (20%), Often-to-Rarely (32%), Always-to-Never (21%), and Always (27%). For females, these trajectories were Rarely-to-Never (34%), Rarely (23%), Always-to-Rarely (33%), and Always (10%). We observed that the developmental patterns of ESPE varied by sex such that there was an earlier decline in the trajectories of ESPE in females than in males and that, compared with males, fewer females maintained exercise habits in young adulthood. Furthermore, we found several sex-specific factors related to ESPE, namely, stress, BMI, and parental exercise. Body dissatisfaction and individual screen behavior were associated with trajectories of ESPE for both sexes. Conclusions We found distinct trajectories of ESPE from childhood to emerging adulthood for both sexes. The trajectories of ESPE for males and females, however, differ in terms of patterns and associated factors. Our findings suggest that efforts to increase ESPE should be initiated early, and may be made more effective by considering sex differences.