The men were surveyed about their physical activity and number of hours a week they watch television in addition to a host of other health factors that may affect sperm quality, including diet, smoking and stress levels.
Results showed that men who watched more than 20 hours of television weekly had a 44 percent lower sperm count than men who watched almost no television.
However, men who exercised for 15 hours or more a week at a moderate to vigorous rate had a 73 percent higher sperm count than those who exercised less than five hours a week.
"The majority of previous studies on physical activity and semen quality had focused on professional marathon runners and cyclists who reach physical activity levels that most people in the world cannot match," said Jorge Chavarro, senior author of the study and an assistant professor of nutrition and epidemiology at the school of public health.
It is unclear exactly why the difference in sperm count happens but scientists suspect that sedentary lifestyles may warm the scrotum and affect semen concentrations. Physical inactivity has also been linked to increased levels of oxidative stress, in which rogue oxygen compounds degrade cells.
Obesity and high-fat diets have been linked to lower sperm counts in other studies.
Researchers are looking into fertilizers, plastics and other environmental factors.