In boys, hormones made by the testicles keep breast tissue from growing too much. Men's breast tissue has ducts and very few, if any, lobes.