As the popularity of antidepressant medication has burgeoned over the past few decades, serotonin has become one of the more publicly recognized neurotransmitters. Along with that popularity has come a trend of attributing a wide variety of behaviors (especially depression) to “serotonin imbalances”. While this is a gross simplification in most cases, it does seem to be clear that there is a correlation between serotonin transmission and behavior.
A group of researchers at Case Western Reserve University has recently shown that the disruption of serotonergic function in mice is powerful enough to inhibit one of their strongest instincts: caring for their young. They used female mice with a mutation that causes a reduction in the expression of serotonergic genes and in the synthesis of the neurotransmitter, and monitored the survival of their young after they gave birth.
99% of the pups of the wild-type (normal) mice lived past the nurturing period of youth, but none of the pups of the serotonin-inhibited mothers survived. In fact, most of them were dead after 3-4 days. When the researchers took pups born to the serotonin-deficient mothers and gave them to the wild-type mothers to raise, the pups survived. The serotonin-deficient mothers failed to nurse their pups, didn't build nests for them, and didn't organize them near her in a huddle (which is necessary for their warmth and survival).
The serotonin-inhibited mothers did not seem to exhibit deficiencies in any other behavioral assay, such as maze-running or olfaction. They were not deemed to be overly anxious as measured by locomotor tasks, but instead of mothering they often simply paced the cage and engaged in repetitive digging. The authors suggest that anxiety behaviors may have been more prominent if not for the relaxing effect lactating has on rodents.
How applicable these findings are to humans is, of course, completely unclear. Postpartum depression is often treated with selective serotonin reuptake inhibitors, but even if untreated doesn’t generally lead to abandonment of one’s children. Regardless, finding a neurochemical substrate for an instinct like caring for one's young is notable, as it is a behavior essential to what is widely considered the goal of existence: high reproductive fitness.
Lerch-Haner, J.K., Frierson, D., Crawford, L.K., Beck, S.G., Deneris, E.S. (2008). Serotonergic transcriptional programming determines maternal behavior and offspring survival. Nature Neuroscience, 11 (9), 1001-1003. DOI:10.1038/nn.2176