Neglect can be as impactful (if not more so) on the brain and body of the developing child as physical abuse. The Center on the Developing Child at Harvard University has published a brief on this issue as well as a video illustrating that impact.
Research shows that caregiver absence and/or unresponsiveness during a child's early years can lead to cognitive delays, disruption of the body's stress response systems, and impairment in executive functioning. Such deprivation in the earliest most formative stages of the child's development can also cause impairment in health, behavior, and learning.
Attachment to one's primary caregiver during the first months is a critical foundation on which a human being's brain architecture is formed and creates the ability to form healthy relationships throughout life. This attachment includes a sensitively attuned responsiveness by the caregiver. The Harvard team's brief creates a chart describing differing levels of neglect and their effects, distinguishing between chronic under-stimulation and severe neglect.
I would suggest many people in the general population are unaware of the degree to which neglect can impact human behavior, cognition, and even physical health.