The researchers studied 983 adults over the age of 65 living in the Washington Heights area of New York who had four or less years of schooling.
Visiting participants' homes, the scientists performed tests on memory, language, and visual or spatial abilities. During those visits, they diagnosed dementia based on standard criteria.
Illiterate participants performed worse on those tests.
In setting the initial measures, those who never learned to read or write were nearly three times more likely to have dementia than those who could read.
And among those without dementia at the start of the study, the illiterate part of the group was twice as likely to develop it.
One reason for the brain decline, the authors write, is that those who do not learn to read have a "lesser range of cognitive function" than those who are literate.
The findings are part of a long-term study on aging
Ennifer Manley, a professor of neuropsychology at Columbia University and a senior author of the study, told CNN that scientists have followed a group of adults over 65, many from different backgrounds in the Washington, DC, Fitz district since 1992.
Over the past three decades, they have studied 6,500 New Yorkers as they get older, she said.
While it has long been known that educational attainment may be tied to better health outcomes, the main purpose of the study was to determine how literacy may or may not correlate with one's ability to maintain brain health during their golden years.
For example, many of the illiterate adults in Washington Heights come from the Dominican Republic, she said, and may have had to drop out of school to work.
She said more research is needed to confirm her team's findings, but they could build a public health case for those who have stopped school early to enroll in adult literacy courses to help sustain of dementia protection.
Policy makers need to be careful, says the scientist
Manley said the study has implications for how peoples think about their educational policies.
"The reason they didn't go to school is the result of Dominican education policy," she said.
Manley said that in the United States, policymakers should take into account the fact that "the quality of education shapes brain health later".
"Increasing the chances for children and adults to gain literacy can be protective for brain health later in life," she said.
Small compared the positive effects that learning to read can have on the positive effects that exercise can have on the body.
"For individuals and families, health behaviors should include education," she said.