The first Irvington Public Library was established in the mid-19th century by an interested group of Irvington residents. It was located in "the little red schoolhouse." The schoolhouse library evolved into the short-lived Irvington Free Library, which later became the Atheneum. In 1902, the library moved into rooms in the Village Hall which was built on the site of the original Antheneum. The library furnishings were donated by philanthropist Helen Gould of Lyndhurst, the daughter of railroad financier Jay Gould. Frederick Guiteau provided funds for the initial collection of books.

The Irvington Public Library was incorporated by the Board of Regents of the State University of New York on May 3, 1902. In that year the library served a population of about 2,000. Over the years the population grew and the library added a storeroom here, a closet there, part of the lobby and part of the back stairway. Though squeezed and cramped, it retained its original Tiffany glow and warmth. In the late 1990s, the Library Board concluded that a new library was essential for more space, better access for those with disabilities and to support new technologies.