The library has had many homes since first opening in 1891. After years in the front rooms of many homes, the library moved to the old church, which then burned to the ground in 1953. The town herb garden now occupies that spot. A new building, known as the Albert J. Sargent memorial Library located at 575 Middle Road, was opened in February 1966 and functioned as the town library until early in 2005. Over the years, this building underwent some changes including the creation of the Ella Whitcomb Room in 1977, which housed the children's collection, and the removal of the cupola for safety reasons.

In May 2000, the town approved a project to build a new library. Through the generosity of the residents of Boxborough, and the MLBC grant, plus the hard work of many volunteers, Town Officials, and employees, the new Sargent Memorial Library, in its current location at 427 Massachusetts Avenue, opened its doors to the public in March 2005.