The Lyceum inevitably led to the formation of a library. In 1850, a local businessman and philanthropist, Samuel E. Sawyer, offered the Lyceum $100 if additional funds could be raised to develop a library collection. With additional support from Mr. Sawyer and funds donated by the public, a library collection of 1,400 volumes was established by 1854. When all but 300 of its 3,000 volumes were lost in a major downtown fire in 1864, Mr. Sawyer stepped in and added $500 to the insurance settlement to rebuild the collection. Again, in 1871 he made another gift of $10,000.

Membership fees were suspended and the library was officially named The Sawyer Free Library. The library did not yet have a permanent home. Several different locations and another major fire followed in the course of the next decade. In 1764 Thomas Saunders, a merchant and state representative built a sturdy house on the corner of Dale Avenue and Middle Street. Subsequently, the house passed through several owners and further architectural enhancements. In 1884 Mr. Sawyer purchased this prominent residence and donated it to the library corporation. At the dedication ceremony on July 1, 1884. Mr. Sawyer explained the reasons for his generosity: “It has always been a prominent motive or object of my life to do something to promote the best interests of the young, for in them lie the germ, the roots and fibres (sic) of civilization. Books are the food of the mind; from the earliest years of childhood books are sought to feed the intellect, and so from school to college; later on they are a course of recreation to the idler, the tools of the student, the scholar and the man of letters.”