The first library in New Milford, known as the Union Library, was established in 1796 and housed in "public spirited citizens' " homes. Open only 5 days a year, the library catalog listed 350 volumes. In March of 1886, the New Milford Library Association was established, and the collection was located in Town Hall. The General Assembly of Connecticut presented and adopted a special bill "authorizing the Town of New Milford to erect a library building and memorial hall " in 1893. The hall being dedicated to the veterans of the Civil War.

Mr. Egbert Marsh offered a lot on Main Street on which a building was erected in 1897. The reading Room opened on January 19, 1898, and the circulating library began operation on March 21, 1898. An addition was built in 1977 which incorporated the Goodman House along with the new modern section to create the library building as we know it today. Since the 1977 addition, various offices and rooms have been relocated within the present structure to make better use of the existing space for the ever-increasing number of patrons.

In 1998 the Library celebrated its 100th birthday. The year was filled with exciting activities and special programs. The final event for the year included a time capsule "trunk" filled with library memorabilia.

The meeting room upstairs in the original library building was officially named "Memorial Hall". Civil War photographs and memorabilit were used to furnish the hall and a formal dedication ceremony was held.