The Brown-Pusey House built in 1825 by John Y. Hill was home for he and his wife, Rebecca. This warm and stately old home was for many years the Hill House, a boarding house operated by "Aunt Beck" Hill. Among the guests at the Hill House were General George Armstrong Custer and his wife Elizabeth. General Custer's assignment in Elizabethtown was to combat the influence of the Ku Klux Klan and the illegal distilleries.

Other notable visitors to the Hill House were the Swedish Nightingale, Jenny Lind, who sang on the front steps, and General John Hunt Morgan of the Confederate Army who dined at the Hill House on several occasions. In 1923, the Brown-Pusey House was restored and given to the community by Dr. William Allen Pusey and Dr. Alfred Brown Pusey (great-nephews of "Aunt Beck") as well as Mrs. Sallie Cunningham Pusey and the related Hastings family.

The Brown-Pusey House serves the community in many ways, as envisioned by its grantors. This Georgian mansion houses the Pusey Room Museum, a genealogical library, and meeting rooms for private and public functions. There is also a beautifully maintained garden for the public to enjoy.