The southwest corner of the Town of Vernon was part of the community generally known as Caldwell's Prairie, which also encompassed areas in the vicinity of the Village of Mukwonago. In 1835 Joseph and Tyler Caldwell and their families traveled from Hubbard Town, Vermont and settled about three miles from Kenosha, then known as Pike River. Abraham Resseguie, William Cheney, and Clavin Gault moved to Caldwell's Prairie. At the time the Resseguie family moved to Caldwell's Prairie about 100 Pottawatomie Indians lived in Mukwonago. Their pow-wows could be heard at night. In 1836, members of the Vernon section of Caldwell's Prairie arrived: Henry Camp in Section 30, Ira Blood in Section 30/31, Wilder Champin in Section 20, and Whitney Hudson in Section 31 staked their claims in 1837. Other Caldwell Prairie neighbors shown in 1859 mapping included the Porter and Craig families. Their descendents continue to live in the Town.

Grammatic evidence remains of the Town’s first inhabitants. Twenty-nine historic mounds said to be burial grounds of Native Americans known as the Mound Builders, are located on the Born farm in Section 21 (previously owned by the Dewey's after 1920 and the Hollisters before 1920). Most of the mounds are enclosed in a park-like area enclosed by giant oaks. In 1956, Mrs. Sylvia Dewey, who's family once owned the mound area, said that their family often found arrowhead hatchets and other items left by the first settlers in the Town of Vernon.