When the five Trustees of the Rackham estate were considering the projects they should support in carrying out the provisions and intent of Mr. Rackham's will, the establishment of a community center in a small community seemed to them appropriate and desirable, and as four of them knew the Fenton Community well, its interests and needs, Fenton was chosen for undertaking.

Of the five Trustees, the three members of the Horton family, Mrs. Rakham, Mrs. Bussey, and Mr. Horton, had been born in Fenton; and another, Mr. Rolland, had grown up there. Mr. Rackham had known and loved the town. In the account of their various grants and endowments, the Horace H. Rackham and Mary A. Rackham Fund, published by the Trustees in 1940, Fenton was described as "a decorous, tree-shaped, typically American village in the heart of a agricultural district...without a central gathering place" and with "no community facilities for the promotion of leadership, educational advancement, social enjoyment, or civic improvement." The population numbered around four thousand.

In 1919, in the center of the town, along the Shiawasee River, the Horton's had established and presented to the village of Fenton the Dexter Horton Park, on what had been part of their father's land, and they proposed that the community center be located on adjoining land. Accordingly, on January 20, 1937, the Trustees created a $200,000 trust fund to be used by Fenton to buy and clear the land and build a community house, and any remaining balance to supply income for upkeep. Since clearing the land would involve tearing down the old firemen's hall that contained the cities offices, a new building for the purpose had to be provided, and the Trustees made an additional appropriation of $5,000 toward its construction. The highly prized town clock, nearly seventy-five years old, was to be transferred from the old building to the new City and Fire Hall.

Plans for the Community Center building proceeded. It was designed by Eliel Saarinen, who had designed all of the buildings of the Cranbrook Foundation at Bloomfield Hills, and his son Eero. It was built by the W. E. Wood Company and was landscaped according to the plans of the landscape artist, Mrs. F. W. Whittlesey, of Phoenix, Arizona, daughter of Mr. Horton. The Trustees formally presented the building to the Village of Fenton on the afternoon of October 3, 1938, with president Ruthven addressing the audience of five hundred in the auditorium-ballroom on the second floor.