The Town had originally been named Douglass, Texas by officials of the Texas and Pacific Railroad in the 1870's. The T&P was crossing Red River County and there were so few people in the area that nobody objected to the decision. A newspaperman from Clarksville visited the site in 1881 and reported that residents were living in a tent village. Things were about to change. Isaac Bradford opened a store there the following year and in the custom of the day, the store housed the post office. With postal service established, Douglass, Texas became Isaca, Texas. The population, however, was less than 50 residents.

The name only lasted until 1902 when it was changed to honor Ed Avery, the T&P station Agent. Avery had a population of 176 by 1900 and cotton was the town's lifeblood. Gins opened and by 1914, Avery had two banks, a weekly paper and a population of 500. At its peak ( in the late 1920's) Avery had nearly 800 citizens which declined to a mere 300 during the onset of the Great Depression. In 1940 it had a population of just under 500, and it has remained between 430 and 500 ever since.