Augustus W. Sullivan established a trading post named Port Sullivan in 1835 about one-mile northeast of Farm Road 485 on a low bank overlooking the Brazos River in Eastern Milam County. In 1936 the Texas Centennial Commission had a historical marker put up at the location that was misnamed as Fort Sullivan. The first steamboat that traveled up the Brazos to this location was the Mustang in 1843; several other steamboats traveled along this same route in 1850. In May 1851, Reuben Anderson convinced Sullivan to subdivide his land into town lots; the first lots were sold to Andersons sons helping to establish this new town. By 1852, Port Sullivan had its own post office, several warehouses, a sawmill and four stores with a population of about two hundred. The town also had a lawyer and several doctors. Steamboats in Galveston shipped cotton, pecans and various other items upriver to Port Sullivan. The town continued to grow to a population of almost seven hundred by 1860, and the Port Sullivan College for men and women was established. Because the town was set up on a bluff above the river, it was flood free.
The Lone Star flag was raised to support succession on November 24, 1860. On February 23 1861, Port Sullivan voted for secession by a vote of 227 to 24. Because of a dispute between Reuben A. Smith and Emil Adams, the votes from Milam County were not counted possibly affecting the outcome to the election.
Steamboat transportation started to decline during the 1860s and 1870s as railway construction entered the area. In 1869, the Houston and Texas Central railway reached Hearne and moved toward Calvert a new town which soon became larger than Port Sullivan in business and population. Many of the residents of Port Sullivan moved to other towns. The Port Sullivan collage closed by the end of the 1870s with a remaining population of less than 125 by 1880 and the town of Port Sullivan died away by 1890.
The only thing remaining at the old town site is the Port Sullivan cemetery west of the Brazos River on an old gravel road to the north of Farm Road 485.