Most Americans, no doubt, imagine the prewar South as a region so thickly dotted with immense plantations on which most of the black and white populations worked and lived. But, on the contrary, while slaves made up 40% of the total population of the South, only 25 percent of free families, most of them white, owned any slaves at all, and fully one-half of this minority (12.5%) held fewer than five slaves. Only an owner of twenty or more slaves, and of substantial land, could qualify as a planter, and fewer than 10 percent of slave-holding families qualified. The plantation elite of the antebellum South made up less than 3 percent of the free population in the region and less than 2 percent of the total free and slave populations combined.
Bitter Fruits of Bondage: The demise of slavery and the collapse of the Confederacy, 1861-1865. Armisted L. Robinson. Univ of Virgina Press, 2005.