born:   3 April 1803 in Spartanburg County, South Carolina
died:  12 April 1873 in Clay County, Alabama
1830 Henry County, Georgia Census: Stephen Smith appears with 1 male under 5, 1 male 20-30, 1 female 5-10, and 1 female 15-20. Nearby is a Lewellin Morgan who may be related to Griffin Morgan who married Stephen's sister Letty.  
1840 Talladega County, Alabama Census: Stephen Smith appears with wife, four children and one slave. 
1850 Talladega County, Alabama Census: Stephen Smith 47 (farmer, b SC), Nancy 38 (b SC), Miles L. 21, Elizabeth 18, Martha A. 15, James H. 12, Hiram J. 10 (all b GA), Mary J. 8, Lovey L. 5, William 2 (all b AL), Mary E. 29 (b SC), Martin M. 3/12  (b AL)
1860 Talladega County, Alabama Census (Court Hill): Stephen Smith 57 (farmer, b SC, 600 RE, 600 PE), Nancy 48, James 23, Mary 24 (error for Martha), Hiram 20, Mary 18, Lovy 15, William 14, Marcus 11, John 5, Wiley 3.  Nearby is his son Miles L. Smith 32 (1500 RE, 900 PE), Martha 30, Robert 4, James 3/12, and C. C. 24, laborer. I am not sure who C. C. Smith is. Next door is Alex Morgan, 21, Lucinda 18, and Martha 1, who is probably W. A. Morgan, son of Stephen's sister Letty and Griffin Morgan. In the neighborhood is Emlay Russell 47 and Martha 35, who may be related to Miles's wife Martha Russell. 
1870 Clay County, Alabama Census (Ashland): Stephen Smith 67 (farmer, 800 RE, 695 PE), N. W. 58, M.A. 31, M. J. 28, and 
A. B. Tate 2; L. L. 24 (teacher), M. M. 20 (works farm), J. A. 16 (works farm), W. M. 12, E. Arnold 38 (Elizabeth Ann, daughter), K. Arnold  9,  G. W. H. Arnold 8. In the neighborhood is Stephen's nephew Leonard Morgan 23, wife Cynthia 22, and daughter Ella 8/12. 
In the household of W. T. Whatley, there is a J. C. Smith, 55, born in SC (carpenter).  I do not know how he fits into the family. William Taylor Whatley (W. T.) was the son of Martin Whatley and Stephen's sister Sophia. 

Father:  William Jackson Smith
Mother: Elizabeth (Betsy) Ann Smith

Spouse:  Nancy Whatley
Married: 3 October 1826

Children:
1. Levi A. Smith
2. Miles L. Smith (m Martha E. Braswell)
3. Elizabeth Ann Smith (m George Lewis Arnold)
4. Martha Ann Smith (m // Lowery)
5. James H. Smith (not married)
6. Hiram John Smith (not married)
7. Mary Jane Smith (not married)
8. Lovey Louise Smith (m William B. Hardegree)
9. William Taylor Smith (m Nettie Frost)
10. Martin Marcus Smith (m Alice Herring; Rena Herring)
11.  John Almon Smith
12. Wiley Smith (m.Mattie Bruce, Vembry E. Bunn)