Born in Farmersville, Texas, and raised in the suburbs of Dallas, Ellis first heard the electric guitar performed by George Barnes on a radio program. This experience is said to have inspired him to take up the guitar. He became proficient on the instrument by the time he entered North Texas State University. Ellis majored in music, but because they did not yet have a guitar program at that time, he studied the string bass. Unfortunately, due to lack of funds, his college days were short-lived. In 1941, Ellis dropped out of college and toured for six months with a band from the University of Kansas.