Mosley was born in Little Rock, Arkansas, United States. He played trombone in high school and then joined Alphonse Trent‘s territory band, playing with him from 1926 to 1933. Following this he played with the Jeter-Pillars Orchestra (1934), Claude Hopkins (1934–35), Fats Waller, and Louis Armstrong with the Luis Russell Orchestra (1936–37), in addition to playing with his own groups. After this Mosley settled in New York City.

Mosley spent most of his career on trombone, but also invented an instrument called the slide saxophone, which had both the slide portion of a trombone and a saxophone mouthpiece. The instrument is prominently featured in his 1940 recording The Man with the Funny Little Horn.