Mitchell was born in New York City. His younger brother, Whitey Mitchell, also became a jazz bassist.
Mitchell was raised in New Jersey by a father who was an engineer and loved music, and a mother who loved poetry. His first instruments were piano, alto saxophone, and clarinet. Although Cornell University awarded him an engineering scholarship, by 1947 he was in the U.S. Army playing bass. The next year, he was in a jazz trio in New York City.
Mitchell performed and/or recorded with Mundell Lowe, Chubby Jackson, Charlie Ventura, Woody Herman, Red Norvo, Gerry Mulligan, and, after joining the West Coast jazz scene in the early 1950s, with André Previn, Shelly Manne, Hampton Hawes, Billie Holiday, Stan Seltzer, Ornette Coleman, and others such as Mahalia Jackson. He also worked as a bassist in television and film studios around Los Angeles, occasionally appearing on screen. Mitchell also appeared in documentaries about Tal Farlow and Zoot Sims.