Born in Atlantic City, New Jersey, Albany studied piano as a child and, by 1943, was working on the West Coast in Benny Carter‘s orchestra. In 1946, Lester Young recorded with Albany as his pianist, for Aladdin Records. Also that year, he played at least once with Parker and then 20-year-old Miles Davis. He continued for a few years afterward, and in 1957 recorded an album for Riverside with an unusual trio line-up with saxophonist Warne Marsh and Bob Whitlock on bass, omitting a drummer. Despite that, most of the 1950s and 1960s saw him battling a heroin addiction, or living in seclusion in Europe. He also had several unsuccessful marriages in this period. He returned to jazz in the 1970s and played on more than ten albums. He died of respiratory failure and cardiac arrest in New York City at the age of 63.
Albany was the focus of a 1980 documentary titled, Joe Albany… A Jazz Life. His daughter Amy-Jo wrote a memoir about her father called Low Down: Junk, Jazz, and Other Fairy Tales from Childhood. The book was adapted for the screen and released in 2014 as the biopic Low Down. Joe was also the biological father to Benjamin David Goldberg, who was adopted by another family shortly after being born. Benjamin David was also a musician, who studied percussion at Juilliard, played for Broadway shows and was in the US Army Band.