Born in Atlantic City, New Jersey, Albany studied piano as a child and, by 1943, was working on the West Coast in Benny Carter‘s orchestra. In 1946 he at least once played with Parker and then 20-year-old Miles Davis. He continued for a few years afterward, and in 1957 recorded an album for Riverside with an unusual trio line-up with saxophonist Warne Marsh and Bob Whitlock on bass, omitting a drummer. Despite that, most of the 1950s and 1960s saw him battling a heroin addiction, or living in seclusion in Europe. He also had several unsuccessful marriages in this period. He returned to jazz in the 1970s and played on more than ten albums. He died of respiratory failure and cardiac arrest in New York City at the age of 63.