Scott began playing in small jazz clubs at the age of 16, his claim to fame then being that he was taught to play by “Vera Lynn’s father-in-law!”. Scott toured with trumpeter Johnny Claes from 1944 to 1945, and with Ted Heath in 1946. He also worked with Ambrose, Cab Kaye and Tito Burns. Scott was involved in the short-lived musicians’ co-operative Club Eleven band and club (1948–50), with Johnny Dankworth and others. He was a member of the generation of British musicians who worked on the Cunard liner Queen Mary (intermittently from 1946 to around 1950) in order to visit New York City and hear the new jazz music that was emerging directly. Scott was among the earliest British musicians to be influenced in his playing style by Charlie Parker and other bebop musicians.

In 1952, Scott joined Jack Parnell’s orchestra, and from 1953 to 1956 led his own nine-piece group and quintet which featured among others Pete King, with whom he later opened his jazz club, Victor Feldman, Hank Shaw and Phil Seamen. Scott co-led The Jazz Couriers with Tubby Hayes from 1957 to 1959, and was leader of a quartet that included Stan Tracey (1960–67).

During this period Scott also did occasional session work; his best-known work here is the solo on The Beatles’ “Lady Madonna”. Scott also played on film scores, including the score for Fear Is the Key, composed by Roy Budd. Scott continued to be in demand for guest appearances in later years, such as providing the tenor sax solo on Phil Collins’s 1981 hit single “I Missed Again”.