In early 1935, Gray began attending Northeastern High School, and then transferred to Cass Technical High School, which is noted for having Donald Byrd, Lucky Thompson and Al McKibbon as alumni. He left in 1936, before graduating. Advised by his brother-in-law Junior Warren, as a teenager he started on the clarinet, but after hearing Lester Young on record with Count Basie, he was inspired to switch to the tenor saxophone.
Gray’s first musical job was in Isaac Goodwin‘s small band, a part-time outfit that played local dances. When auditioning for another job, he was heard by Dorothy Patton, a young pianist who was forming a band in the Fraternal Club in Flint, Michigan, and she hired him. After a very happy year there, he moved to Jimmy Raschel‘s band (Raschel had recorded a few sides earlier in the 1930s but did not do so again) and then on to the Benny Carew band in Grand Rapids, Michigan. It was at around this time that he met Jeanne Goings; together they had a daughter, Anita, who was born in January 1941.