The son of an Italian-American instrument maker, Lang was born in Philadelphia, Pennsylvania, and grew up with violinist Joe Venuti. His first instrument was violin when he was seven. He performed on violin in 1917 and became a member of a trio. In 1920, he dropped the violin for banjo and worked with Charlie Kerr, then Bert Estlow, Vic D'Ippolito, and Billy Lustig's Scranton Siren Orchestra. A few years later, he dropped the banjo for guitar when he became a member of the Mound City Blue Blowers led by Red McKenzie. He recorded one of the first solos in 1924 on "Deep 2nd Street Blues". His performances with McKenzie's band drew attention, and he found many jobs as a freelance guitarist. Before Lang, the guitar hadn't been a prominent instrument in jazz bands and dance orchestras.

Lang and Joe Venuti recorded with Roger Wolfe Kahn and Jean Goldkette and performed with the Adrian Rollini Orchestra. Lang recorded with blues guitarist Lonnie Johnson under the name Blind Willie Dunn to hide his race and as a tribute to blues guitarist Blind Lemon Jefferson. He also worked with Frankie Trumbauer, Hoagy Carmichael, Annette Hanshaw, Red Nichols, Jack Pettis, Bessie Smith, and Clarence Williams.

Friendship with Bing Crosby

In 1929, Lang and Venuti became members of the Paul Whiteman Orchestra, and again Lang made an impact. Whiteman was impressed by his ability to learn songs quickly, though Lang had little education and couldn't read music. During the same year, vocalist Bing Crosby made his first solo recordings. His guitarist was Snoozer Quinn, but for the second session he invited Lang. Their friendship grew when Crosby joined the Whiteman Orchestra on its trip west to Hollywood to make the movie King of Jazz, in which Lang and Venuti appeared. In 1930, when Crosby was looking for a job in radio, he insisted on having Lang as his accompaniment. Aside from his friendship with Crosby, he had experience accompanying vocalists, such as Rube Bloom. When Crosby toured soon after, Lang sat on a stool next to him to share the microphone. Lang's wife Kitty, a Ziegfeld girl, was friends with Crosby's wife, Dixie. He became a regular in Crosby's orchestra in 1932, the same year he appeared in the movie The Big Broadcast (1932).