Born at Compton Place, Eastbourne, Sussex, Cavendish was the second son of William Cavendish, 7th Duke of Devonshire, by his wife Lady Blanche Howard, fourth daughter of George Howard, 6th Earl of Carlisle, and the brother of Spencer Cavendish, 8th Duke of Devonshire, who had also been Chief Secretary. Cavendish, after being educated at home, matriculated in 1855 at Trinity College, Cambridge, where he graduated B.A. in 1858, and then served as a cornet with the Duke of Lancaster's Own Yeomanry cavalry.