Fox was born in the parish of St George's, Hanover Square, Westminster, the son of Arthur William Fox and Hilda Louise Fox (formerly the actress Hilda Hanbury, real name Hilda Louise Alcock), a member of Herbert Beerbohm Tree's theatrical company), and was the grandson of Samson Fox (1838–1903), a British engineer and philanthropist, principal founder of the Royal College of Music and inventor of the corrugated boiler flue. His mother and his aunt Lily Hanbury were first cousins of Julia Neilson, mother of Phyllis and Dennis Neilson-Terry. Julia Neilson was married to Fred Terry, brother of Dame Ellen Terry. Altogether seven of his cousins were actors.