Elgort was born in New York. His parents are Arthur Elgort, a fashion photographer who worked extensively for Vogue for over 30 years, and Grethe Barrett Holby, an opera director. Elgort’s father is of Russian Jewish descent and his mother is of Norwegian, English and German descent.