On his death the line of the eldest son of the first Baron failed.

Peter Campbell Scarlett, third son of the first Baron.

Robert Liddell, fifth son of the first Baron.

Robert Dormer, third son of the first Baron.

He was the great-grandson of the third son of the third Baron.

Sir Robert Vivian, illegitimate son of the first Baron, was also a prominent soldier.

The artist John Collier was the younger son of the first Baron.

Edward Cust, another younger son of the first Baron, was a soldier and politician.

He was the grandson of a younger son of the second Baron.

Joicey was the second son of the first Baron.