In 1872, he started working with Spencer Baird, soon becoming his trusted assistant.

At Fort Hall, Hayden wrote Spencer Baird in his Report No. 10 (excerpted):

The colony was discovered by Spencer Baird in 1845.

Specimens were taken that might be of interest to Spencer Baird and his fisheries work.

Spencer Baird and Ichthyology at the Smithsonian 1850-1900.

He sometimes had to visit the Smithsonian Institution where he met Spencer Baird, who knew all about fish and birds.

He was friends with Spencer Baird and J.G. Bell.

As a result, Spencer Baird's Fish Commission was given 20 acres of land.

Out of all the three secretaries that he worked with, he worked closely with Spencer Baird.

Spencer Baird was a successful ornithologist and Brown spent a lot of time assisting him.