The oldest surviving brick building in Salem, Mass., the Derby House was built by affluent merchant Captain Richard Derby in 1762 as a wedding gift for his son, Elias Hasket Derby, and Elizabeth Crowninshield Derby.

The couple had seven children. Hasket established himself as a merchant, amassing a sizeable fortune to become one of the country’s richest men at the time.

However, all of the history, even the not-cool part, should be noted. The Derbys had at least two slaves in this residence, and Haskett made much of his money selling items produced by slave plantations in the Caribbean.

The Derbys lived here for nearly 20 years before moving to a larger house. After that, they built a mansion in the center of town.

Hasket called this place “the little brick house,” and sold it to merchant Captain Henry Prince in 1796. It changed hands over the years, serving at one point as a tenement house.

Ultimately, it wound up in the hands of the Society for the Preservation of New England Antiquities (now Historic New England), and restored to its 18th century grandeur.

In 1937, SPNEA transferred the house to the newly formed Salem Maritime National Historic Site, the website says.