The Thomas Brown House is a two-story dwelling with two interior chimneys and a gable roof. Built c.1860, it was originally located on the north side of Frank Street, in the middle of what is now Queen Anne Square, a short distance away from its current site. When the Newport Restoration Foundation (NRF) purchased the house from the Redevelopment Agency of Newport in 1978, there was a good amount of detail in place on both the interior and exterior of the house. Restoration began that same year and was completed in 1979.

On the exterior, there are Greek Revival elements, as well as features common to many Victorian period Newport houses. The arch-topped sash in the dormers and parts of the front entry particularly represent Victorian period detail. The interior of the house also has a mix of Greek Revival and Victorian detail, indicating a transitional building or one that was subjected to changes from one style period to the next. Most of the mantelpieces reflect Greek Revival tastes, while the moldings are a mixture of both periods.

The city block on which the Brown House was originally located was dense with commercial buildings, warehouses, barns, sheds, and a few houses. As the development of Queen Anne Square began, buildings that occupied the area bordering Trinity Church, Thames, Mill and Spring Streets) were moved or demolished. Many on Thames Street were lost to fire. Only those few owned by the church remained.

When the Brown House was moved from Frank Street to Mill Street, a small barn on the property at Frank Street was moved to Cozzens Court, off upper Thames Street . Referred to as the NRF Barn, it still serves as a small workplace for NRF staff.