Construction of the Castle began as a roughly rectangular fortified house with towers in each corner. The original plan may never have been finished and by 1558 it is likely only the southern tower remained. In 1580 the south wing was rebuilt in Elizabethan architecture style, and around 1630 the eastern range was rebuilt in three story Inigo Jones style. The Elizabethan wing remained a bailiff's residence until 1905, but the eastern range was partly dismantled on the completion of the new house in 1843 leaving the ruin as a garden feature.
Catholic Recusant owner Thomas Darrell hid Jesuit Father Richard Blount S. J. in the Castle while he administered to Roman Catholics from 1591 to 1598. Catholicism was then illegal in England, and during the second raid by authorities to arrest the Father he fled over a wall into the moat and escaped.
The Darrell family owned the estate for some 350 years. In 1778 Edward Hussey bought the estate and his grandson, also Edward, built the 'new' Castle to the designs of Anthony Salvin from sandstone quarried from the slope below. The hollow created was developed into a Quarry Garden and contains a 100 million year old impression of a dinosaur's footprint.
On Christopher Hussey's death in 1970 the estate was left to the National Trust.