Since 1919, the house had been used as Council offices and from 1979 a restoration programme began and preparations were made for it to open in 1982 as a unique combination of historic house and modern art gallery. In 1985 an independent trust, consisting of the Friends and representatives of the Council, was formed to manage the Gallery.

in 1989, another gift followed - from Charles Kearley, a local businessman. His collection included works by international artists such as Paul Cézanne, Paul Klee and Gino Severini, complementing the Hussey bequest and contributing to the Gallery's growing status as a ‘collection of collections'.

By the end of the 1980s it was clear that the success of the Gallery meant it would soon outgrow the confines of the townhouse. So when Professor Colin St John Wilson, the architect of the British Library, offered his extensive collection of Modern British Art in the 1990s it was further incentive to build a new wing in order to show the collections in their entirety. The Gallery reopened in Summer 2006 with a new wing and vastly improved facilities.