Walter Hussey, the Dean of Chichester Cathedral, left his personal collection to the city in 1977 with the condition that the collection be shown in Pallant House, a Grade 1 listed Queen Anne town house dating from 1712. Since 1919, the house had been used as Council offices and from 1979 a restoration programme began and preparations were made for it to open in 1982 as a unique combination of historic house and modern art gallery.
In 1985 an independent trust, consisting of the Friends and representatives of the Council, was formed to manage the Gallery. Since then the collections and the Gallery's activities have expanded to the extent that it was decided a new building was needed in order for it to survive. The Gallery reopened Summer 2006 with a new wing and vastly improved facilities.