Splendid the isolation may be at the great Palladian hall and estate in Norfolk – but a sense of purpose is missing without visitors, write its chatelains