Long eclipsed by his fellow Nabis artists Bonnard and Vuillard, this Swiss painter and printmaker produced brilliant and unsettling work