The Guggenheim explores French mystical symbolism – and looks beyond the famous figure of Joséphin Péladan