Falcons of Narabedla by Marion Zimmer Bradley

Mike Kenscott is having a really bad vacation. One minute he's camping in the Sierra Mountains with his brother Andy, and the next minute he's on a different world - or in a different time - or both. He's also in a different body.Now he's Adric, Lord of the Crimson Tower, of the Rainbow City of Narabedla. He has to cope with his fellow Narabedlans: the Dreamer Rhys, the mysterious veiled Gamine, the dwarf Idris, his brother Evarin the Toymaker (whose Toys are deadly), and Karamy, the golden witch, who is either his lover or his greatest enemy - with most of both his and Adric's memories gone, he's having trouble knowing what she is. Then there are the people outside the city, led by the man called Narayan. Mike/Adric knows that they are important to him, but he desperately needs to remember why.