Tried to keep any spoilers of the story of the Rings of Power out of what I quoted, by removing all mentions to the rings and who wielded them. We all know Gandalf eventually is given one of the Rings, but don't want to spoil who gave it to him or who the Elven Ring Bearers are, aside from Galadriel.
From the Appendices of the Return of the King, Appendix B:
"When maybe a thousand years had passed [from the Last Alliance and Sauron's defeat], and the first shadow had fallen on Greenwood the Great, the Istari or Wizards appeared in Middle-earth. It was afterwards said that they came out of the Far West and were messengers sent to contest the power of Sauron, and to unite all who had the will to resist him; but they were forbidden to match his power with power, or to seek to dominate Elves or Men by force and fear."
...The two highest of this order (of whom it is said there were five) were called by the Eldar Curunir, 'the Man of Skill', and Mithrandir, 'the Grey Pilgrim', but by Men in the North Saruman and Gandalf.... Mithrandir was closest in friendship with the Eldar, and wandered mostly in the West, and never made for himself any lasting abode.
... For Cirdan saw further and deeper than any other in Middle-earth, and he welcomed Mithrandir at the Grey Havens, knowing whence he came and whither he would return.
Also, from the Silmarillion:
"Even as the first shadows were felt in Mirkwood there appeared in the west of Middle-earth the Istari, whom Men called the Wizards. None knew at that time whence they were, save Cirdan of the Havens, and only to Elrond and to Galadriel did he reveal that they came over the Sea.
... Chief among them were those whom the Elves called Mithrandir and Curunir, but Men in the North named Gandalf and Saruman.. Of these Curunir was the eldest and came first, and after him came Mithrandir and Radagast, and the others of the Istari who went into the east of Middle-earth, and do not come into these tales.
... He [Gandalf] wandered far in the North and West and made never in any land any lasting abode; but Curunir journeyed into the East, and when he returned he dwelt at Orthanc in the Ring of Isengard, which the Numenoreans made in the days of their power."
Honestly, after rereading both of those, the Stranger could very well be a younger Saruman, seeing as there are multiple mentions of Cirdan meeting Gandalf at the Grey Havens in both references, but we are told Saruman came before him (though still in the Third Age) and he actually traveled to the East. I actually wouldn't mind this, because we've already seen the Stranger basically show he isn't as "good" as Gandalf is always shown to be.