The Dark Wizard says "5 of us there were" so if he's not the White Wizard, and not The Brown Wizard, he must be one of the Blue Wizard. The TV show is not chronologically linked to the books. The Rings of Powers were forged in about 1500 of the second age, and the fall of Numenor and happens in 3200 of the second age, but the show has them happening at the same time. Gandalf and the other wizards don't come to middle earth until 1500 of the third age, but in the show they are already there in the second age.