In a small saucepan, add in the water, sugar, cinnamon stick, vanilla, ginger, cardamon pods, cloves, and black peppercorns. Bring to a boil, then reduce the heat and add in the teabags. Simmer for 10 minutes, uncovered.
Remove the teabags and strain out the spices. Allow the chai tea to cool until it is warm to the touch. Stir in the hazelnut milk. (Hazelnut milk often "separates" when mixed with other liquids, and the "separation" is exacerbated when added to liquid that is too hot. Allowing the chai tea to cool will temper the "separating" effect.)
Pour the hazelnut chai into popsicle molds. Freeze for at least 6 hours, or until frozen. If you prefer a creamier texture, every 30 minutes for the first 2 hours, remove the popsicles from the freezer and stir the mixture to break apart any pieces of ice.