Yes you can do M.Sc. Mathematics in IIT. For getting admission to M.Sc. programs you have to write JAM examination. Eligibility for admission is Bachelors degree with Mathematics as a subject for at least two years/four semesters. At least 55% aggregate marks, without rounding off (taking into account all subjects, including languages and subsidiaries, all years combined) in the qualifying degree for General and OBC category candidates.