The One Direction star briefly dated the 23-year old pop singer between October 2012 and January 2013.
The 19-year old British heartthrob has allegedly told friends that he regrets being involved with Swift who continues to poke fun at him in her music.
A source told RadarOnline, "Harry might have taken Taylor's swipes with an air of grace, but he now wishes he never ever dated her."
"I mean, the fact she's aiming a jab at him in the 22 video has got Harry wondering what's coming next! He just doesn't know what to expect."
The source continued, "It wasn't the most amicable of splits, but Harry said all along that he wanted to take things slow and easy with Taylor."
"There was certainly no commitment made, despite Taylor's protestations and the fact that they were so busy with their careers and lived in different countries meant that it could never have really got that serious."
"What for Harry was 'a bit of fun' at the time has slowly turned into a personal nightmare. But he won't stoop to Taylor's level, he's just going to continue to be the more grown-up person and get on with his life. Perhaps something Taylor should consider too."
The song, released March 16, is taken from Taylor's fourth studio album, "Red".