Tempo mapping help

[Raoul23]

Hi guys

A session i had yesterday we track drums with the digital kit and the drummer struggled to play to a click. In the end i let the band play altogether and decided i would tempo map the session with identify beat. Everything is all good here so far but every no and then when the drummer does a drum fill he either speeds up or slows down quite a bit. What i'm trying to do is even out these fills but when i move the tempo of the bar to either slow it down or speed up it moves the tempo of the previous bar. Anyone on here shed some light advice on this and how i can just smooth out certain bars. At the moment im working with midi.

Thanks in advance