Absolute beginners, Beat Detective

[Wire57]

I had hoped that my last post in my thread "Absolute beginners questions" would go noticed, but apparently it didn't.
After many more trials I messed up my session so much I had to delete it completely. So here comes my unsolved question. The screenshots are from the now deleted session, but the project I still wish to do.

So my next question seems to be tricky - maybe it's not, but for me it is.
Concerning the Beat Detective.
I am in an adventurous project with a young guy from Zimbabwe who sent me an audio track of himself singing and being accompanied by a guitar, and he plays some Mbira (Kalimba) pattern to his song.
I am trying to have some instruments to this and started with adding a drum.
Of course, for synching the audio with the added instruments in ProTools, the Beat Detective is my choice.
And for the first measures, everything works well.
But then, all of a sudden, the beat detective finds a tempo change which is, from the audio, absolutely not there. But this tempo change causes the drum to be ahead of the audio for the rest of the session.
Even worse, everytime a new (false) tempo change is found by the Beat Detective, the aberration of timing is being added to the previous ones in the session.
I am adding 2 labeled screenshots to this post to make you see what is going wrong.
The bounce of this session is here:
https://soundcloud.com/werner-linden/wirirano01test.
Now I had hoped when changing from "grid" mode to "slip", or "shuffle", I could grab those time markers and move them to the correct position, but then they are gone. Would I have to try to enter the speed line and try to change the speed the detective has found?