Well, if the rifle grouping gets better, then worse, it's usually the shooter. Fatigue may be a factor for you after 20 rounds.
How do you do with lighter recoiling cartridges/rifles?
I don't have anything else with a scope to test that out. After 20 rounds I just let rifle sit there due to heat.
Or it is the scope mount having some movement. I never did like the side mounts. Try getting a good grip on the scope and try to move it. Don't worry it should take some good strength. Try to wiggle it a bit. If you can get any movement you can bet that recoil will move it too.
I gave it a good pull (so to speak) and it seems nice and solid (so to speak again). Scope isn't moving.
Good for ya.

Is that a Soviet TT-33?
Nope. Romanian. It shoots very well in my hands and ammo is very cheap.
Also, thin barrels get hot easily, and get inaccurate when hotter. So, try shooting from a 'cold bore', meaning let it cool about a minute between shots, or even more. Do at least three "5 shot geoups
Usually I shoot 3 rounds and see how that works out. As I said before two mags (10 rounds each) barrel is untouchable due to heat. I'll do 5 rounds grouping next time and see where that takes me.