First question, is it a new gun with problem at outset or older gun which once shot well and then developed this problem? You apparently have at least one other similar gun. How does it perform for you?
Anyhow-
1. Is this consistent? That is, if you aim at a specific point, do the bullets always go to another specific point? In gunfolk talk, how large is your pattern?
2. If someone else shoots your gun, do they have results similar to your results?
If 1 and 2, you may have a sight adjustment problem, however You seem to be describing a problem beyond the range of sight movement.
If your pattern is large and inconsistent your gun may not be returning to the same battery position, that is an indication that your gun needs cleaning or major repair. Without pointing in a dangerous direction between shots, can you verify that the barrel is in exactly the same position from shot to shot? My limited analysis of my SD9 tells me that the barrel position is sensitive the the behavior of the slide blocker, that is the little take down bar. Have a look at that.
Anxiously waiting for your response before asking more questions.
Have you made any "upgrades" to that gun? If so, when?