A lot of things to consider. Regardless of which material you choose, I believe rigidity of design is important. When I first began carrying I went with the soft suede holsters that collapse. Mainly this was so that I could see if I liked the IWB type holster before spending on something more expensive. I didn't consider the inconvenience of taking the holster off to re-holster, as I like to practice from a carry rig before actually carrying it. So I believe that is an important consideration.
Leather does tend to flex to fit your body better. For years this is why I stuck to leather, for the comfort issue.
Down side of leather, in the summer if your climate is at all hot, you will sweat and the holster will get wet with sweat. This, for me at least, results in the gun carried around all day in sweat soaked leather. This won't happen with kydex, at least it doesn't for me.
Another advantage of kydex, it can be thinner and still be plenty rigid to stay open after the pistol is drawn. Depending on cover garments, size of the pistol, position of carry, this added thinness can help with concealment.
Personally, I was concerned for sometime with the comfort level of the kydex, and for that reason resisted trying it for some time. Now, that is all I use.
Either way, consider a holster that puts the loops or clips on the outer edges of the holster body. Think pancake holster, only inside your pants. When the loops or clips are on top of the holster, this adds to bulk. Also, with the loops or clips at the outer edge of the holster body, the holster feels more stable to me and less prone to shifting. This in turn goes a long way to eliminate the feeling of needing to adjust all the time.
This is of course is just my opinion from carrying concealed for several years. I have the usual box of holsters that didn't work out.
Of course for some people it doesn't matter the material, as IWB just doesn't work for them.