I clean that part till it shines. I have not cleaned it on the inside though, just always on the outside. I will make an effort to clean it on the inside of the tube as well. Then I will polish the outside. There can't be any factory lube on the outside of this part, or on the rails of this gun left, as I've cleaned this pistol a couple of times now.
So the problem can most likely be dirt on the inside of this part?
I will surely look into this.

Edit: This morning before going to work I quickly looked at the possible issue.
I noticed that when slowly pulling back the slide, it comes to about half-way, then the amount of force needed doubles till it is fully cocked. As if there is something that wants to block it from cocking all the way. Should the cocking action be like this, or smooth with a constant force all the way?