I use 4 grs. 231 with a 158 grain lead semi-wadcutter bullet. Muzzle velocity is about 810 fps from a 4" barrel, about like a standard pressure factory .38 Special load. I've never noticed this load being more than slightly smoky, if that, and that would be from bullet lube.
Depending on the source, there is considerable discrepancy with 231/HP-38 data and a 158 gr. LSWC bullet. Hodgdon recommends a maximum of 3.7 grs. while most others sources list heavier maximum charges, even for standard pressure loads. Granted, different bullets of the same style and weight may require at least slightly different data, but Speer shows 4.3 grs. max. in book #13 for 231.
3.2 grs. 231 used by the original poster is a very light but safe charge, probably getting 650-700 fps from a 4" barrel. I certainly wouldn't go any lower.