This is just my personal opinion, but I kind of like pearly grips. I know they're a lot harder to make and have a tendency to crack and chip out at the corners. We've got a friend up here who sidelines as a dentist (he's a full time scoundrel who covers as many gun shows as he can.) Over the past few years he's wandered out of gun shows on Friday or Saturday, coming back the next day with the grips he took home. He somehow knows how to fix the bad places with the same stuff they use on teeth. Some kind of epoxy and I think UV light. If your a real good eye, you can sometimes tell where the fix went in. But only if you knew before.
And I agree, Ivory is a wonderful substance. Its a lot like the soap, only heavier. It files, grinds, and sands at a very predictable rate. You can even polish it with a rag and one finger, using Flitz or Simichrome. Better still, if you get lazy and don't use a progression of finer sandpaper, it still polishes up so good it takes a magnifier to tell. I don't think there is much of a health risk from inhaling the dust. Probably no worse than human tooth grinding, and there sure are a bunch of retired dentists floating around.
Pearl is a toxic substance. I have no idea how much you can breath in before you croak, or flop around on the floor like a carp. I've never been afraid to use a dremel tool on it, but for polish only. I do have this fear of it getting hot and splitting out. Buffing it lightly shouldn't be bad.