In my job with the U.S Forest Service here in Southeast Alaska, I deal with outfitters and guides who hunt on the Tongass National Forest. Because I have an interest in firearms, I like to talk to them about rifles, handguns, and shotguns for both hunting brown bears, and for protection. There are widely varying opinions on what should be used for hunting bears, for stopping bears, and for last-ditch defense. After listening to these guys, and working in the woods myself for the last nineteen years, I have developed some views as to what works and what doesn't.
First off, there is a difference between HUNTING bears, and being a guide who needs a STOPPING rifle for dealing with wounded bears. Although there are a couple of guides who insist their clients bring a .375 H&H for hunting, they are in the minority. Most are happy as long as the person brings a reasonably large rifle WITH PROPER AMMUNITION, and can shoot well. For them, they will be very happy with your choice of a .300 Magnum as long as you shoot it well and you use bullets designed for expansion combined with deep penetration. Hopefully you know that not all bullets are created equal, and you don't pick a load with a standard el-cheapo 180 grain bullet. Use a Nosler Partition, Swift A-frame, Kodiak, Barnes Triple-Shock, or some other kind of high-tech bullet desiged for expansion AND deep penetration. The bullet should be 180 grains at the lightest, and heavier is better. Obviously an all-copper bullet like the Barnes Triple-Shock can be a bit lighter than those with lead in them and still do the job. For brown bears, however, I still wouldn't go lighter than 180 grains.
For your handgun, people are telling you the right thing when they say you need penetration. I don't know what is available in the 10mm, but look for a heavy bullet with a big flat point. At the low velocities of a handgun, a round nose bullet separates muscle tissue as it passes through. Muscle tissue is elastic. After a round nose bullet passes, the tissue snaps back leaving little evidence of a wound. A flat point bullet, however, goes through like a cookie cutter severing the muscle tissue creating a hole that lets blood out and air in. You do not want any handgun bullet that expands when dealing with brown bears.
Most people don't realize that most expanding bullets for handguns, that aren't designed for self defense against people, are designed to work on whitetail deer. A few years ago here in Southeast Alaska, while packing out a deer, a guy shot a charging bear in the side of the head with a 240 grain JHP from a .44 Magnum. It was a sow that had twin cubs almost the same size as her. The bear went down, and he beat feet away from the cubs, who were milling around trying to figure out what to do. He went back to camp to get his buddies to help him recover the bear head and hide, as required by Alaska law when you shoot a bear in defense of life and property. When he and his friends got back to the scene, mom was back on her feet and she and the cubs were happily feeding on his deer. She didn't look very worse for wear, so they let the bears have the deer and they retreated. From what they can figure (obviously they couldn't autopsy the bear) the bullet hit the large jaw muscles, expanded but didn't break bone, but it knocked the bear out for awhile. Everything happened to work out OK, but this result isn't anything someone can depend on. The guy started carrying heavy LBT style bullets (big flat point lead bullets that don't expand) in his .44 from that point on.
As for leaving your handgun at home... I wouldn't. Since you will be carrying a rifle, the handgun is only for last-ditch defense. You won't use it unless something goes to hell in a handbasket. It is going to be far more effective than your fingernails.
