I dislike speaker phone, as a rule. What I'm hearing is "This phone call is so unimportant, I can't even be bothered to hold the receiver."
What's worse is that, unless you're an important corner-office type (lucky you) chances are that someone near you can hear your conversation. Now you're being rude to the person on the line AND every other person in earshot. If your office doesn't have a door, you don't get to use the speaker phone. It's just that easy.
There's a website called Smart, Pretty, and Awkward, which gives some good advice (although I don't seem to have the same fashion aesthetic as the author), including things like this. Maybe I should write an unsolicited advice column. Maybe that's my key to blogging fame and fortune, eh?