The first thing to mention though, is that I don't believe in problem players. I'm not even sure I believe in problems, which I discuss here and here. I do believe in problematic behaviours, but I'm even a little cagey about that. I'd mark a behaviour inefficient or non-constructive before problematic, because they're usually better labels. It's often a behaviour that isn't helping the person achieve their goals, and we can take that to be a problem, or to be an opportunity to help them achieve what they're really after. I like to give people the benefit of the doubt, and find that very few people who might seem like jerks think that they're being jerks. People do things for reasons, and if we can understand what some of those reasons are, we can help.
Focus on the BehaviourFirst and foremost, this. I don't believe in problem players, but I do believe in problematic behaviours. And I don't have the moral high ground on this, because sometimes those problematic behaviours are mine. By focusing on what is happening rather than on what a person is like, we avoid labelling them and have an issue which can actually be resolved. This is especially true for players who are new to a game or to roleplaying in general. It takes time to learn the kinds of group dynamics which occur in roleplaying games, but by focusing on the behaviour, we can work with them to help them have more fun along with everyone else. A problem player can't be changed, they are what they are, but a problematic behaviour can be addressed.
Locate GoalsThe first thing I try and do is find out their goals. Behaviours are things we do to accomplish certain ends, and if we can talk about the goal they're trying to reach, then we can discuss whether or not that's the best way to reach them, and whether or not it's a good goal. Some goals can conflict with other, overriding goals. This is true in real life, where if it's my goal to get a new car, I still might not steal one, because I might have the goal of respecting people's property or staying out of jail, which would override it, and equally true in the context of a roleplaying game. I talk about some of the goals a GM might have here, and there are some pretty reasonable group goals, like respecting everyone's stake in the activity. If their goals do conflict with more general goals, it becomes a matter of helping them understand why certain goals are in place. For example, if someone doesn't have the goal of respecting the stake of others in the group, that harms everyone else's ability to achieve that same goal. Having it means that everyone has more fun beccause they're confident that people with try to respect their interests. Other conflicting goals might mean that the game isn't a good fit. If someone wants to fly an X-Wing in Greyhawk, and there's no mechanics which can express that, they might just be playing the wrong game for their needs. Take this step first, and it'll be a good one. And never ever presume to know what their goals are. Try and convince them to tell you, a lot of the time they'll surprise you, I wager.
In Game vs. Out of GameRoleplaying games carry with them two separate group dynamics, that of players and characters, and this can cause tension in different ways. Sometimes the behaviour isn't with the player, but with the character. It's still possible to discuss the goals of that behaviour, though characters often get more leeway in terms of respecting people's stake, because tension in a party can create interesting drama. And, if a character's behaviour proves problematic, it can be an opportunity for the other player characters to perform the process in game, helping them build a rapport with the character and creating great roleplaying situations. As well, each character always needs to have a satisfactory answer to "Why am I with this bunch of jokers?" If the behaviour seems like an unresolvable issue, it can result in the party parting ways with the character, which is unfortunate, but can be necessary. Sometimes a character idea just doesn't work with the group dynamic, and that's okay. An example of this is the Lone Wolf. In a four player co-operative game, the lone wolf who stays in the shadows and is a friend to no one maybe isn't the best bet for enjoying and participating in a co-operative game.
Avoid AuthorityAt all costs, avoid trying to exert authority as the GM. If only because you don't really have any. No one likes ultimatums, and it's often far easier to get people to acknowledge other people's stake in things rather than ordering them to shape up. It also doesn't help them understand what's going on or what they're expected to do. You can exert leverage as the GM, but when you do that, you're trying to get them to acknowledge your stake in the game, which is often a bigger investment than most of the players, though that definitely varies from group to group.
Group EffortIn the same vein, instead of exerting authority, try and make it a group effort. A problematic behaviour affects everyone in the group, because they all have a stake in it. It's not your job as the GM to resolve every issue that arises, rather it's everyone's job to try to make it a fun game. Once you can locate the goal of the behaviour, work with everyone in the group to see how they can be reached in a way that creates opportunities. Many hands make light work, and there's no harm in inviting people to be involved in something in which they have a stake.
Kicking People OutThis is pretty common advice on forums, and I'll be honest, it bothers me a lot. Kicking people out, whether as a group decision, or worse as a unilateral GM decision, isn't just the last resort, it's the last resort of last resorts. In fact, I'll go as far as saying that kicking someone out for behaviour related to the game is unethical, and should never be done. It involves disrespecting their stake in a way that no one would prefer. If they're not respecting the stakes of others, that makes it more important to respect theirs, because if you can't set an example, they have no reason to listen to you about it. This isn't to say that there aren't reasons to kick someone out, only that those reasons are the same reasons you'd kick someone out of a dinner party. Issues like sexual harassment or violence (I've heard stories) go beyond the game, and that can make it necessary to ask someone to leave, just as it would if they were at a dinner party.
The JerkBut what do you do with the player who's a jerk? The one who remains utterly obstinate and refuses to communicate, or who openly states that they're committed to having fun by harming the interests of others? I'll be honest, I've never met one. I've played with people who sometimes do jerkish things (one of them is me), but never one wholly committed like that, to my knowledge. I hope that some of the ideas here can help with that, but it's perfectly conceivable to imagine a person who resists all of them. If they're openly committed, then it seems like the dinner party logic applies. But I don't know that this is a real case. When I do hear about them, I often wonder what kinds of behaviour the GM and other players are engaging in that makes this possible or enables it in the group, and how that relates what the person trying to accomplish.
Wow, this one ran long. Still, I feel like it's useful, and if you've got some other tips, I'd love to hear them. I'm also interested in doing some posts that break these down, talking about how, and not just what. It's easy to say things like "Talk with them about it," but there's a lot that goes into that in terms of negotiating strategies, what to say and how to say it that isn't obvious.