The analogy simply refers to a dog chewing endlessly on, and "picking clean," a large bone. A "bone to pick" is thus a subject or issue that is expected to require considerable discussion or argument.
Two people will debate or argue until there is nothing left to argue about on a subject or issue. Usually, when a person introduces a conversation that way, expect a complaint or a grievance to come up. Make no mistake about it, it is never about anything good.
The idiom means to have a small dispute to settle with someone. The expression is an introduction to what is coming next.
The next time someone tells you he has a bone to pick with you, be prepared for a debate of some kind.