This is an expression that comes from the Bible. When you say that you are going to separate the sheep from the goats, what you mean is that you are going to separate the good from the bad. It is also possible to say, “sort out the sheep from the goats”. These two expressions have the same meaning as “to separate the wheat from the chaff”.
Source: “The Hindu” – Know Your English Column – Feb 12, 2008