The primary issue I have with "If you knew what I knew" is that it implies "You'd obviously agree with me if you were as enlightened as I, but since you're not as enlightened, you're not entitled to any opinion on the matter."

It can legitimately happen, but there are ways (as you've mentioned above) to express it without insulting the other person.