My wife and I will do a call back if we need to talk about anything sensitive over the phone. I can call her, say that I'll need an account number or whatever the situation calls for, then we hang up and she calls me back so we can both be sure we are talking to each other. If I don't know what she's talking about during the return call then it's a scammer.

This is rare, but it gives protection against this type of scam.