Caller ID is not a reliable method of authentication when scammers can spoof any number they like. People with reservations for High Tea at The Ritz discovered this as a result of an attack that exploited information about their reservations.
There is one thing about phishing and social engineering scams against customers or external partners that impersonate your organization, which makes them impossible to detect with technology...