Downing Street has slapped down Liberal Democrat Cabinet minister Vince Cable over his claim that the migration target had a "damaging impact" on the economy.
A report by the Migration Observatory found the number of recently arrived highly educated migrants from outside Europe had dropped by more than a third since tougher visa rules came into force.
The Business Secretary said the findings of the report, commissioned by the Financial Times, were "deeply worrying".
The Oxford-based Migration Observatory analysis of the Labour Force Survey statistics found the number of highly skilled recent migrants from outside the European Economic Area fell from 154,000 in 2011 to 94,000 in 2013, a 39% decrease.
"There has been a negative trend in this number since 2011, when policies to reduce net migration to the UK presumably began to have the bulk of their effects," the report said.
But in an indication of the difficulties faced by the Prime Minister in meeting his target of reducing net migration to the "tens of thousands" , the number of recent highly-skilled migrants from the long-standing members of the EU - which are not subject to visa controls - rose by 53% from 51,000 to 78,000 over the same 2011-2013 period.
Mr Cable told the Financial Times the net migration target had "clearly had a damaging impact on UK PLC by reducing the talent pool available".
But the Prime Minister's official spokesman said David Cameron " wouldn't share that view".
The spokesman told reporters at a regular Westminster briefing: " We want a country that succeeds in the global race, is able to attract the brightest and the best and I think you see that we are able to do that.
"To give you one specific example, if you look at sponsored visa applications for skilled workers in 2013/14 they are up almost 20% on the previous year.
"I think alongside controlling immigration you can, and we do ... compete to attract the brightest and the best."
Asked if Mr Cable was wrong, the spokesman said: " The Prime Minister's view is the one I have expressed."