An independent report commissioned by the Department of Health estimated these groups cost the NHS £2bn a year.
Ministers said some of that spending was unavoidable, but said it would be realistic to save a quarter.
The savings would come from a levy on non-EU visitors, deterring so-called health tourism and getting the NHS to recover money owed by other countries.
One senior doctor questioned the government’s figures and said doctors should not have to spend time “vetting eligibility” of patients.
Health Secretary Jeremy Hunt said he did not want to “turn GPs into border guards” and no-one would be denied emergency treatment, but foreign visitors must make a “fair contribution”.
‘National not international’
Mr Hunt said he recognised the NHS’s duty of care must “transcend” financial concerns, but collecting an extra £500m a year could pay for 4,000 extra doctors.
“We have one of the most generous systems in the world when it comes to healthcare for foreign visitors, but it’s time for action to ensure the NHS is a national health service – not an international one,” he added
The latest research is published in two separate reports by firms Prederi and Creative Research.
The government has been criticised for using financial estimates from the Prederi report, but Mr Hunt said it was the “most thorough research there has ever been” in this area.
The report itself notes that it is based on “incomplete data, sometimes of varying quality, and a large number of assumptions”, but it says the estimates are the “best that can be made at present”, and Mr Hunt said researchers had been “completely honest about what they do know and what they don’t know”.
‘Claw back’
The government is currently consulting on the measures it will take to tackle this issue.
It has already announced a £200-a-year levy on migrants from outside Europe staying for up to five years, and Mr Hunt said this could raise £200m a year.
A cost-recovery unit will also be set up to help hospitals claw back money they are owed by other governments for treating foreign nationals visiting the UK.
At the moment just £73m a year is recouped out of more than £460m owed.
Meanwhile, health tourism – whereby people deliberately travel to the UK to get NHS care – is estimated to cost between £60m and £80m.
The government accepts it cannot recoup all of this money and entirely stop health tourism and so it has put forward the “conservative” £500m figure.
To achieve this, it said a better system of identifying when visitors and migrants got treatment was needed. More details on what this will involve will be revealed in the coming weeks.
‘Exaggerated’
Shadow health secretary Andy Burnham said: “We are not against improving the recovery of costs from people with no entitlement to NHS treatment.”
But he added the figures were based on a “large number of assumptions” and was more about “spin than substance”.
Shadow Health Secretary Andy Burnham: “There is a need to look at the small print behind the spin”
Dr Chaand Nagpaul, chairman of the British Medical Association’s GPs committee, said the government’s £500m savings figure was “exaggerated”, partly because it did not take account of the costs of implementing its plans.
He said migrants paying the proposed £200-a-year levy could be more likely to use the NHS to “get their money’s worth”, and he said the cost of a single hospital outpatient appointment would be more than £200.
Dr Nagpaul said doctors should not be distracted from their work in order to check the eligibility of patients for treatments.
Jonathan Portes, of the Institute of Economic and Social Research think tank, said the research revealed the extent of deliberate health tourism had been “hugely overstated” and was in fact a “very small part of NHS expenditure”.
Source BBC News