The world's refugee population swelled to more than 13 million last year, says a report on refugees that sharply criticizes U.S. policy toward those seeking asylum.

The report by the U.S. Committee on Refugees says the number of refugees worldwide increased from 11.7 million in 1986 to 13.3 million last year, but says that many of those seeking asylum in the United States are discouraged by harsh policies of the Immigration and Naturalization Service.Citing research by the General Accounting Office, which watches government agencies for Congress, the report complains that INS detains ordinary refugees and criminals together. The report also accuses some guards hired by contract with INS of "gross negligence to duty, gross misconduct and suspected collusion in detainee escapes."

The committee is an advocacy and monitoring group that receives support from the Ford Foundation and help from private and religious groups concerned with refugees. It says that under current U.S. policies, it is not surprising that some refugees agree to return home.

"Although technically voluntary, the evidence strongly suggests that many of those departures are coerced," it says.

In addition to 32,000 Salvadorans deported from the United States since 1977, many others have abandoned their claim before receiving a chance to explain why they fled and qualify for asylum, the report says.

"Measures such as detention deter not only those who abuse the immigration system without any justification, but also bona fide refugees," writes Bill Frelick, assistant editor of the committee's survey.

Verne Jervis, spokesman for the Immigration and Naturalization Service, said at the beginning of last week that he would supply some comment on the survey, but he had not done so by the end of the week.

The major increases in refugees in 1987 occurred in Africa and Asia, the report says.

The biggest new wave was fleeing a rebellion in the southeast African republic of Mozambique, once a colony of Portugal, the report says.

The situation there has divided U.S. political leaders, with some such as Sen. Edward M. Kennedy, D-Mass., saying the rebellion is being fomented by South Africa's military.

Conservatives, meanwhile, describe the Mozambique government as pro-Soviet and the rebels as freedom fighters.

The committee says about 350,000 refugees fled from Mozambique to Malawi last year, the biggest single new group recorded in the survey.

The largest displaced population, meanwhile, consists of 2.8 million Afghan refugees in Pakistan, the report says. They have fled the rebellion against their government and the Soviet troops who support it.

Other large groups of refugees include 2.6 million in Iran, most of them from Afghanistan, and 817,000 in Sudan, most of them from Ethiopia.