Vietnam, which once called the situation in Cambodia "irreversible," held its first peace talks Monday with leaders of three guerrilla groups fighting its decade-old occupation of that country.

Foreign Minister Nguyen Co Thach of Vietnam searched for compromise in an opulent reception room at the late President Sukarno's summer palace with the guerrilla leaders and foreign ministers of non-communist nations that back them.The informal "cocktail party" talks weren't confirmed until the last minute, but they have raised hopes for ending a monotonous cycle of dashed diplomatic moves followed by bloody fighting in Cambodia's malarial jungles.

Vietnam maintains more than 100,000 troops in Cambodia to support a government it installed after invading the country on Christmas Day, 1978.

Around tables arranged in a circle sat Thach; Hun Sen, prime minister of the Vietnamese-installed government; a representative of Vietnam's communist ally, Laos; and the guerrilla leaders: Khieu Samphan of the communist Khmer Rouge, Son Sann of the Khmer People's National Liberation Front, and Prince Norodom Ranariddh, who is representing his father, prince Norodom Sihanouk.

The guerrillas are grouped in a resistance coalition that the United Nations recognizes as Cambodia's legitimate government.

Foreign Minister Ali Alatas of host Indonesia moderated the talks, also attended by senior foreign ministry officials of the other members of the Association of Southeast Asian Nations: Thailand, Malaysia, the Philippines, Singapore and Brunei.

The association has called Vietnam's occupation of Cambodia the region's major security threat during the past decade.

The parties all sat quietly in the carpeted, double-chandeliered room as waves of journalists and photographers were rushed in for a glimpse. Thach appeared bemused, and the affable Alatas chatted and tried to ease some of the stiffness of the first encounter.

No definite schedule has been fixed for the duration of the talks.

For years, Thach was a tough hardliner who knew only the language of defiance. Over the past 1 1/2 years, however, he has vigorously pushed for dialogue as Vietnam's pragmatic new leadership focuses on economic development, which it admits requires an end to the international isolation Cambodia has caused Hanoi.

For years, the Association of Southeast Asian Nations has encouraged the guerrillas to force Vietnam out of Cambodia.

The association became more anxious for talks as Vietnam began withdrawing large numbers of troops and pledged a total pullout by early 1990.