The United States' decision to open talks with the Palestine Liberation Organization is merely a first step into a baffling maze toward Middle East peace.

Myriad thorny obstacles lie ahead and centuries of enmity must be surmounted. The competing claims of Arabs and Jews to the same lands are rooted in thousands of years of history.Many Israelis and Palestinians harbor the often justified fear that their foes would like them to forcibly disappear. It is a fear that makes every compromise a national trauma, seemingly a step toward national doom.

Every Israeli recalls the vow of former PLO Chairman Ahmad Shukairy to "drive Israel into the sea."

Every Palestinian recalls the contention of Israel's late Prime Minister Golda Meir that the Palestinian people "do not exist."

Both sides cherish the memories of thousands of heroes who died in the name of nationalism.

PLO chairman Yasser Arafat has now met U.S. criteria for negotiation and Washington has begun talking with PLO representatives.

But Israel's leaders, despite nearly universal acknowledgement that the PLO represents the Palestinians, still refuse to deal with what they call a "terrorist" organization.

Before any kind of peace talks can start, Israel's objections must be overcome - or a way found to sugarcoat the PLO's bitter presence.

The Reagan adminstration has favored submerging the PLO in a Jordanian delegation to talks. It has also toyed with the idea of having the PLO represented in talks by outside negotiators. The PLO rejects both plans.

Israeli leaders have suggested staging limited elections to choose Palestinian negotiators from in the occupied West Bank and Gaza Strip. But the last elections there produced a clutch of PLO supporters, most of whom were deposed or deported by Israel.

Due in part to their year-long uprising in the occupied lands, the Palestinians have won wide international acceptance of their call for a U.N.-sponsored peace conference.

But the ground rules remain a major point of contention, even beyond the invitation list.

Israel and the United States reject the Arab idea that a conference could force a solution if the parties themselves reach a deadlock.