Nine of the 11 members of a National Endowment for the Arts literary panel have resigned to protest an alleged attempt by Congress to restrict freedom of artistic expression in the endowment's new $174 million budget, the dissenters said Tuesday.

In a letter informing NEA Chairman John E. Frohnmayer of their resignations on Monday, the panelists denounced restrictions that Congress imposed on the endowment's grantmaking authority in the fiscal 1991 budget year, which began Oct. 1."We question the constitutionality of the language," they wrote. "We deplore the implied threat of censure and attempt to constrict freedom of artistic expression and the rights granted by the First Amendment."

They protested a congressional requirement that the NEA chairman "shall ensure" that grants are made "taking into consideration general standards of decency and respect for the diverse beliefs and values of the American public."

Those curbs, approved shortly before Congress adjourned Oct. 28, replaced a much stricter obscenity ban that covered endowment grants last year. This year, Congress said only that obscenity "shall not be funded" and voted to penalize grant recipients whose works are ruled obscene by the courts.

Even though the new curbs are milder than the former ban, they nevertheless set "a very scary precedent," said Jennifer Moyer, former chairwoman of the NEA's literary publishing panel. "If there are any restrictions on freedom of expression, it's unacceptable."

Frohnmayer issued a statement calling the protest resignations "premature" and "unfortunate."

He said the endowment is examining the new law to determine how it can be implemented in good faith without impeding artistic creativity. "We are certain that reasonable people can arrive at the answer to this question," he said.

Moyer, a poet and literary publisher from Mt. Kisco, N.Y., resigned along with eight other panelists representing writers, editors, publishers, booksellers and arts administrators.

The 11-member panel judged grant applications last year from literary magazines, small, independent presses and non-profit organizations that distribute contemporary creative writing.

Technically, the 1989-90 literary publishing panel was dissolved when it completed its work last December, said endowment spokesman Josh Dare.

He said a new panel, including one holdover, Beverly Jarrett, editor in chief of the University of Missouri Press in Columbia, Mo., has been formed and will meet Nov. 26 to review grant applications for the current fiscal year.

None of the nine who resigned from last year's panel will appear on the new panel. Moyer said she was invited to serve again this year, but refused.

On the basis of the panel's recommendations, Frohnmayer approved 99 literary publishing grants totaling $1.4 million last year.

In a separate letter informing the chairman of her resignation, panelist Helaine Harris, a bookseller from Hyattsville, Md., asked how the endowment can distinguish between "decent or indecent art" under the new law.

"I feel that I can be a judge of artistic merit, but I don't feel that I should pass judgment on decency," she wrote. "I refuse to be a part of this self-censorship process."

The other panelists who resigned were Michael Anania, a poet, fiction writer and teacher from Chicago; Denise Chavez, a Houston writer; Katharine Harer of Berkeley, Calif., director of a distributing company for small literary presses; Brooks Haxton, a poet and teacher from White Plains, N.Y.; poet Susan Howe of Guilford, Conn.; Peter Meinke, a writer and director of a literary workshop at Eckard College in St. Petersburg, Fla., and E. Ethelbert Miller, a poet and teacher at Howard University in Washington, D.C.

The two panelists who did not resign were Anne Bourget of the California Arts Council in Sacramento and Jarrett, who said she refused to resign because it "seemed like throwing the baby out with the bath water" at a time when the NEA's financial support is crucial to the survival of the arts.

"Resigning didn't seem to accomplish anything except getting your name in the paper and making a bit of a splash," she said.

Bourget could not be reached for comment.