The nation's seafood inspection system needs improvement, a scientific panel said, urging closer scrutiny to prevent harvest of tainted fish and to head off danger during handling and processing of seafood.

Americans consume an average of about 16 pounds of seafood per year but only a fraction of it is inspected under the voluntary inspection program now in place. Consumer groups are expected this year to renew a push for a comprehensive, mandatory federal program."The present federal monitoring and inspection system is too limited in frequency and direction to ensure enhanced safety of seafoods," the panel of 13 scientists said in their report released this week.

They emphasized most seafood sold in the United States is "wholesome and unlikely to cause illness."

The greatest risk comes from consumption of raw oysters, clams, mussels and other mollusks that have been contaminated by disease-causing organisms. Shellfish accounted for two-thirds of the 5,980 cases of seafood-related illnesses reported in a 10-year period ended in 1987.

At the request of a Commerce Department agency, the Institute of Medicine, a part of the National Academy of Sciences, selected a 13-member panel of experts to assess seafood safety issues.

Seafood is the only so-called flesh food not subject to mandatory inspection.