The decision by Maxicare Health Plans Inc., the giant California-based health maintenance organization, to file for protection under federal bankruptcy laws is typical of a shakeout affecting the entire HMO industry, analysts say.

HMOs, which expanded rapidly during the early 1980s, have fallen on hard times. Two-thirds lost money in 1986, and their financial picture has improved only modestly during the past two years. For the first time in more than a decade, the number of HMO plans decreased in 1988, from 662 to 643.Maxicare's experience was typical.

Saddled by debt, the company lost $250 million in the first nine months of 1988. In an effort to stay solvent, it began selling off several of its individual health plans, which provide health coverage to 1 million enrollees in a dozen states. And it is looking for a buyer for its California plan, the core of the organization.

Part of the leveling off in HMO growth was expected, as inexperienced, poorly run or underfunded plans close their doors, merge or are bought out by stronger plans.

And part of the slowdown is due to the financial woes which hit the health insurance industry in general over the past two years, with many firms recording huge losses.

But HMOs may also be a victim of their own success.

In a period of tumultuous change in the health care industry, the ideas on which HMOs are based - emphasis on preventive medicine rather than high-tech intervention to cure illness, and careful management of finances and services to patients - are being adopted industry-wide by health insurers, where "managed care" has become the watchword.

HMOs are, in effect, competing against their own best ideas, and in many cases against competitors with far better financial backing.

"The growth of the pure HMO clearly has slowed, but the ideas that flowed from HMOs are rapidly becoming universal," says Dr. Paul El-wood of InterStudy Inc., a Minnesota health care think tank.

"The HMO was the lead dog in effecting almost total change in the structure of incentives in the American health care system. That process is still under way," says Elwood, who is widely regarded as the father of the modern HMO movement.

While the biggest and strongest HMOs will continue to thrive, the prepaid plans of the future will look quite different from the HMOs of the past, analysts say.

In the traditional HMO, the health care purchasers, usually employers, pay a set fee to the HMO to provide a package of health care services to employees. The employees and their families must use the doctors and hospitals employed by the HMO.

Future plans will give employees flexibility to chose other doctors and hospitals, but at an added cost.

That could eliminate some of the confusion when employers offer two, three or more different options to employees, and give employees more choice in picking their own doctor - one of the chief criticisms of HMOs.

But it would leave in place the critical financial incentive to hold down costs by carefully monitoring and controlling the amount of medical services used by enrollees.

"Before we had HMOs, there was no such thing as preference shown to one doctor or one hospital by the payer," Elwood said.

Meanwhile, the traditional fee-for-service health insurance - in which employers pay a premium to the health insurer and employees go anywhere they chose for care and send the bills to the insurance company - is decreasing. Many large insurance companies are dropping their so-called indemnity plans.

"We have probably seen in last couple of years a greater number of insurance companies exiting the group health side of the business than in recent memory. They were losing money," says Allan Fine, a senior associate at Witt Associates Inc. in Oak Brook, Ill.

Others had to institute large premium increases in the past year to cover losses of the last three years, Fine says. "In the last couple of years, the industry has sustained tremendous losses and has had to compensate by large rate increases," he said.

HMOs are, in effect, small insurance companies and they have suffered many of the same problems that have beset the health insurance industry in general, Fine says.

Unlike bigger insurance companies, which have large cash reserves and revenues from other businesses such as property and life insurance, HMOs depend almost solely on enrollee premiums and thus have shallower pockets.

"Hospitals and HMOs don't have the diversity to cover losses in medical insurance," Fine says.

Fine expects a lot more failures, mergers and buy-outs before the HMO industry stabilizes. "There is no reason to believe that has plateaued," he says.

But the picture is not totally gloomy, argues Marsha Gold, research director at the HMO trade association, Group Health Association of America in Washington.

"After a period of considerable financial strain, things are improving," Gold says.

A survey late last year indicated that 57 percent of HMOs expect to make money this year, although Gold said the sample was weighted toward larger, older and thus more stable plans.

"I think there is still going to be low profitability and more consolidation. But I think that will strengthen the industry overall," Gold says.

As the industry consolidates from its growth of the early 1980s - the number of HMOs more than tripled from 203 in 1978 to 662 in 1987 - the plans appear to be neither as good nor as bad as their supporters and critics had argued a decade ago.

"Most evaluations of HMOs indicate that the care is just as good as in conventional plans, if not better," says John Gabel of the policy development and research division of the Health Insurance Association of America.

But, says InterStudy's Elwood, "HMOs have not succeeded in putting a cap on questionable procedures. They've been a little better than the system at large, but everybody is having trouble figuring out what is appropriate."

"We get contradictory messages," Gabel said. "It looks like employers are more satisfied with HMO costs than with other plans, but the (patient) dissatisfaction level is higher than for other plans."

Mixed messages aside, the idea behind HMOs appears ready for a long run.

"The general notion of having organizations compete on price to deliver medical care, and having employers and employees increasingly selective about where they go or where they send people is alive and healthy," says Elwood.