After handing back billions of dollars in surplus money to taxpayers last year, the governors of New York, California and Massachusetts are suddenly confronted with major revenue shortfalls that are forcing them to consider painful spending cuts.

While those governors and legislative leaders blame each other for the miscalculations, budget officials say the culprit is the 1986 federal tax-revision law, specifically the provision dealing with capital gains.The deficits come at a time when all three states are enjoying healthy economies, producing heated budget debates that have been the more unpleasant for being unexpected.

In New York, Democratic Gov. Mario M. Cuomo announced last week that this year's deficit would reach $900 million and Friday recommended a state hiring freeze.

California Gov. George Deukmejian has projected a $1 billion shortfall this year and an additional $1 billion next year. In a news conference Friday, he withdrew his proposal for increased personal and business tax increases and left it to the Legislature to devise a solution.

Massachusetts Gov. Michael S. Dukakis, who is basing his Democratic presidential campaign on his record as a strong fiscal manager, has been embarrassed by new estimates showing revenues off by $252 million. His aides revised that estimate last week to more than $300 million.

The problem, state budget directors say, began in 1986, the last year in which taxpayers could take advantage of more favorable capital gains treatment under old tax laws. As the deadline approached, many corporations and taxpayers sold property, stocks and other assets to get in under the wire.

The result was a surge in financial transactions that brought unexpected windfalls to the three high-income states.

New York budget director Dall W. Forsythe said the value of capital gains transactions in the state surpassed the $23 billion estimate by $9 billion as a result of the frenetic selling activity in 1986. In 1985, the value of such transactions was $16 billion.

Massachusetts also found itself "awash in new revenue because of capital gains," said John Hanson, spokesman for the state finance department. "We were rolling in dough."

California, flush with surpluses, paid $1.1 billion in income tax rebates last year under an initiative that sets spending limits and requires refunds if a surplus remains when the limits are reached. New York lowered maximum tax rates and increased personal deductions at a cost of $1.1 billion this year. Massachusetts adopted $500 million worth of tax cuts over a five-year period.

Then came the miscalculations.

"Because of the recent tax cuts, California, New York and Massachusetts were under political pressure to adopt more optimistic estimates to make things balance out," said Steven Gold, director of fiscal studies at the National Conference of State Legislatures. "They reached a political agreement on what to assume, and that had a lot to do with how much they were going to cut tax rates."

"We knew capital gains would definitely be down in 1987 because of the surge in sell-offs in 1986, but we didn't take it down far enough," said Lois Wallace, assistant director of California's finance department. The error was magnified after the state wiped out capital-gains deductions in its own tax laws. "The new (ax) bill doubled the problem," Wallace said.

Massachusetts started to see red ink shortly after April 15, Hanson said. "We noticed that the trend of capital gains was even lower than we had anticipated."

The Massachusetts Legislature, over Dukakis' objections, is moving to boost taxes on alcohol, cigarettes and sports and theater tickets by $100 million. Dukakis had hoped to offset the shortfall with spending cuts and has threatened to veto the new tax measures.

New York lawmakers, reluctant to raise taxes or cut spending in an election year, have chosen to question the accuracy of the deficit estimates. Cuomo is trying to protect a $650 million surplus fund earmarked for housing for low-income people and repair of roads and bridges, but Republican lawmakers want to hold that money back to cover the budget gap. Cuomo, determined not to tamper with the recent tax cuts, is pushing for other spending cuts instead.

The budget proposal Deukmejian withdrew Friday contained $470 million in spending cuts and would have dipped into a $1 billion emergency reserve to plug the budget gap.