A calculation error by Reagan administration budget experts cannot be rectified by law and is preventing Congress from enacting $1.2 billion in additional spending programs for 1989.

The over-calculation of the size of next year's deficit was pointed out to congressional aides in the past few days by the agency that committed it - the White House's Office of Management and Budget.OMB officials confirmed this past week that the mistake had been made, and said they had told congressional staff members about it.

The revelation came as lawmakers left town for a summer recess with plans to return in September and complete their work for the year by considering a batch of competing spending bills.

As the $1.1 trillion federal budget goes, $1.2 billion normally would barely register a tick.

But the amount is significant because it means that Congress will have to squeeze spending for anti-drug programs, the homeless, the hungry and other popular causes under a $1.8 billion ceiling, rather than what should be a $3 billion limit.

OMB's error came in estimating how close spending programs enacted so far - along with bills that still must be passed to keep government agencies functioning - come to the $146 billion deficit ceiling for next year that is set in the Gramm-Rudman balanced budget law.

OMB's official estimate is that Congress still can spend $1.8 billion before the deficit ceiling is reached, even though the agency now concedes the figure should be $3 billion.

The bills lawmakers would like to pass - or that President Reagan has not yet signed into law - would exceed $1.8 billion in spending.

OMB officials, questioned about the error, said it was an honest mistake.

Asked how it was made, OMB spokeswoman Barbara Clay said, "We don't know."

But the blunder could cause friction between lawmakers trying to choose among what they see as competing spending necessities, and administration officials arguing that the federal deficit should be kept as low as possible.

Rep. Leon Panetta, D-Calif., expected to become chairman of the House Budget Committee next year, said that if the spending pressures are too great, Congress may have to consider legislation that would free up the $1.2 billion.

"If it's an error and does not reflect in fact where we stand, it's an option that should be considered," he said.

But in a prepared statement, administration budget director James C. Miller III said he would "oppose any tinkering" with the law "for purposes that would increase the deficit."

Under the Gramm-Rudman deficit reduction law, OMB analyzes the economy and federal spending and revenues several times annually, projecting how much red ink is likely in the coming fiscal year.

If the final numbers show that the deficit will exceed a target set by the law - it is $146 billion for next year - spending is automatically cut by at least $10 billion. That is a scenario Congress and President Reagan have desperately sought to avoid.