Attorney General Edwin Meese III faces new problems rebuilding a Justice Department rocked by resignations now that his second choice for the No. 2 post has told him to find somebody else.

Meese is looking for a new nominee after former American Bar Association President John Shepherd told the attorney general's aides on Monday to begin searching for a new candidate, Justice Department sources say.Meese, asked Tuesday whether Shepherd had withdrawn, said, "Not yet. I expect we will be hearing today or tomorrow." Meese declined comment when asked whether he was thinking of other names to suggest.

The attorney general was questioned as he left a memorial service for law enforcement officers slain while enforcing drug laws.

While department sources said Shepherd had not flatly withdrawn, they said he already had informed the department that he probably would do so. The sources, speaking only on condition of anonymity, said Shepherd asked that Meese's aides begin looking for other candidates.

Meese's problems in filling the post of deputy attorney general, as well as the position of assistant attorney general in charge of the criminal division, stem from the abrupt resignations March 30 of Arnold Burns and criminal division chief William Weld.

They quit out of concern that the nearly year-old criminal investigation of Meese is hurting Justice Department operations, morale and image.

A number of current and former U.S. attorneys have been approached about taking Weld's post, but all have turned it down, department sources have said.

Meese, who testified before a federal grand jury for four hours in the probe Monday, has refused to resign, predicting that he won't be indicted.

In an attempt to recover quickly from the resignations, Meese hurriedly announced the selection of Shepherd for the No. 2 position two weeks ago without customary White House clearance and FBI background checks. President Reagan still has not nominated Shepherd.

Shepherd, 62, a St. Louis attorney, told Meese last Friday that he was giving serious consideration to withdrawing because of stress on his family stemming from his selection.

A former bookkeeper from Shepherd's law firm testified shortly before Meese recommended him for the post that she had an affair with Shepherd. The bookkeeper, who was convicted of embezzlement, claimed Shepherd had told her to write checks to herself. Shepherd denied that he had an affair with her or that he had authorized her to write checks.

The trial testimony and Shepherd's membership in an all-white country club and an all-male social club brought increasing public scrutiny of Shepherd after Meese announced his selection April 5.

Meese's first choice to fill the post of deputy attorney general was former federal judge Arlin Adams, now a Philadelphia lawyer. Adams said he was unable to take the post because he had a heavy caseload and had gone to work at the law firm just last year. Adams said Meese's legal problems were not a major factor in his decision not to take the Justice Department position.