The federal judge presiding over the case of reputed former Boston mob boss James “Whitey” Bulger has been removed by the First Circuit Court of Appeals in Boston, after the defense raised concerns about his capacity for impartiality.

The judge, Richard Stearns, had previously worked as a prosecutor in the criminal division of the U.S. Attorney’s Office in Boston when Bulger is alleged to have reigned over organized crime in the city, the First Circuit Court of Appeals in Boston noted.

“Despite our respect for Judge Stearns and our belief in his sincerity, we are nonetheless bound to conclude that it is clear that a reasonable person might question the judge’s ability to preserve impartiality through the course of this prosecution,” the appeals court ruled.

Bulger, the alleged former head of Boston’s notorious Winter Hill gang, faces 19 murder charges, as well as charges including extortion, money-laundering and narcotics distribution.

The defense had initially moved to dismiss the case, saying Bulger was granted immunity by federal agents working to infiltrate Irish and Italian mobs in Boston three decades ago. But Stearns ruled this month that Bulger did not have immunity to commit murder or other crimes after his purported deal with the FBI.

— CNN’s Mark Norman, Jason Kessler, Melissa Gray and Jason Hanna contributed to this report.