By Manu Raju, Dan Merica and Julia Horowitz

(CNN) -- Andrew Puzder has withdrawn as President Donald Trump's choice for labor secretary.

Puzder, the CEO of the company that owns the Hardee's and Carl's Jr. fast food chains, faced fierce opposition mostly from Democrats in part related to his position on labor issues as well as the fact that he employed an undocumented immigrant housekeeper.

"After careful consideration and discussions with my family, I am withdrawing my nomination for Secretary of Labor," Puzder said in a statement Wednesday afternoon. "I am honored to have been considered by President Donald Trump to lead the Department of Labor and put America's workers and businesses back on a path to sustainable prosperity.

"I also thank my family and my many supporters -- employees, businesses, friends and people who have voiced their praise and hopeful optimism for the policies and new thinking I would have brought to America as Secretary of Labor. While I won't be serving in the administration, I fully support the President and his highly qualified team."

The decision came as Senate Republicans told the White House he was losing support, a senior GOP source said, adding there were four firm Republican no votes and possibly up to 12.

His team subsequently began to pull back from all their plans regarding the confirmation process. Puzder was scheduled to do extensive preparation on Wednesday afternoon for his confirmation hearing before the Senate on Thursday, but canceled the so-called "murder board" after top Republicans spoke to the White House, a source close to Puzder told CNN.

Around the same time, sources on Capitol Hill were told Puzder was expected to cancel his confirmation hearing. And Puzder's team told the White House that he did not want to go forward with the nomination if there was no clear path forward.

White House press secretary Sean Spicer confirmed Puzder's decision, telling reporters simply, "he withdrew."

Senate Democratic Leader Chuck Schumer fanned the flames behind the opposition to Puzder in a statement Wednesday afternoon.

"No matter how you cut it, there is no worse pick for labor secretary than Andrew Puzder, and I'm encouraged my Republican colleagues are starting to agree," the New York Democrat said. "He does not belong anywhere near the Labor Department, let alone at the head of it. Puzder's disdain for the American worker, the very people he would be responsible for protecting, is second to none."

Fingerpointing among Republicans

The fingerpointing has already begun among Republicans.

GOP sources are privately pointing the finger at business groups for not giving political cover to Republicans. They say that they were swamped by labor groups and progressive forces looking to derail Puzder, while the business community sat on the sidelines.

A source said the unions have eaten "their lunch."

"They literally were raising money in like $5,000 increments when millions were being spent against them," one top GOP source says. "His entire support network made one TV ad that looked like a welcome video for new Hardee's employees, which referred to "her" and "she" while praising the boss who hired them. Everyone knew he was in trouble weeks ago and even his strongest supporters and biggest beneficiaries of his point of view as labor nominee would not open their wallets."

Ran into problems immediately

Puzde faced intense criticism since his appointment on December 8. Current and former employees spoke out against him from the start, and Democrats and labor activists attacked his record on worker rights.

The fast food executive has a long history of fighting against government regulation, a $15 minimum wage and the Affordable Care Act.

Decades ago, Puzder's ex-wife, Lisa Fierstein, levied charges of domestic abuse against him, which she later dropped. She also appeared in disguise on Oprah Winfrey's TV show to talk about domestic violence in 1990. In advance of Puzder's scheduled hearing, senators privately reviewed the tape.

During the episode, Fierstein said Puzder told her, "I will see you in the gutter. This will never be over. You will pay for this," according to Politico.

Senators were shown the video in the lead up to Puzder's nomination fight.

In a letter to senators obtained by CNN, Fierstein said she regretted accusing Puzder of abuse during a rocky marriage.

Puzder has denied he abused his wife.

That hasn't been the only issue. Last week, he admitted to having employed an undocumented immigrant as a housekeeper. He later said that he and his wife let the housekeeper go when they learned of her immigration status, and he said they paid back taxes on the employee and offered her help getting legal status.

Puzder's own employees were some of the first to cry foul when he was nominated.

Last month, current and former workers at Hardee's and Carl's Jr. announced that they had filed 33 state and federal complaints against their employers. Charges include wage theft, manipulated overtime, sexual harassment and unfair labor practices.

All the complaints pertained to franchised locations that are independently operated. But the workers said they reflected poorly on a chief executive who hoped to lead the Labor Department. A spokesman for CKE Restaurants said the company does not comment on pending litigation.

Puzder once called some of his Hardee's employees "the worst of the worst," according to reporting by CNN's KFile. He was also criticized for past statements in support of replacing workers with robots.

"They're always polite, they always upsell, they never take a vacation, they never show up late, there's never a slip-and-fall, or an age, sex, or race discrimination case," Puzder told Business Insider in March 2016.

It took weeks for senators to get Puzzler's financial disclosure information. He had promised to step down as CEO of the company, CKE Restaurants, if confirmed, and to divest his stake in 200 companies and funds.
   
CNN's Jim Acosta contributed to this report.
   
The-CNN-Wire™ & © 2017 Cable News Network, Inc., a Time Warner Company. All rights reserved.