The leader of Israel’s Blue and White party, Benny Gantz, has said he cannot form a coalition government, making a third election in a year more likely.
Mr Gantz was given a mandate by the president last month after Prime Minister Benjamin Netanyahu also failed to secure a majority in parliament.
Lawmakers now have 21 days to nominate a candidate with majority support.
Blue and White and Mr Netanyahu’s Likud party emerged from September’s election with almost the same number of seats.
President Reuven Rivlin called on the two leaders to form a national unity government, but they could not agree a power-sharing deal.
The talks broke down over who would serve as prime minister first, Mr Netanyahu’s insistence that ultra-Orthodox parties allied to him be included in the government, and Mr Gantz’s refusal to serve under a prime minister facing criminal charges.
Israel’s attorney-general is expected to announce within days whether he intends to charge Mr Netanyahu with bribery, fraud and breach of trust in three cases.
The prime minister denies any wrongdoing, but opponents accused him during the election of wanting parliament to grant him immunity from prosecution.
After handing back his mandate on Wednesday, Mr Gantz told a news conference in Tel Aviv: “In the past 28 days no stone was left unturned while we tried to form a government that would bring Israel a leadership of dignity, morals and values, a leadership that has been forgotten.”
“The [right-wing] bloc stood firm, insisting to only see the best interest of one person, before that of the patients lying in hospital corridors,” he added.
Mr Gantz accused the prime minister of making a “dangerous move” by stopping the party that won the most seats from forming a government, and said there remained 21 days “to achieve a democratic solution”.
In a video message, Mr Netanyahu accused his rival of being willing to form a minority government “with terror supporters who receive their instructions from Israel’s enemies” – an apparent reference to Israeli Arab lawmakers from the Joint List, who the prime minister accused of trying to “destroy the country” on Sunday.
“You got one thing straight: Israel needs a unity government,” he added. “Everyone can see what we are facing – we are confronting Iran on different fronts and so for Israel’s security, for the will of the people, for reconciliation among the people, we need to form a unity government.”
Earlier on Wednesday, the effective kingmaker, Avigdor Lieberman, blamed both men for the deadlock and said his secular nationalist Yisrael Beitenu party would back neither of them.
Mr Gantz “was not prepared to accept the president’s plan” for a national unity government, while Mr Netanyahu “was not prepared to separate from his ultra-Orthodox messianic bloc”, he added.
Sixty-one seats are needed for a majority in the 120-seat parliament, or Knesset.
Without the support of Yisrael Beitenu’s eight lawmakers, Mr Netanyahu’s bloc of right-wing and religious parties controls 55 seats.
Mr Gantz’s centre-left bloc has 44 seats without Yisrael Beitenu, and would require the support of at least nine members of the Joint list to get a majority.
Source: BBC