Opposition supporters accused Mr Sargsyan of clinging to power after he was appointed prime minister last Tuesday, soon after finishing two five-year terms as president.
“The street movement is against my tenure. I am fulfilling your demand,” he said in a statement.
Former prime minister Karen Karapetyan takes over as acting PM, reports said.
The announcement came soon after opposition leader Nikol Pashinyan was released from detention. Mr Pashinyan had been arrested on Sunday after televised talks with Mr Sargsyan collapsed.
As well as Mr Pashinyan, two other opposition politicians and some 200 demonstrators were held.
In his statement published on his website, Mr Sargsyan said he was “addressing all citizens of the Republic of Armenia… for the last time as leader of the country.”
“Nikol Pashinyan was right. I was wrong,” he said. “The situation has several solutions, but I will not take any of them… I am leaving office of the country’s leader, of prime minister.”
His spokesman, Hovhannes Nikoghosyan, told the BBC that Mr Sargsyan was behaving responsibly and fulfilling the demands of the street movement which opposed his appointment as prime minister.
“I think his resignation is a clear demonstration of a democracy in force. It’s not that every demonstration in every corner of the world leads to the resignation of the authorities,” Mr Nikoghosyan said.
Protesters chanted “Nikol, Nikol” in the streets on Monday and waved Armenian flags. They were joined by hundreds of uniformed soldiers, despite warnings from the defence ministry that any soldiers protesting would be harshly punished.
Mr Pashinyan congratulated the people on their “victory” following the resignation. “You have won, proud citizens of the Republic of Armenia. And no one can seize this victory from you. I congratulate you, victorious people,” he wrote on Facebook.
Mr Sargsyan had faced criticism in Armenia over his close ties to the Russian President, Vladimir Putin, who has also moved between roles as president and prime minister to maintain his grip on power.
Mr Putin’s spokesman said on Monday that Moscow was closely watching events in Yerevan. “We are very attentively observing what is happening in Armenia,” Dmitry Peskov told journalists, calling the country “extremely important” for Russia.
Asked if Russia would intervene, Mr Peskov said the matter was “exclusively an internal affair” and Russian action would be “absolutely inappropriate”.
At the scene
Rayhan Demytrie in Yerevan
Jubilation filled the Republic Square in Yerevan as people cracked open bottles of champagne, dancing in the fountain, hugging and waving flags.
“This is what victory feels like,” said Lena, wrapped in the Armenian tricolour. “This is a day to live for, this is the day to be here to witness history!” said her husband Armen.
When Mr Pashinyan arrived at Republic Square after being released, the people chanted his name and “Victory!”.
Shortly after, news broke that Mr Sargsyan had resigned and it was as if a firework of joy had exploded in the square. Many here say it is a victory for the whole nation, that the people stood up for democracy and won. And it came on the eve of April 24, which Armenians worldwide mark as Remembrance Day for the victims of mass killings of Armenians at the turn of the 20th century in Ottoman Turkey.
The events in Armenia are significant, because they demonstrate that in a post-Soviet country change is possible through a peaceful, organic, grassroots movement. The fact that the Armenian authorities showed restraint and did not use excessive force against the demonstrators is also an achievement.
But it’s important to remember that the government in Armenia remains the same, and the new acting prime minister is an old ally of Mr Sargsyan. Only the leadership has changed.
Why were there protests?
In 2015, Armenians voted in a referendum to shift the country from a presidential to a parliamentary system, stripping powers from the president and giving them to the prime minister.
The vote was marred by allegations of ballot rigging and claims Mr Sargsyan wanted to simply switch office after his presidency ended.
The ex-president had formally stated he would “not aspire” to the prime ministerial position, but on Tuesday last week the country’s parliament officially confirmed Mr Sargsyan in the post.
Protesters poured into the streets in the days beforehand to try to stop the parliament from passing the measure, and clashed with police. On the day of the confirmation, Mr Pashinyan said the demonstrations constituted a “non-violent velvet revolution”.
Who is Serzh Sargsyan?
Mr Sargsyan served two consecutive terms as president of Armenia, starting in 2008 and ending on 9 April this year. His initial election in 2008 prompted deadly clashes between the state and opposition supporters. At least eight people died.
He won a second five-year term in 2013. Several of his opponents dropped out of the race and one candidate was shot in a suspected assassination attempt.
Mr Sargsyan was also accused of failing to address continuing tensions with Azerbaijan and Turkey, as well as widespread poverty at home.