The killings were in apparent revenge for attacks by Islamic State (IS).
Amnesty said the militias had been supported and armed by the Iraqi government and operated with impunity.
Prime Minister Haider al-Abadi, who took office last month, has admitted to previous "excesses" by security forces and vowed to govern for all Iraqis.
He has not yet commented directly on allegations contained in the Amnesty report but has previously said Iraq faces an "existential" battle against militants from Islamic State, also known as Isis or Isil.
Mr Abadi has also acknowledged, in what is believed to be a reference to Sunnis, that his government must address the "legitimate grievances" of the Iraqi people.
The Amnesty report, based on interviews conducted in Iraq in August and September, provides details of what it says were sectarian attacks carried out by militiamen in the cities of Baghdad, Samarra and Kirkuk.
It says scores of unidentified bodies have been found, many still handcuffed and with gunshot wounds to the head, suggesting execution-style killings. Many others who disappeared remain unaccounted for.
Amnesty says that in Samarra, a mainly Sunni city north of Baghdad, it obtained details of more than 170 Sunni men abducted since June.
More than 30 were taken from or near their homes in a single day - 6 June - shot dead and their bodies dumped nearby.
"The killing spree seems to have been in reprisal for a brief incursion into the city the previous day" by IS fighters, Amnesty says.
Amnesty says the militias - including Asa'ib Ahl al-Haq, the Badr Brigades, the Mahdi Army and Kata'ib Hizbullah - have become more powerful since June, when the Iraqi army fell into disarray in the face of IS advances.
Correspondents say much of the fighting against IS since then has been carried out by militias, who were able to recruit thousands of volunteers, rather than the army.
Many Shia men volunteered to fight IS; there is no suggestion any of those pictured in this article have been involved in killing civilians
Shia militias have carried out much of the fighting against IS after the Iraqi army's retreat
There are now "tens of thousands" of militiamen, who "wear military uniforms but operate outside any legal framework and without any official oversight", Amnesty says.
The report quotes an unidentified Iraqi government official as saying that militias "mostly... kidnap Sunnis, because the victims can easily be labelled as terrorists and nobody is going to do anything about it".
Another unnamed government official said some Sunni men were considered to be "terrorists or terrorist supporters" because of where they lived. Others were killed "in blind revenge".
"I'm afraid that we're regressing back to the situation as it was seven or eight years ago, when this behaviour was very widespread," he said.
Militiamen have also tried to extort ransoms, sometimes killing their captives even after payments have been made, Amnesty said.
"I begged friends and acquaintances to lend me the ransom money to save my son but after I paid they killed him and now I have no way to pay back the money I borrowed, as my son was the only one working in the family," one mother said.
Amnesty says the militias have taken advantage of an "atmosphere of lawlessness" but the Iraqi government, which has armed and supported them, bears responsibility for their actions.
"By granting its blessing to militias who routinely commit such abhorrent abuses, the Iraqi government is sanctioning war crimes and fuelling a dangerous cycle of sectarian violence," said Amnesty's senior crisis response adviser, Donatella Rovera.
"The new Iraqi government... must act now to rein in the militias and establish the rule of law."
Iraqi Shia militias accused of murder spree
Amnesty International says sectarian groups have abducted and killed scores of Sunnis during war against ISIL.
Last updated: 14 Oct 2014 02:54
Shia fighters allegedly have carried out abductions and killings in retribution for crimes committed by ISIL [AFP]
Shia militias have abducted and murdered scores of Sunni civilians in Iraq in crimes committed in retribution against the actions of ISIL, according to a new report by Amnesty International.
The London-based rights group on Tuesday published what it said was evidence that Shia militias abducted civilians in Baghdad, Samarra and Kirkuk, and killed them even if families paid tens of thousands of dollars in ransom.
The Amnesty report, Absolute Impunity: Militia Rule in Iraq, said scores of unidentified bodies had been discovered handcuffed and with gunshot wounds, indicating a pattern of deliberate killings.
The group called on the Iraqi government, which has armed and encouraged militias including the Badr brigades and the Mehdi army, to fight ISIL, to hold them to account.
Militias operate outside any legal framework and without official oversight, and had contributed to a deterioration in security and to the increasing lawlessness in Iraq, Amnesty said.
"Shia militias are ruthlessly targeting Sunni civilians on a sectarian basis under the guise of fighting terrorism, in an apparent bid to punish Sunnis for the rise of ISIL and for its heinous crimes," Donatella Rovera, Amnesty's senior crisis response adviser, said.
"By failing to hold militias accountable for war crimes and other gross human rights abuses the Iraqi authorities have effectively granted them free rein to go on the rampage against Sunnis. The new Iraqi government of prime minister Haider al-Abbadi must act now to rein in the militias and establish the rule of law."
The Amnesty document included evidence from relatives of those who had gone missing or were killed.
It reported that one family had paid $60,000 to have a family member released, only to find his body two weeks later in a Baghdad morge, his head crushed and his hands cuffed.
Amnesty also accused Iraqi government forces of serious human rights violations, presenting what it said was evidence of torture and ill-treatment of prisoners, and deaths in custody of Sunni men held under the 2005 anti-terrorism law.
It cited one example of a 33-year-old lawyer who died in custody, his body showing open wounds and burns consistent with the application of electric shocks.
Another man was held for five months and tortured with electric shocks and threatened with rape before being released without charge.
"Successive Iraqi governments have displayed a callous disregard for fundamental human rights principles," Rovera said.
"The new government must now change course and put in place effective mechanisms to investigate abuses by Shi’a militias and Iraqi forces and hold accountable those responsible."