Between 3,923 and 6,102 non-combatants were killed in the two countries, said Airwars, a London-based journalist collective that compiles data from public sources.
That is sharply up from its estimate of the previous year's toll of between 1,243 and 1,904.
backed Iraqi forces in a gruelling battle last year to oust IS from second city Mosul
, as well as supporting a Kurdish-dominated force that seized Syria's Raqqa
city from the group after months of fighting.
Airwars said the surge in deaths was likely partly caused by intense fighting in urban areas.
But it also said the administration of US President Donald Trump may be partly to blame.
"This toll coincided with the start of the Trump presidency, which has declared the conflict against ISIS to be a 'war of annihilation'," it said, using an alternative name for IS.
"The scaling back by the new administration of some measures aimed at protecting civilians" may have contributed to the "steep increase" in casualties, it said.
Airwars said the coalition had carried out some 11,573 artillery and air strikes, some 50 percent up on the previous year. More than 70 percent were in Syria.
It said 766 attacks last year wounded some 2,443 civilians in both countries.
The monitor, which also tracks casualties of Russian operations in Syria, said coalition-linked civilian deaths in 2017 had "far" outnumbered
those attributed to Russia.
It said the toll for civilians killed in coalition strikes in Syria had quadrupled compared to 2016, with the battle for Raqqa killing at least 1,450 civilians, while in Iraq deaths were up by 87 percent.
Between the start of the battle for Mosul in October 2016 and the announcement of its "liberation" in July, between 1,066 and 1,579 civilians were killed by coalition raids there, Airwars said.