By Elizabeth Roberts and Ray Sanchez CNN

(CNN) -- [Breaking news update, published at 7:50 p.m. ET]

The death toll from Hurricane Matthew has risen to 264, Haiti's Civil Protection Service said.

[Original story, published at 5:14 p.m. ET]

Rescue workers in Haiti warned Thursday that the death toll from Hurricane Matthew could rise as they worked to establish the full extent of the damage.

"The most concerning thing is we really don't have a full picture. We don't have all the information we need in order to see how bad it is," said Christy Delafield, a communications officer with Mercy Corps, a global humanitarian aid agency.

"This is due to a bridge being out," she said. "There have been landslides and the cell phone service has been unreliable. We need to get to the south of the country," she added.

Matthew made landfall in Haiti as a Category 4 hurricane on Tuesday. At least 108 people have been confirmed dead.

On Thursday, Civil Protection Service spokesman Joseph Edgard Celestin said the death toll does not take into account potential loss of life in the far southwest of the country, which has not been assessed due to "communication issues."

"What we are realizing, as we are going further ... is that there's been a lot of devastation and a lot of damage and probably a lot more than what we had initially assessed," said Ariane Quentier, spokeswoman for the UN Stabilization Mission in Haiti.

Infrastructure destroyed

The hurricane exacted a heavy toll on the country's infrastructure, agriculture, and houses.

An estimated 350,000 people have been affected, with 15,623 displaced people living in shelters, according to officials.

In southern Haiti, winds of 125 mph (200 kph) destroyed homes, flooded villages and cut off the island from the rest of the country.

The main road that connects the capital, Port-au-Prince, with Haiti's devastated southern peninsula, broke apart when a bridge collapsed Tuesday, severely hampering relief efforts.

On Thursday, the bridge, which spans the Rivière la Digue in Petit-Goâve, southwest of the capital, became partially passable as the waters subsided.

"Some vehicles are going over it but the number that can do that is very limited," Delafield said. "Our team are going to have a go today."

She said Port-au-Prince has not suffered such bad damage as the south, although some neighborhoods on the outskirts of the city have been flooded.

Tim Callaghan, assistance response team leader for the US Agency for International Development in Haiti, said much of the damage in Haiti's hardest hit areas -- the southwestern cities of Les Cayes and Jeremie -- appeared to be structural, and not the result of heavy rainfall.

"The urgent need we're focusing on right now is food, safe drinking water and ... things like plastic sheeting, hygiene kits and so forth," he said.

"We're in the most critical phase to support people."

Dominique Fevry-Gilliand, a spokeswoman for Oxfam Canada, said up to 80% of homes in the most affected areas were destroyed. There was also extensive damage to crops along swaths of southern Haiti.

"Most likely, in the next couple of weeks and months, one of the things they will have to deal with in that region is hunger," she said.

Fevry-Gilliand said many Haitians initially refused to evacuate, fearing that thieves would make off with their meager possessions.

"There were many who stayed," she said. "In some cases, police forced people to evacuate."

The poorest of the poor are suffering

Dave Brodsky, president of Chanje Movement, a California-based church partnership that has been working in Haiti since the 2010 earthquake, said the "poorest of the poor" have been worst hit by flooding.

"A lot of the homes here are just leveled with layers of mud and there are sanitation issues because the standing water is mixing with human waste and animal waste," he said speaking to CNN from Croix-des-Bouquets, a major suburb to the north east of the capital.

"About 10% of the homes have been destroyed in Croix-des-Bouquets and about 50% damaged. The rain and the mud caused them to collapse."

He said the homeless are seeking shelter with friends and families.

"People here really are most concerned for their relatives and friends who are in the south," he added.

Assessing situation in devastated south

Kevin Novotny, country director for Save the Children in Haiti, said the coastal cities of Jeremie and Les Cayes "have borne the brunt."

"We have an office in between the two of those. It was in a forested area. Now very few trees are standing there.

"In Les Cayes staff did a tour Wednesday, a lot of houses are destroyed there. Anything with a tin roof lost it."

Major flooding, landslides and damage to a hospital have also been reported in Les Cayes.

Worst humanitarian crisis since earthquake

UNICEF said on Wednesday that Haiti is facing the largest humanitarian event since the devastating earthquake six years ago which killed at least 220,000 and left 1.5 million people displaced and 300,000 injured. Tens of thousands were still living in makeshift homes after the earthquake when the hurricane hit.

"Identifying who needs what, that's the challenge now," Yvetot Goui, a journalist based in Port-au-Prince told CNN.

"Every minute and hour we're hearing something new, but rescuers will need to go town-to-town to judge the extent of the damage."

CNN's Deborah Bloom and Nick Thompson contributed to this report

TM & © 2016 Cable News Network, Inc., a Time Warner Company. All rights reserved.