Ten troops were killed and one was taken hostage in clashes with IS-linked jihadists in Borno state, according to two security sources.
Clashes erupted on Monday when a team of soldiers stormed a camp of Islamic State West Africa Province (ISWAP) in Alagarno village in Damboa district.
“We lost 10 soldiers in the fight and one was taken by the terrorists,” a security source told AFP on Tuesday.
The hostage was seized while fleeing to safety after the troops were outgunned, he said.
“It was an intense battle and the terrorists also suffered casualties but they were able to overwhelm the soldiers,” said a second security source who confirmed the toll.
The insurgents seized four vehicles, including a truck and an armoured vehicle, the second source said.
Both sources asked not to be identified.
Alagarno, which lies 150 kilometres (90 miles) from regional capital Maiduguri, is a stronghold of ISWAP, which split from the Boko Haram jihadist group in 2016 and rose to become a dominant force.
ISWAP has increasingly been attacking civilians, killing and abducting people on highways as well as raiding villages for food supplies.
On Tuesday, ISWAP said its fighters killed seven soldiers while repelling an attack in Alagarno forest.
“Clashes took place with a variety of weapons, which led to the killing of seven elements and taking an eighth prisoner,” the group said in a statement, according to SITE Intelligence, which monitors jihadist activities worldwide.
The insurgents seized “an array of weapons” and vehicles, the statement said.
ISWAP claimed it killed four Nigerian troops on the same day in a separate attack near the town of Gamboru close to the border with Cameroon.
AFP could not independently verify the claim.
At least 36,000 people have been killed and around two million displaced from their homes since the start of the conflict 11 years ago.
The violence has spread to neighbouring Chad, Niger and Cameroon, prompting a regional military coalition to fight the militant groups.