Inlining the OP pics


If that battery is at 4v, you likely have damaged it.
LiPo's do not like going down lower that 3.0v per cell, and your 2S battery is at 2.0v per cell.
Maybe it will be usable? Maybe your charger will still charge it?
But it will likely have a reduced capacity in the future, if it takes a charge at all.
In healthy shape, it would be a workable battery. The wires are a bit marginal. I would check that they do not get too hot in use.
Also, if the battery is healthy, you do not need a step-up converter because it would be at 6v or higher.
If I were doing this, I would go with a 3s battery. This puts the pack range from about 14v at full charge down to 9v at full discharge. This gives a margin so that a voltage regulator can drop down to 6v with enough margin that the input side does not go too low.
HobbyKing has a lot of good selections. Let us know if you want help picking out a suitable battery.