I am not using a regulator
The battery charger is a regulator of a sort. I should have use battery charger instead.
In that case it may be some type of low voltage or overheat problem. The datasheet does mention on overheat, it will taper off the charge rate current. On low voltage (3.5v) it will shut off until the input goes above 3.9v.
How does that charger do if you power it with a 5v-6v wall wart?
Edit: My apology. I was interrupted by work.
If the voltage from the solar panel drops below its Vmp
(voltage at maximum power), then you are probably using more power than the solar panel is capable of producing. But I can't see the "big picture" from here.