I have been able to trickle charge a 12V AGM battery from a solar panel (/w night blocking diode) at 0.02C (e.g. 200mA on a 10AHr battery) without needing a charge controller, but I do use a switchmode power supply (OKI-78SR-5) to convert the battery voltage to 5V for the controller.
I am setting up to try a Raspberry Pi Zero like this, but it will eat a lot more power and need a larger battery to do the same trickle charge rate (1A on a 50AHr battery). The Pi Zero and AVR combo use about 170mA which is 4.25AHr. The trickle charge will (on a good day) last for about 5Hr yielding 5AHr of charging if I can find a spot that has sun for that long. That may not be enough charge from what I have seen, so I will have the Pi Zero halt at midnight or poor sun to be safe.