Garfielder:
I don't need the USB unit, the regulator and any onboard led but I don't want to mess with the hardware. So what is my best option?
A Li-ion or Li-Po battery with at least 5000 mAh (for really long runs), because the on-board LEDs do take up some juice (between 5-10 mA) each one.
To bypass any regulator, don't feed the power from the barrel jack, VIN or RAW pin; but from the 5V pin. Yeah I know it says 5V, but by experimenting I discovered that the microcontroller can perfectly live with the voltage range of a Li-ion or Li-Po cell; just don't let it fall below 3 volts or the brown-out detector may kick-in and cause erratic resets (that is, unless when it's asleep with that turned off).
Now, there's also the issue of whatever storage medium you wanna use; it also has to tolerate the voltage range of a Li-ion or Li-Po cell. Remember, you are avoiding regulators to minimize power losses.
Garfielder:
I prefer to write to a file on a SD card but I saw that those are problematic power consumption wise. Do I have a better alternative?
In this topic we were talking about an interesting type of memory that claims to have a very low power consumption suitable for battery-powered devices; the only downside is that the largest capacity you can find is 2 Mbits (256 KB). Maybe by logging everything as raw binary, but still 256 KB can get filled up for not so long (specially at a pace of one second). Assuming 16-byte records (4 float variables) being saved every second, 256 KB fills up in 16384 seconds (273 minutes or roughly 4 and half hours).
So for mass storage options there aren't many; in fact I dare to say an SD card is your only feasible choice in this regard; even USB pendrives draw the same or more power. That is, unless I missed an actual low-power NAND flash chip with built-in SPI interface and massive capacity (other than an SD card of course).
Some forums suggest to turn on/off the card via MOSFET to save power; but I'm dubious about how much in the end. Powering it off is the same as removing it from the slot; so every time you wanna do something with it, you'll have to repeat all the initialization process (SD.begin() and open the file). I don't know you, but this sounds counterproductive to me, at least for an update rate of once per second.
Some also mention that power consumption varies from manufacturer (both idling and busy), so there's no better way of knowing it by looking at a datasheet (if it has a brand printed on it), or measuring its current draw with a precise enough ammeter (in the milliampere range).
This is important to make your own conclusions about whether or not a complete power-off of the card actually worths the hassle involved, for the sake of saving precious juice on the battery.