Thanks DVDdoug and MarkT, writing and reading or better recording and playing of the raw digital data should work. I found this thread in the meantime: ICS43432 I2S sound recording and playback · Issue #2 · arduino-libraries/ArduinoSound · GitHub
There is also example code from Paul Stoffregen for the Teensy with the Teensy Audio Library - just as note, this will not work with the Arduino Audio lib - but you will find from line 148 on an optimized method to write sound data to SD: Audio/Recorder.ino at master · PaulStoffregen/Audio · GitHub
Yes, writing to a storage medium will be challenging. The Arduino MKR ZERO has a "dedicated SPI interfaces (SPI1)", let's try if this is sufficient. But the I2S lib allows also to specify sampleRate and bitsPerSample in begin().