After more analysis with some output to Serial, delay is not the problem. It's quite smooth with just a few 5-10 ms delay between command and starting.
What I tough was a delay is in fact some clipping sound from stopping the playing file, and another clipping sound from starting the file. It's not constant, sometime more clipping sounds occur, sometime less.
I don't have clipping sound when files switch in automatic cycling mode or playall mode. Only if I ask to play a specific file with playFileNum() or playSpecFile().
Those clipping sounds are ruining my sound effects, it's annoying...