Sending only 1 frame and flushing it should take less time than filling the whole buffer with frames and than emptying it, right?
Yes, but that is not what you were measuring, before. You were measuring the time to just write to the buffer, sometimes, and the time needed to write some data to the buffer and the time needed to wait for the buffer to partially drain, other times.