SHOUTcast buffers around 15 to 45 seconds worth of audio on the server to keep the stream consistent and for quality gains. Without the buffer, the quality of audio would be significantly worse with poorer sound quality, gaps and audio glitches.
This behaviour is built into SHOUTcast and can’t easily be removed, nor would it be sensible to attempt to remove it. Higher bitrate plans will have slightly lower delays as the buffer will fill up quicker. This is the only way to lower the delay caused by the buffer.