A harsh solution could be disabling the chat while you're spectating but then you can also leak their spots in VC although I don't think many users would do that.