You're right that I only thought of it from a tdm perspective...since you know...that's all the community even cares to play (for the majority). Simple solution for those other game modes where speed vials may be OP is to just not allow them in those modes. Keep it for DM/TDM only.
Also people can just use speed vials already in these modes, but no one has spoken against it yet, so it couldn't cause that big of an issue to have an unlimited option.