Are you only changing mode (fullscreen/windowed/borderless) or are you also changing the resolution?
If you are changing both every time, try changing the mode and resolution seperately to find out what exactly crashes you. Maybe it's the resolution change that crashes you, and not the mode change.
When you change the resolution make sure you don't select resolutions with 32 bpp.
You can also try windowed instead of borderless mode.