side mouse button alt tabs me form game

I have keybinded an inventory slot to mouse button 4 which is what I use for pots and later on for certain items. For some reason when I click it alt-tabs me from the game. This for some reason only happens once per game and it's also random. It sometimes happens the first time I click it, and sometimes the 10th. I try spamming it when the game starts so it can alt-tab me at that point but I sometimes forget and I die as a result of it. Does anyone else have this issue?
Report as:
Offensive Spam Harassment Incorrect Board