So I have no problem wearing shorts to work. Which I've been doing for the last month. It makes me feel like I'm on vacation all the time. Or at school.
The downside? When you wear shorts, the splashback from the urinal is very prominent. With the jeans on, you didn't know and you didn't care. With the shorts, ugh, it's a different story.
Just try not to think about it.