In my humble opinion, it comes down to matchups.
I was re-watching the UMBC vs Virginia game from a few years ago and I noticed that Virginia's defense failed them in that game. Their lone calling card failed them and they couldn't keep UMBC from getting to the lane. They outscored them in the paint and then torched them from downtown. UMBC dictated the tempo to their favor and Virginia just couldn't keep up because they were a slower paced team.
That's not to say that the same couldn't happen to us. Not only do we have a defense solely built on preventing players driving down the lane, but our attention to rebounding affects both the offensive and defensive side of the ball.
When it comes down to understanding the game, rebounding is what connect offense to defense and vice versa. It might be the only tangible common denominator actually. If you can rebound the basketball, your offensive metrics will be better because you'll have better opportunities from 2nd, 3rd, etc shots. On the defensive side, if you can rebound at a high percentage, you're keeping the other team from scoring on the 2nd, 3rd, etc chance opportunities.
However, as we have all witnessed, all it takes is for someone to go on a scoring binge and all bets are off. You just have to hold on for dear life and hope that you only have one guy beating you and not two or 3.