Let's say I wanted to backtest a theory in performance of a handful of stocks, something like "if a stock is up/down X% in the weeks before earnings, what happens to it the day before/of/after the earnings announcement?"

What tool/website could I use to build and test something like that?

TradingView is probably your best bet

Stockfetcher has some limited backtesting capabilities. If you're computer savvy, everything can be done with Python.

The challenge with Python or something like that would be getting the data.

I'd like to see what happens to a stock that has lost some large percent of its value in the weeks/months leading up to earnings, what happens on the day of/before earnings release? Is there a nice pop from shorts covering just in case earnings are positive or have some positive comments from the company.

Example, ENPH recently. Lost a good bit, I think the day of earnings release it gained a couple points, then dropped like a rock further after release. My curiosity is on the day of the earnings, or before.