My concern is mainly with acceleration. With no power increase, if you have a wheel for which its weight is farther from the hub because it is bigger (never mind being heavier), i imagine it will be harder to accelerate
if its weight is further from the crank
but yes you're right. the longer the distance, the worse off you are, just you want to measure the distance from the crank.