Enyon,
I haven't driven an STi yet (mine arrives at the beginning of August) but I do own a 3rd gen RX7. I've also driven hot WRX's as well as 2nd and 1st gen RX7s. Here is my take on things:
Many will tell you that rotary engines are unreliable. This is not strictly true. Much of the reliability nightmares came from the 3rd gen (1993-95) which was related to poor design of the exhaust system and other related issues. The new Renesis engine is a significant evolution of the 13B platform and Mazda appear to have done a superb job with the design and tuning of the car.
Don't be too put off by the low torque numbers. The RX8 gearing is very different from the STi's due to the extremely large rev range (redline at 9000). The STi gets it's impressive 0-60 times more from the AWD than the 300HP. Furthermore, rotaries are quite happy to rev to their redline as the reciprocating mass is extremely low.
In terms of power-to-weight, the rotary is unmached by any piston engine. The 13B (two-rotor) can produce upwards of 600HP and yet weighs much less than a similar 4-piston engine. It's also smaller and can be mounted much lower in the car. This is the primary reason to choose a rotary-based car--reduced weight and better handling. If you like the way a very light car feels as it turns in, then the RX8 (2970lbs) is your car. And remember, less weight means better acceleration, cornering, and braking.
Okay, so, now the bad part: rotary motors MUST be tuned correctly. Many a 13B have been destroyed by a single detonation event. The apex seals, which are similar to rings for piston engines, are extremely fragile. This means that tuning a rotary to higher power can be a daunting task. With the RX8, it's going to be even harder becuase the car doesn't even start with a turbo. My bet is that even with tuning, intake, exhaust, ingnition, and other mods you *might* see 300HP. Put a turbo on for more power and you really run the risk of toasting your rotary.
Rotary engines also are highly inefficient as compared to a piston engine. Fuel economy is poor, and fuel consumption at the track is high. This has impact on endurance races. That said, Mazda did an amazing job winning Le Mans in 1991 with their quad-rotor prototype cars.
http://www.scuderiaciriani.com/rx7/787B/
On the chassis side, the RX8 will probably provide a smoother driving experience and ultimately a higher cornering speed. However, the horsepower of the STi offsets this advantage. I expect the STi to be quicker at the track, but it will probably lose at the autocross. If you don't believe me, go to a local autoX event and watch a boxter or S2000 outrun a 996TT. HP and AWD are great, but weight is always a killer.
The STi will be dominant at the drag strip until you get to the highly modified classes.
The STi has a huge advantage in tuning. You can spend about $3000 on your STi and probably see an extra 50-100HP. The car will be devastatingly fast. Another $5k and you'll be in the high 400HP for sure. It's too early to talk about reliability, but if the past is any indication, the Subaru platform will be reasonably sound.
For full-bore racing (i.e. spend $50-100k on your car) the RX8 will ultimately be the quicker car. It's low-slung design, far superior aerodynamics, and rotary powerplant will defeat a similarly-prepared STi. AWD is not really an advantage on the track, where weight and chassis dynamics are key.
There's other big issues to consider. The styling of both cars are a love-it-or-hate-it kind of affair, but I think it's clear that the build quality and interior design are much nicer on the Mazda.
That said, I'm not going to upgrade my RX7 to the RX8. I don't like suicide doors, and my current car weighs under 2800lbs and will have about 350HP to the rear wheels when it gets back on the ground. I'll outrun any 500HP STi at the track. However, I'm going with the STi because of it's daily driving pluses: 4 real doors, cargo space, AWD, and great low-end grunt. Maybe an extra 50HP from a tuner wouldn't hurt...
Good luck with your car selection,
-ch