This is solid in my wheelhouse. Before going for the blue oval I was a huge Chevy fan and worked primarily on modding GM W-Body cars (GrandPrix, Impala, Monte Carlo etc). The GrandAM shares an engine/drivetrain options with the lower model GrandPrix.
The 3.4 AKA the 3400 is a pushrod 60*v6 engine that is internally balanced with high compression (for the day) non-interference engine (timing chain style) with forged internals, aluminum heads and a cast aluminium intake (it think it was still cast in '04) .
The 3400 is a VERY solid engine and will take mass amounts of punishment and will refuse to die (you can even turbo them with stock compression, the issue is fuel and head gasket; not the engine itself). There is even a CSC kit for the 3400 ;-)
The biggest issue is the transmission which I think is either the 4t40-E or the 4t45-E. Good units that will run for 100k+ when taken care of but will flake when beat hard. Make sure the shifts are smooth and crisp. The good news is even if the trans fails these are SO common you can get a rebuild cheap. If you go the custom re-build route about 1/3 to 1/2 of the parts will swap with more durable volvo parts thanks to the transmission being from Allison.
IMO the 2.4 4cyl (that is an interference engine with a belt instead of a timing chain IIRC) is junk. It does not do much better gas mileage wise when compared to the 3.4 and tends to have all kinds of little issues like intake gasket failures. It is a much newer engine from GM so you know how that goes. The 3400 in contrast was first used in the late 80s as a 2.8L (or was it a 1.8?) MPFI v6 which evolved over 15+ years to the 3.4 that is in that '04 so many of the main problems were solved long ago (in the mid 90's actually).
The other thing I would look at are the motor mounts, if they have not been replaced you will likely want to put all new ones in, stock only lasts about 5-8 years in best case conditions before the urethane breaks down and the mount gets SUPER soft.
Feel free to ask anything specific, I know entirely too much about the GAs and GPs.