This may be a problem with the GPU of your MacPro, especially if it is an early MacPro with an NVidia 7300 or an ATIX1900 video card as they are not fully compatible with GPU and OpenGL requirement of Photoshop CS5. (The ATIX1900 is compatible but will only run Open GL Advanced in Basic mode.)
See Tech Note: http://kb2.adobe.com/cps/404/kb404898.html
Other factors may involve your current Photoshop CS5 settings including scratch disk space. The following Tech Note covers memory optimization in Photoshop CS5:http://kb2.adobe.com/cps/404/kb404440.html
I have an early version Mac Pro running OSX 10.6.3, 12 GB of RAM, dedicated 500 GB scratch disk, ATIX1900XT video card (512MB vram), and had a similar experience with Photomerge and Content Aware Fill. As you know Photomerge can produce a very large file and use a lot of memory when all of the auto options are engaged. What worked for me was to reduce the file size and save the combined file prior to attempting the Content Aware Fill. Reducing the size of the resulting file or the bit depth (16 to
and saving the file frees up the necessary RAM for the Content Aware Fill.