The maximum custom paper size for the 3800 is 43.18cm x 95cm.
If on a Mac you set up a custom paper size larger in width or length then only a portion of the image is printed aprox A4 size (with OSx there is no warning that you have exceeded the printers maximum size).
The other thing tho be carfull of is the seting of the margins in the custom page setup.
So setting the sheet to 43.2 might explain the first problem but not the second one which was set to 40x40cms with an image of about 38x39cms.
The margins options allows either for user defined or a specific printer. I chose teh 3800. The defaults were all .32cms which I didn't change. Can you be more specific about what I should be careful of in the margins?