5 Inch Exhaust..Good or Bad?

[06Cummins]

I recently put in a 5 Inch downpipe back straight pipe. I read somewhere that people were either losing power with the bigger exhaust or it was making the turbo spool slower. I know the turbo creats the backpressure so i really wasnt hesitant putting it on, but now i am rethinking it after i read that. Also, I saw it mention that the 5 inch really didnt do anything until your running over 550rwhp. Is this true? Should I switch back to 4 inch on my stock truck, which will be getting intake and a chip soon...i just dont want to lose power or slow the spooling of the turbo. Any suggestions??