Cleaning it out can't hurt. If there is oil and debris in there, you are not using your intercooler to its full potential. The gains you will get are cooler air, that = more hp, if there is crap in there. Get a few cans of oxygen sensor safe carb cleaner and pull the intercooler off, its only a few bolts and clamps.
~Murt
TTA #750
If you take a look at the valvecover breather on the drivers side, follow the pipe from the breather and you find it connected to your turbo just before the compressor house...... it´s not nescessary to have a bad turbo to get oil in the IC, it comes from the valvecover breather.