Where do you suppose the energy comes from that makes the post-turbo air charge hot? It's proportionate to the temperature of the air coming into the turbo. Aside from a bit of soak from the impeller itself, it's mainly the compression that increases the temperature - see ideal gas law. The higher the starting temperature the more energy in the resulting compressed air, and the more effort it takes to compress. The intercooler can take 10 degrees or more out of the intake charge, but I don't believe the cooling efficiency increases with temperature as much as the heat of compression does.
That's before you take into account the practical effects of a car that heatsoaks at idle and has to drain the engine bay of hot air, instead of just the intake volume. I've lost count of how many times I've seen people complaining about wobbly idles and erratic fuelling from replacing their intakes and filter boxes. Good air supply matters! At least to a point. NA ram-scoops have a tiny effect, and most cold air induction kits trade weather resilience for marginal improvements over OEM offerings.