The CRDs get their brake vacuum from the Vacuum Pump whereas the Gassers get it from the Intake Manifold.
The rubber pipe from the Vacuum Pump goes up to the Brake Booster and has a branch going into the cabin to drive the interior vents ie. directs where the airflow goes...feet, side vents, demister.
Check if all these vents work correctly...if the vacuum pipe to the cabin has come off or is torn then the air will only come from the demister vents....this may affect the Brake Booster vacuum!
If that is not the problem then there may be a problem with your Brake Booster and I have heard of a few problems with the Vacuum Pump starting to fail....it has carbon vanes that can wear out but this is not common.
My 2002 Export 2.5L CRD has no problems in this area..years ago the Dealers knocked off the vacuum pipe to the interior so I only had air-flow to the demisters.
Do you have the Service Manuals for your Jeep?
You can download them here..2004 is not covered but the 2003 will be close enough!
But very important is to download the section marked:
"2_5 2_8 export diesel" as it has information specific to these old Export Diesels of ours...there is one section that lists the Error Codes and tell you what to do about them!
The Error Codes cannot be pulled out by any scanners other than the Dealers' OBDIII Scanner as the early KJs use a PCI bus.
There is a way to pull the codes out using a "Key" method...get back to me if you need to know how to do this!
www.colorado4wheel.com/manuals/Jeep/KJ/