Issue with the PWM of my motor when using the IRremote library (4.4.1)

I have an issue with my code. I'm trying to create a code to control a motor's speed using an IR remote. I have a dedicated code to try out my IR remote and it works perfectly. The same goes with my motor it also works perfectly on other codes. The issue is when I try to use both it doesn't exactly work. I can get the pwm of my motor to 0 and 255 (HIGH and LOW) without any issues but when I try any other values between these it doesn't work and the motor just stops running. I have figured out that the issue is caused by this line but the thing is that without it I cannot recieve IR signals so the whole code would be meaningless. Here's the line that makes the motor not work with intermediatre values :

IrReceiver.begin(IR_RECEIVE_PIN, ENABLE_LED_FEEDBACK);

Here's the code :

#include <IRremote.hpp>
const int IR_RECEIVE_PIN = 6;
int Z=0;
int pwm = 3;
int dir = 2;

void setup() {
  // put your setup code here, to run once:
Serial.begin(9600);
IrReceiver.begin(IR_RECEIVE_PIN, ENABLE_LED_FEEDBACK);
pinMode(pwm, OUTPUT);
pinMode(dir, OUTPUT);
}

void loop() {
  // put your main code here, to run repeatedly:
 digitalWrite(dir, LOW);
 if (IrReceiver.decode())
 {
  Serial.println(IrReceiver.decodedIRData.decodedRawData, HEX);
  IrReceiver.printIRResultShort(&Serial);
    switch(IrReceiver.decodedIRData.decodedRawData)
    {
      case 0xF50ABF00:
        Z=100; //Works fine with 255 and 0 but not with 100 or any values between 0 and 255
        break;
      default: 
        break;
    }
    switch(IrReceiver.decodedIRData.decodedRawData)
    {
      case 0xF708BF00:
        Z=0;
        break;
      default: 
        break;
    }
 IrReceiver.resume();
 }
 analogWrite(pwm, Z);
 delay(200);
}

Keep in mind im using the latest release of the IRremote library (4.4.1) and every single piece works perfectly on it's own.

Use any other pin (5,6,9,10) except 3 or 11 for analogwrite.