Hello. I've been trouble shooting my problem for days and have not been able to resolve the issue. I've simplified the hardware and firmware as much as possible to find the problem. Fortunately, I have narrowed in on the problem but don't understand why it is a problem.
For my test I'm using a modified version of the Blink example. If I use "INPUT_PULLUP" as the pin mode and then spin the encoder the built in LED will hesitate and eventually hang. If I change the pin mode to "INPUT" and spin the encoder, the LED continues to flash as expected. I do understand that I can test an external pull-up but I'm concerned as to why the internal pull up is having an issue. Thanks for your help.
Hardware:
- Arduino Nano clone running Optiboot (had factory old bootloader originally with same problem, also tried loading sketch HEX file with Atmel Studio)
- Rotary encoder (center pin to ground, First pin to D5, Second pin to D6, Switch to D3 and ground) There is a .1uF capacitor between D5 and ground and D6 and ground. I've tried with and without the capacitors as well as trying different pins on the Nano.
Firmware
#define PINA 5
#define PINB 6
#define PINSW 3
void setup() {
// initialize digital pin LED_BUILTIN as an output.
pinMode(13, OUTPUT); //built in LED
pinMode (PINA, INPUT_PULLUP); // enable pull-up for Encoder A
pinMode (PINB, INPUT_PULLUP); // enable pull-up for Encoder B
pinMode (PINSW, INPUT_PULLUP); // enable pull-up for switch
}
void loop() {
digitalWrite(13, HIGH); // turn the LED on (HIGH is the voltage level)
delay(250); // wait for a second
digitalWrite(13, LOW); // turn the LED off by making the voltage LOW
delay(250); // wait for a second
}
Hi,
Welcome to the Forum.
Can you please post a copy of your circuit, in CAD or a picture of a hand drawn circuit in jpg, png?
Karma for using code tags.
Tom... 
It's been a while since I used rotary encoders, but I pulled an old sketch I used to test my rotary encoders back then ... maybe it can help?
/*******Interrupt-based Rotary Encoder Sketch*******
by Simon Merrett, based on insight from Oleg Mazurov, Nick Gammon, rt, Steve Spence
*/
// PIN SELECTS --------------------------------------
int setPin = 7; // Select input pin for rotary encoder button
static int pinA = 2; // Our first hardware interrupt pin is digital pin 2
static int pinB = 3; // Our second hardware interrupt pin is digital pin 3
// VARIABLE DEFINITIONS -------------------------------------------------
int setbutton = 0; // variable to store realtime reading of the rotary encoder button (preset store and recall button)
int setbuttonstate = 0; // variable to store previsous state of the rotary encoder button (preset store and recall button)
// -------------- ROTARY ENCODER VARIABLES -----------------------------------
volatile byte aFlag = 0; // let's us know when we're expecting a rising edge on pinA to signal that the encoder has arrived at a detent
volatile byte bFlag = 0; // let's us know when we're expecting a rising edge on pinB to signal that the encoder has arrived at a detent (opposite direction to when aFlag is set)
volatile byte encoderPos = 0; //this variable stores our current value of encoder position. Change to int or uin16_t instead of byte if you want to record a larger range than 0-255
volatile byte oldEncPos = 0; //stores the last encoder position value so we can compare to the current reading and see if it has changed (so we know when to print to the serial monitor)
volatile byte reading = 0; //somewhere to store the direct values we read from our interrupt pins before checking to see if we have moved a whole detent
// INITIALIZE ----------------------------------------------------------
void setup() {
// declare Pin I/O ----------------
pinMode(setPin, INPUT); // declare the Rotary Encoder Button pin as an INPUT
pinMode(pinA, INPUT_PULLUP); // set pinA as an input, pulled HIGH to the logic voltage for rotary encorder knob
pinMode(pinB, INPUT_PULLUP); // set pinB as an input, pulled HIGH to the logic voltage for rotary encoder knob
// Initialize Interrupts -----------------------------------
attachInterrupt(0,PinA,RISING); // set an interrupt on PinA, looking for a rising edge signal and executing the "PinA" Interrupt Service Routine for rotary encoder
attachInterrupt(1,PinB,RISING); // set an interrupt on PinB, looking for a rising edge signal and executing the "PinB" Interrupt Service Routine for rotary encoder
Serial.begin(9600); // start the serial monitor link
}
//MAIN CODE ----------------------------------------------------------------
void loop() {
setbutton = digitalRead(setPin); // Read the rotary encoder button's state for preset storage and recall
if(oldEncPos != encoderPos) {
Serial.println(encoderPos);
oldEncPos = encoderPos;
}
}
//FUNCTION DEFINITIONS ----------------------------------------------------------------
void PinA(){
cli(); //stop interrupts happening before we read pin values
reading = PIND & 0xC; // read all eight pin values then strip away all but pinA and pinB's values
if(reading == B00001100 && aFlag) { //check that we have both pins at detent (HIGH) and that we are expecting detent on this pin's rising edge
encoderPos --; //decrement the encoder's position count
bFlag = 0; //reset flags for the next turn
aFlag = 0; //reset flags for the next turn
}
else if (reading == B00000100) bFlag = 1; //signal that we're expecting pinB to signal the transition to detent from free rotation
sei(); //restart interrupts
}
void PinB(){
cli(); //stop interrupts happening before we read pin values
reading = PIND & 0xC; //read all eight pin values then strip away all but pinA and pinB's values
if (reading == B00001100 && bFlag) { //check that we have both pins at detent (HIGH) and that we are expecting detent on this pin's rising edge
encoderPos ++; //increment the encoder's position count
bFlag = 0; //reset flags for the next turn
aFlag = 0; //reset flags for the next turn
}
else if (reading == B00001000) aFlag = 1; //signal that we're expecting pinA to signal the transition to detent from free rotation
sei(); //restart interrupts
}
Minor detail, doubt it is the problem but code says switch on pin 3, post says pin 4.
There has to be more to the story that we cannot see. Please post a photo of your encoder and wiring to the Nano as I suspect the 100nf caps are not connected to ground.
How are you powering the Nano?
Try using a variable name other than PINB, that is already used to refer to the Port B input register.
I've corrected the D3 and D4 confusion in my first post.
I renamed PINA to EncPINA. Same for PINB. This stopped the hanging problem but I still have a reset problem after I spin the encoder by hand for a minute.
The Nano is now being powered through the USB port attached to my computer. When it ran on the LiPo battery, the problem persisted.
As I continue testing, I added two 10k external pull-up resistors and modified my code. The reset is still occurring. Could I be dealing with a faulty Nano?
I'm also attaching a schematic. The LED in the image is a Neopixel 16 LED circle. The intent for the rotary encoder was to switch modes and colors.
#define EncPINA 5
#define EncPINB 6
#define PINSW 3
volatile boolean changed;
byte portD;
boolean up;
boolean D5;
boolean D6;
int lastPin5;
long mycounter;
unsigned long mytimer;
void setup() {
// initialize digital pin LED_BUILTIN as an output.
pinMode(13, OUTPUT); //built in LED
pinMode (EncPINA, INPUT); // enable pull-up for Encoder A
pinMode (EncPINB, INPUT); // enable pull-up for Encoder B
pinMode (PINSW, INPUT_PULLUP); // enable pull-up for switch
Serial.begin(115200);
Serial.println("start");
}
void loop() {
readPortD ();
if (changed) {
if (up) {
mycounter++;
}
else {
mycounter--;
}
if ((millis() - mytimer) > 100) {
Serial.println(mycounter);
mytimer = millis();
}
changed = false;
}
}
void readPortD () {
int d5;
int d6;
portD = PIND & B01100000; //Read and mask for D5 and D6 on jar light
d5 = bitRead(portD, 5);
d6 = bitRead(portD, 6);
if (lastPin5 != d5) {
if (d5)
up = d6;
else
up = !d6;
changed = true;
lastPin5 = d5;
}
}
I feel I may have confused the issue by changing my code. I just went back to the original code with one change.
- PINA and PINB renamed. I think this might have been the major problem. Thank you david_2018 for pointing that out.
I've only been able to get one unexpected reset after several test cycles. I'm detecting the restart by repeatedly seeing "start" print in the Serial Monitor.
Thank you all for your assistance.
#define encPINA 5
#define encPINB 6
#define PINSW 3
void setup() {
// initialize digital pin LED_BUILTIN as an output.
pinMode(13, OUTPUT); //built in LED
pinMode (encPINA, INPUT_PULLUP); // enable pull-up for Encoder A
pinMode (encPINB, INPUT_PULLUP); // enable pull-up for Encoder B
pinMode (PINSW, INPUT_PULLUP); // enable pull-up for switch
Serial.begin(115200);
Serial.print("start");
}
void loop() {
digitalWrite(13, HIGH); // turn the LED on (HIGH is the voltage level)
delay(250); // wait for a second
digitalWrite(13, LOW); // turn the LED off by making the voltage LOW
delay(250); // wait for a second
}
}
Hi,
OPs circuit;
What is the LED and why don't you have current limit resistors in series with it/them?
What voltage do you measure the 5V pin when you are using your project?
Thanks.. Tom... 
The LED in the image is a Neopixel 16 LED circle.
But the software doesn’t have any Neopixels LED driving code, and only connections to data and 5V, they require three signals, power, ground and data. The data needs a 220R to 470R in series with it. And you need a large capacitor across the supply pins of your ring.
This should speed up your function and let you get rid of a variable.
void readPortD ()
{
byte d5 = PIND; // port read
byte d6 = ( d5 & 64 ) && 1; // should give 0 or 1
d5 = ( d5 & 32 ) && 1; // 32 is 2^5, 64 is 2^6
if (lastPin5 != d5) {
if (d5)
up = d6;
else
up = !d6;
changed = true;
lastPin5 = d5;
}
}
AVR is an 8 bit machine. An int has to process in 2 parts, a long needs 4 parts.
https://playground.arduino.cc/Code/BitMath/
warp929:
I feel I may have confused the issue by changing my code. I just went back to the original code with one change.
- PINA and PINB renamed. I think this might have been the major problem. Thank you david_2018 for pointing that out.
I've only been able to get one unexpected reset after several test cycles. I'm detecting the restart by repeatedly seeing "start" print in the Serial Monitor.
Thank you all for your assistance.
#define encPINA 5
#define encPINB 6
#define PINSW 3
void setup() {
// initialize digital pin LED_BUILTIN as an output.
pinMode(13, OUTPUT); //built in LED
pinMode (encPINA, INPUT_PULLUP); // enable pull-up for Encoder A
pinMode (encPINB, INPUT_PULLUP); // enable pull-up for Encoder B
pinMode (PINSW, INPUT_PULLUP); // enable pull-up for switch
Serial.begin(115200);
Serial.print("start");
}
void loop() {
digitalWrite(13, HIGH); // turn the LED on (HIGH is the voltage level)
delay(250); // wait for a second
digitalWrite(13, LOW); // turn the LED off by making the voltage LOW
delay(250); // wait for a second
}
}
This code doesn't even pretend to read any the pins. If you get a reset, check your wiring.
Also, port D pins 5 and 6 are Nano board pins D5 and D6, not sure if 5 and 6 are the same.
Why is D4 permanently connected to 5.5V?
aarg:
Why is D4 permanently connected to 5.5V?
I first thought that too.
But its two connections to the D4, it is a Neopixel ring.
Its one of the annoying things that Eagle does with all the crap text, half or more you don't need displayed.
Tom.... 
aarg - D4 is the Data wire for the Neopixel. The power pin of the Neopixel is going to the power switch. Like TomGeorge said, it's Eagle putting text over things.
GoForSmoke - I kept removing items from my project code and the problem persisted. I then created this very simple code from the Blink example and found the problem also persisted. No point in adding more code if the simple one doesn't work. Also, thank you for the code tips.
Grumpy_Mike - I removed the code for the Neopixel during troubleshooting. I will add the resistor on the data line. As for the capacitor, I only add it when there are 50+ LEDs. I'm not sure it is necessary for this project but I'm always willing to learn. Just to note, some of my similar projects have been running for years without a capacitor.
At this point, I'm going to rewire the project in case there is a bad connection somewhere. I do think the big problem was using PINA and PINB in the #define statement. Thanks again to david_2018 for pointing that out.
Thank you all for your help. I will post again once the project is rewired and tested.
Thing is we need to see the code and schematic that relate to each other, otherwise it leads to all sorts of confusion as seen here.
warp929:
GoForSmoke - I kept removing items from my project code and the problem persisted. I then created this very simple code from the Blink example and found the problem also persisted.
#1 problem now is finding out how come that sketch resets/hangs. Are the neopixels connected?
Hi,
Can you post a clear picture(s) of your project so we can see your component layout please?
Thanks.. Tom... 
Only thing I can think of is that some of the pins connected to your rotary encode might have been damaged, depending on what the redefinition of PINA and PINB did to the underlying code.