I tried the recursive format you suggest:
- Add a library.properties file (I modified the one from the LiquidCrystal I2C library
- Add a src folder, and place the .h and .cpp files.
Unfortunately the result was pretty similar.
Alternatives for FSM.h: [FSM@1.0]
ResolveLibrary(FSM.h)
-> candidates: [FSM@1.0]
"C:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino7/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega2560 -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\cores\\arduino" "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\variants\\mega" "-If:\\Evan\\projects\\arduino\\mega_shiled_for_CNC_router\\software\\arduino-ATC-spindle-controller\\libraries\\LiquidCrystal_I2C" "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\libraries\\Wire\\src" "-If:\\Evan\\projects\\arduino\\mega_shiled_for_CNC_router\\software\\arduino-ATC-spindle-controller\\libraries\\FSM\\src" "C:\\Users\\evan\\AppData\\Local\\Temp\\arduino-sketch-0232EEA6C455D494983E7C8CA1928504\\sketch\\Spindle_program.ino.cpp" -o nul
"C:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino7/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega2560 -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\cores\\arduino" "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\variants\\mega" "-If:\\Evan\\projects\\arduino\\mega_shiled_for_CNC_router\\software\\arduino-ATC-spindle-controller\\libraries\\LiquidCrystal_I2C" "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\libraries\\Wire\\src" "-If:\\Evan\\projects\\arduino\\mega_shiled_for_CNC_router\\software\\arduino-ATC-spindle-controller\\libraries\\FSM\\src" "f:\\Evan\\projects\\arduino\\mega_shiled_for_CNC_router\\software\\arduino-ATC-spindle-controller\\libraries\\LiquidCrystal_I2C\\LiquidCrystal_I2C.cpp" -o nul
"C:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino7/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega2560 -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\cores\\arduino" "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\variants\\mega" "-If:\\Evan\\projects\\arduino\\mega_shiled_for_CNC_router\\software\\arduino-ATC-spindle-controller\\libraries\\LiquidCrystal_I2C" "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\libraries\\Wire\\src" "-If:\\Evan\\projects\\arduino\\mega_shiled_for_CNC_router\\software\\arduino-ATC-spindle-controller\\libraries\\FSM\\src" "C:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\libraries\\Wire\\src\\Wire.cpp" -o nul
"C:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino7/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega2560 -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\cores\\arduino" "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\variants\\mega" "-If:\\Evan\\projects\\arduino\\mega_shiled_for_CNC_router\\software\\arduino-ATC-spindle-controller\\libraries\\LiquidCrystal_I2C" "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\libraries\\Wire\\src" "-If:\\Evan\\projects\\arduino\\mega_shiled_for_CNC_router\\software\\arduino-ATC-spindle-controller\\libraries\\FSM\\src" "C:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\libraries\\Wire\\src\\utility\\twi.c" -o nul
"C:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino7/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega2560 -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\cores\\arduino" "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\variants\\mega" "-If:\\Evan\\projects\\arduino\\mega_shiled_for_CNC_router\\software\\arduino-ATC-spindle-controller\\libraries\\LiquidCrystal_I2C" "-IC:\\Users\\evan\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.3\\libraries\\Wire\\src" "-If:\\Evan\\projects\\arduino\\mega_shiled_for_CNC_router\\software\\arduino-ATC-spindle-controller\\libraries\\FSM\\src" "f:\\Evan\\projects\\arduino\\mega_shiled_for_CNC_router\\software\\arduino-ATC-spindle-controller\\libraries\\FSM\\src\\FSM.cpp" -o nul
Alternatives for ../FSM_Lib/FSM.h: []
ResolveLibrary(../FSM_Lib/FSM.h)
-> candidates: []
Using library LiquidCrystal_I2C at version 1.1.1 in folder: f:\Evan\projects\arduino\mega_shiled_for_CNC_router\software\arduino-ATC-spindle-controller\libraries\LiquidCrystal_I2C
Using library Wire at version 1.0 in folder: C:\Users\evan\AppData\Local\Arduino15\packages\arduino\hardware\avr\1.8.3\libraries\Wire
Using library FSM at version 1.0 in folder: f:\Evan\projects\arduino\mega_shiled_for_CNC_router\software\arduino-ATC-spindle-controller\libraries\FSM
Compilation error: Error: 2 UNKNOWN: exit status 1

Here is the library.properties file.
name=FSM
version=1.0
author=Evan
maintainer=Evan
sentence=A library for arbitrary Finite State Machines
paragraph= The library provides a base structure for the Finite state machine. It is meant to be paired with a user defined library that defines the specific events, actions, and transitions.
category=Structure
url=https://github.com/mccartes/arduino-ATC-spindle-controller
architectures=avr