The problem with using other chips "not on a premade thing" is that the leads are so small and close together that they are difficult to mount on a breakout board or something where you can access them.
Why not look at the Teensy boards as a start? www.pjrc.com They are like a system on a board, while still being compact, and programmable in the Arduino IDE still.
One thing to note is that, with the exception of the Nano Every, all of Arduino's recent boards are have ARM cores. So if you're using something like a MKR board, Nano 33 IoT, or Nano 33 BLE, you're already using ARM!
If you're really set on building a board from scratch using an ARM core microcontroller, you might look at the ATSAMD11D14A-SSU. This uses a SOIC-20 package, which is pretty easy to solder with only a cheap iron and decent solder and flux. You can get SOIC-20 breakout boards cheap from all the usual places, then build the rest of the board on protoboard or stripboard. Or go the full distance and design your own PCB if you prefer.
The ATSAMD11D14 doesn't have a lot of memory or IO, but it seems like a fun chip for people in the Arduino world to play with. Sometimes it's fun to work with a more limiting chip to see just how much you can manage to do within those constraints. It's used as the programmer and USB to serial converter chip on the Arduino Nano Every and also on the Arduino MKR Motor Carrier.
There's a nice open source CMSIS-DAP debugger that uses the 14 pin version of the chip (ATSAMD11C14):
If you decide you still want to use the Arduino framework, there's pretty good support for it via this 3rd party boards platform:
If a 16MHz Arduino can do all you want then I see no need to change.
If you need features (or performance) that the 16MHz Arduinos can't provide then identify a chip that has the features you require. There are several more powerful Arduino boards and there are very inexpensive boards using ESP8266 or ESP32 chips that include WiFi etc.
All of these can be programmed with the Arduino IDE.