Position tracking/motion controller

Here's my project: I want to control a robot arm with a handheld device (think Wiimote) that inverse engineers its movement into movement of the "hand" of the arm. So it's distances between 1 cm and 1 m, and the device could be reset to take the current position as the new zero in the coordinates.
I've been trying to do this with an IMU (Pololu LSM6DS33 - 3 DoF accelerometer + 3 DoF gyro). Apparently the problem is much more difficult than double integration. But it seems like it can be done with those constrains.
I've been also thinking about just using a PS Move or a Wiimote, but I want to make my own system (and getting PS Move to work on PC seems like a harder problem, tbh).
This was supposed to be an improvement over the standard Master-Slave in which the master is a basically the scaled varsion of the slave.
Your insights? Is it possible with an IMU (perhaps even with the help of a camera for reference)? If it isn't, then what would be a simple alternative? If it is, then how would I actually get it done?

So you have something that you wave around in the air and want to track position of?

You need beacons & triangulation of sorts to measure the position. An IMU is quite hopeless as you already found out.

"Your insights? Is it possible with an IMU (perhaps even with the help of a camera for reference)? If it isn't, then what would be a simple alternative? If it is, then how would I actually get it done?"

Not a beginner project. Probably no "simple alternative", as if there were, everybody would already be using it. I'd suggest using a less complex control method to get your arm working as desired, then work on the controlling the arm with a remote control. Start working on the most complex part of the project first.

I actually got it working just today with the Move, so that's what I'm going to be focusing on now, even if it isn't the best option.
Just in case anyone finds this post looking for a solution I used
https://github.com/thp/psmoveapihttps://github.com/HipsterSloth/psmove-unity5
http://r3dstar.co.uk/?p=211
but most improtantly I had previously installed Scp Toolkit, so to get PS Move API working I had to use ScpToolkit Clean Wipe Utility.