I agree that any formal training will nail your face to the wall with math, but I am not an EE or have even been a formal electronics student since high school, where I talked my way into 4 years of a 2 year class (starting as a freshmen).
I have a CS degree, a hand full of microsoft, linux and various database certs, Apple and A+ technician cards, and did rather ok considering I am not the best or fastest programmer ... I mostly did network management and design. Got laid off, bounced around various jobs, got laid off again ... but during this time I started to get back into electronics (bout 4 years ago) and started tinkering and reading a LOT, it got to the point where I would come home and sit at the bench or in the books for hours.
Eventually I had posted enough (interesting) crap on instructables and submitting stuff on hack-a-day I got a writing gig with hack-a-day.com, and when I got laid off that second time (just over a year ago, not from hack-a-day) I wanted to get back into a more serious IT position, though a position with electronics would be quite awesome IMO, but not really realistic.
Working with Randstad at this point, just to keep money coming in, I got a semi frantic call from my agent almost begging me to go do a 1 day only manual labor job. I usually dont play that crap, but something kicked me in the head and I ended up going to this place. As soon as I walked out onto the production floor I got a big ass smile as this place was jammed packed full of electronic production machinery, I watched every stupid machine on my breaks, and acted very eager to get my hands on them.
By the end of the day I was telling anyone who would listen "I would LOVE to work in a place like this", and eventually got sat down in a room for a short interview, then had a long interview, then got a trial period, and have been fully employed by the company for 5 months in the engineering department as a "technician" (cause they wouldnt let me put "super awesome hacker" on my card), and doing just as well as any of the CS/IT related jobs I have had in the past.
Whats the point of this book?
Any knowledge you have is not useless: After I was hired I was notified that six other people were trying for the job I didn't even know was available, and that most of them did not even know the basics of office, or if they did, it was the 1 class 10 years ago they didn't remember. I know office well, I know computer hardware well, I know how to program in a few different languages and systems, I know how to make a 1024 LED matrix on a radio shack board (its a LED lighting company) and I was able to show them in a confident manner.
Just cause you have a degree in something, it doesnt mean the rest of your life will always follow that path: and likewise if you have an interest in something, just start doing it. Life changes, go with it, if you want to get into electronics, and embedded systems, go for it, maybe you will have fun with it, maybe you will start a career with it. If you decide you need a degree in it, its never too late to get one.
Lastly in the real world, math is not really that hard: We know electronics pretty well as a planet, theres always a formula to follow, and its often right there in the datasheet. realistically you need to be able to understand that formula, and plop in your variables. Even some of the brightest people I have met will take a big nasty formula, plop in XYZ and let excel deal with it. Most of my time programming now is just making widgets that span gigabytes worth of data to do 4 function math and basic algebra on it.