As someone who made the switch last year and had hesitated for pretty much the same reasons as you, I can with all honesty tell you that it was much less of a hassle than I had feared. Basically not very different than when you do an install of Windows. There was some struggle understanding the concept of mounting drives and getting that to work, which was probably the biggest issue for me, and there was plenty of guides out there which helped me through.

The best part is when it is done and that looming depressing feeling of having to deal with Windows on a daily basis was gone. I had had no idea of how much that feeling actually had negatively influenced me until it simply wasn’t there anymore.