First of all, cornflake is right, you're not using ASL, and it's sorta rude to say it like that. Harlequin is also right, 'somatic' is probably the best word to use.
To actually answer the question, it sounds like you're making more work for yourself. If it's a whole system, you're going to have to distinguish between different spells, and different magical schools, using descriptions of body language and movement. Maybe casting a fire ball means moving my left hand from my left shoulder to my right shoulder, then moving my left hand quickly down my right arm and then, in a fluid motion, gesturing at my target. That's a lot more words than 'Alakazam!'. And since my enemy will have to counter my fire ball, his ice shield might need to touch both hands to the ground, then put his right hand on top of his left and spin around with his hands still on the ground. We're taking up a lot of ground describing the fight, so you need to be really sure you're ready to not only do this, but keep a detailed record of how to cast every single spell. And you need to know what types of gestures categorize each different type of magical casting: My fire mage is using her left hand and my ice mage is using his right, is that what divides their magical schools? Is the spinning a gesture common to defensive spells, and her fire shield would also involve spinning? Or is it something else, do defensive spells need you to be hunched down like he is, and all offensive magic requires you standing straight on both feet?
I'm not saying that a physically based casting system isn't a cool idea, it totally is and it totally opens up avenues into how to stop an enemy caster and how to hide the fact that you're casting a spell and what about dance or sports, how do those physical activities affect spell casting, because all of that is super neat. I'm saying that doing it to get out of making up a dozen words for magic spells is cheap, and it won't pay off if that's the only reason for doing it. You'll need to make a language either way, it's just the kind you want to create.
So either really commit to a cool idea, or find a way to make the more traditional route work for you. (I recommend Holly Lisle's "Create-A-Language Clinic", if it's just that making up words intimidates you.)