RPG (Role Playing Game) magic systems can roughly be divided up into "fixed spell" and "freeform" mechanics. Fixed spell systems are often highly mechanistic, where the operation of each spell is exactly calculable. Freeform mechanics, on the other hand, call for the GM (Game Master) to judge the difficulty of a spell based on little information as well as a large degree of randomness.
Neither of these, however, is "mysterious". A mystery means that no pattern is obviously visible – but there is a hidden pattern. For a magic system to be mysterious, there must be hidden patterns which the magician character does not know at first, but which can with effort be discovered. In a game, this means that there must be either hidden variables or even hidden rules. An extreme of this would be that the GM secretly designs the magic system and only lets the player learn it a bit at a time (i.e. completely hidden rules). However, mystery can be injected by having hidden variables. i.e. How a PC's magic works depends on factors which are defined by GM, but which the player must deduce from other clues.
Via Hacker News [1], “Breaking Out of Scientific Magic Systems [2]”
This is more observations about magic in role playing systems than it is a system to use to replace an existing magic system. The author is right that we (modern players) tend to be reductionist about magic systems because of modern science in today's world (I know I am a reductionist when it comes to magic systems as a player—I never did get a good grip on the magic system in Mage [3], probably the closest role playing system where “magic” is still mysterious and needs to be discovered because the system was so vague and contradictory) and maybe we need to loosen up a bit. I don't know … it sounds like it would be a lot of work for the GM, and possibly alienate the players.
[1] https://news.ycombinator.com/item?id=9631825
[2] http://www.darkshire.net/jhkim/rpg/magic/antiscience.html