Babar,
I can see the problem if there aren't enough eyes, but if there are, it might be fun. Kind of a team effort craft.
When you say "a glider this size", are you saying it's harder because it's big, or it's harder because it's small? If the wing loading is high, the glide may be fast, but that's no reason for it to be less stable, this side of Mach 0.6 or something. If the wing loading is low, that can be made to work, too. The bigger and/or heavier model is likely to have a flatter glide, even if it has a somewhat higher sink rate, because drag coefficients tend to be less at higher Reynolds numbers,.
I have had gliders with stable, hands of the radio (or free flight) with weights from maybe 8 grams on up (not counting the paper airplanes) to maybe 60 ounces. Come to think of it, I think one of them was more like 80 ounces. Spans from 8 inches (Fliskits Nanite, slightly modified) to 120 inches or more. The Nanite glided far too well, so I didn't have it for very long.
----------
Mr. Slumber,
Another thing that goes on, when wings are swept, is spanwise flow that screws up the angle of attack. A swept wing will have a bit of upwash at the tip, giving it a higher angle of attack, unless it's given washout (twist) to counteract the problem. Too much taper does the same thing. And if there's a lot of twist, then it's only right for one coefficient of lift, corresponding to stable flight at one particular speed. If one is really clever, it may be possible to make a wing that untwists itself at high speed. But that's tricky. It may be easier and more reliable just to use another gram of propellant.
A guy who just retired from NASA named Al Bowers is really into this stuff. Depending on just how deep you want to go down the rabbit hole, you might look into the Prandtl Project. Or his lectures to the Experimental Soaring Association, which can be found on Youtube.