I very much agree with this sentiment. I like pico-8 in a lot of ways though, actually _because_ it's a very self-contained and quite simple system, but I've often thought it would have been a lot more fun if we'd just get to program in some sort of fantasy assembly. I would have loved to see an instruction set and architecture on such a machine that's very different from what we're used to on the likes of x86 and arm. That would have been so much fun.
It's one of those project ideas that I have every now and then, and I get excited about for a while, but then I can't muster the energy and focus to actually work on it.