Do you carry your camera and your music player with you all the time? Those are two big blockers for a dumbphone for me. Not the only ones, but the big ones. I've settled, in the mean time, for making my phone less appealing to use:
An Inhabitant of Carcosa: Minimizing phone use (re: All a phone needs to do)
The music player, yes. It's a Surfans F20 and not at all heavy. I'm not carrying redundant devices as often as you might expect because the phone stays in the glove box when I'm out. I only bother carrying it because you could walk a hundred miles without seeing a single public phone where I live.
I don't have a discrete camera yet. Might buy myself one for Xmas so I'm not constantly borrowing my wife's DSLR.