It heavily depends on what your product is, but you've at least these possible
models:
1) Fully open source with lack or light documentation. This makes your product
essentially free but users pay for support and/or the docs. I can't remember
any specific example of a project selling the docs but I'm sure someone will.
2) Dual License model. A very popular example is ExtJS which is GPL (v3 iirc),
however, if you wish to keep some code secret (including server parts) you
might need a commercial license. And of course there are support plans
available, as well as SVN/GIT access to the latest (devel) version.
3) Dual License with a Enterprise version. Essentially what MySql does where
they offered an open source version but if you wanted fine tunned performance,
support for enterprise hardware and support then you need the Enterprise
version.
4) Dual License with long term support. Some projects like Liferay or Red Hat
Enterprise use free versions as beta versions - after a while they release a
long term supported version for enterprises and backport the important security
and bug fixes. Maybe you already know but some companies are very slow to adopt
new tech and ever slower to keep up, if they can keep a 4 year old version of
the software that does the job well and still get support and bug fixes, you're
best pals.
5) Early access model. Another possibility is to offer early access to new
versions. For instance, the Xming project (a X11 server for Windows) offers
donators access to new versions much earlier. You can even create a "pool" mode
where you release the new version once X dollars are donated.
Depending on your target audience and the possibility of some of the
adjustments required by those suggestions you might find a suitable model or
cook some solution with ideas from several.
I can't think of a way of guaranteeing that you can feed yourself whether or
not you open-source your code! Making it as an independent software vendor is
hard. Above you, you have big companies who like money and won't hesitate to
offer similar software, independently developed, if it looks like you've found
a good market. Below you, you have FLOSS developers who won't hestitate to
offer similar software for free if it looks like your software offers useful
features for users. (In some cases, these groups may overlap.)
That said, you haven't given us anywhere near enough information to answer your
question. Are you talking about highly specialized software for a niche market,
or general purpose software with a potentially huge market? The edge-effects of
open-source development are much more likely to be useful and beneficial to you
in the latter case.
What do you get out of open-sourcing your software? Free publicity is almost
certainly the biggest factor. How big is your advertising budget? Also, what
about distribution channels? Remember, you're competing with big companies and
(if you go the non-free route) open-source developers/companies. How are people
going to hear about your software, and find it if they do hear about it, and
decide if they like it better than other similar software?
Making your code proprietary greatly increases your per-user income, but makes
it much more difficult (and expensive) to get new users. Open-sourcing your
code makes it much easier to get new users, but greatly reduces your per-user
income. Independent comic artist Phil Foglio started putting his Girl Genius
comic up as a free webcomic, and said that his readership grew tenfold and his
sales quadrupled. But that may or may not be typical.
There's also the possibility of hybrid models, like releasing the core as open
source, but charging for add-ons, or, if you think other companies may want to
adapt and sell your code, offering a choice between a restrictive free license
(e.g. GPL) or a commercial for-pay license. Depending on what your program is
and how it works, those may or may not be viable options--you haven't given us
enough information to tell.
Bottom line, though: all the cards are stacked against you no matter which way
you go. And, while you've given us very little to go on, it's quite likely that
even if you gave us ten times the details you have so far, it still wouldn't be
enough information to make more than a wild guess. Going it independent is hard
and extremely risky. There's a reason that something like 90% of all
programmers are employed developing internal software that never gets licensed
or distributed outside of a single company--it's one of the few ways to be sure
you eat.