2012-04-09 07:42:30
It heavily depends on what your product is, but you've at least these possible models:
1) Fully open source with lack or light documentation. This makes your product essentially free but users pay for support and/or the docs. I can't remember any specific example of a project selling the docs but I'm sure someone will.
2) Dual License model. A very popular example is ExtJS which is GPL (v3 iirc), however, if you wish to keep some code secret (including server parts) you might need a commercial license. And of course there are support plans available, as well as SVN/GIT access to the latest (devel) version.
3) Dual License with a Enterprise version. Essentially what MySql does where they offered an open source version but if you wanted fine tunned performance, support for enterprise hardware and support then you need the Enterprise version.
4) Dual License with long term support. Some projects like Liferay or Red Hat Enterprise use free versions as beta versions - after a while they release a long term supported version for enterprises and backport the important security and bug fixes. Maybe you already know but some companies are very slow to adopt new tech and ever slower to keep up, if they can keep a 4 year old version of the software that does the job well and still get support and bug fixes, you're best pals.
5) Early access model. Another possibility is to offer early access to new versions. For instance, the Xming project (a X11 server for Windows) offers donators access to new versions much earlier. You can even create a "pool" mode where you release the new version once X dollars are donated.
Depending on your target audience and the possibility of some of the adjustments required by those suggestions you might find a suitable model or cook some solution with ideas from several.
I can't think of a way of guaranteeing that you can feed yourself whether or not you open-source your code! Making it as an independent software vendor is hard. Above you, you have big companies who like money and won't hesitate to offer similar software, independently developed, if it looks like you've found a good market. Below you, you have FLOSS developers who won't hestitate to offer similar software for free if it looks like your software offers useful features for users. (In some cases, these groups may overlap.)
That said, you haven't given us anywhere near enough information to answer your question. Are you talking about highly specialized software for a niche market, or general purpose software with a potentially huge market? The edge-effects of open-source development are much more likely to be useful and beneficial to you in the latter case.
What do you get out of open-sourcing your software? Free publicity is almost certainly the biggest factor. How big is your advertising budget? Also, what about distribution channels? Remember, you're competing with big companies and (if you go the non-free route) open-source developers/companies. How are people going to hear about your software, and find it if they do hear about it, and decide if they like it better than other similar software?
Making your code proprietary greatly increases your per-user income, but makes it much more difficult (and expensive) to get new users. Open-sourcing your code makes it much easier to get new users, but greatly reduces your per-user income. Independent comic artist Phil Foglio started putting his Girl Genius comic up as a free webcomic, and said that his readership grew tenfold and his sales quadrupled. But that may or may not be typical.
There's also the possibility of hybrid models, like releasing the core as open source, but charging for add-ons, or, if you think other companies may want to adapt and sell your code, offering a choice between a restrictive free license (e.g. GPL) or a commercial for-pay license. Depending on what your program is and how it works, those may or may not be viable options--you haven't given us enough information to tell.
Bottom line, though: all the cards are stacked against you no matter which way you go. And, while you've given us very little to go on, it's quite likely that even if you gave us ten times the details you have so far, it still wouldn't be enough information to make more than a wild guess. Going it independent is hard and extremely risky. There's a reason that something like 90% of all programmers are employed developing internal software that never gets licensed or distributed outside of a single company--it's one of the few ways to be sure you eat.