Open Source Solutions for Small Business Problems

I started writing a response to a discussion in the latest “Linux Link Tech Show” episode, but ended up with something far too long, so I’ve split it up into 4 posts. The next post is about the TLLTS vs TWIT debate, and introduces this set of post. The previous two are about open source support–a true story of a support incident I had, and the unwritten rules of open source support. In this post, I analyze the fundamental differences between Windows, Apple, and Linux when it comes to support.

Dann and Linc had a quite spirited debate about the merits of having a company hire low-end tech support with scripts (Dann) versus having an experienced, savvy, tech professional able to really solve the user’s problem (Linc). Dann’s point was that it can easily be more cost-effective for both the support company and the end user to go straight to reinstalling a system if rebooting doesn’t solve the user’s problem, while Linc seemed to think a savvy tech person could get to the root of the issue much quicker, and brought up the point that there’s a cost to the frustration of users being put through the whole front-line support nightmare.

I’d suggest the situation is even more complex than that, but it also differs greatly between the open source projects and proprietary operating systems. First, let me make a bold statement:

Linux support is far better than Windows or Mac.

But it also has a completely different set of rules. Learn those rules, and you’ll be able to solve your problems more satisfactorily than it’s possible to do in the proprietary world. Let’s talk about these differences, looking specifically at a who, where, what, and how long.

Who can help with your problem?

Let’s take a look at who can help with your problem:

The bottom line here is that while you probably know fewer people personally who can help you with Linux, there are more options for commercial support, and you can reach people who can do more to solve your problem for free, than you can with either Mac or Windows.

The fundamental reason for this is that anybody can read the raw source code of any open source product out there, and with enough skill and talent, can solve your problem without needing to pay anybody for the right to do so. In the proprietary world, only one company can help beyond a certain point: for Windows problems, that’s Microsoft. For problems in an application, it’s the application developer.

So the next question is:

Where can you get help?
Again, let’s compare the options:

The main point of the table above is that in the proprietary world, the harder your problem is to solve, the fewer people can help you solve it. You quickly get down to one place to go, and if it’s in the operating system, it might be expensive or not possible to fix. In the open source world, it’s nearly the opposite case–it can be harder to find the simple quick answer to your question, but the harder and deeper your problem, the more places you can go to get help.

Where do you look for help first? This is the single stumbling block for most otherwise tech-savvy users new to Linux. To learn Windows or Mac you can take a class, talk to neighbors and friends, and find lots of very low-end help that way. For Linux, unless you’re friends with some hard-core geeks, you need to go online to find help. Once you’re there, it really depends on where your problem lies.

For just figuring out how to use a system, go to the forums for your distribution. The Ubuntu forums are a great place to ask general questions. Be specific about what you’re trying to do and you’re more likely to get help. Remember that this level of support is free, and people helping you are volunteering their time and knowledge. While you’re there, see if you can answer somebody else’s question–the more help you give, the more you’ll get in return.

The other fantastic place to go for help, especially for quick questions, is IRC. IRC is a system that provides chat rooms and instant messaging. Most open source projects with any community behind them have a chat room on irc.freenode.net. Install an IRC program like Chatzilla, Konversation, Pidgin, or any number of others, connect to irc.freenode.net, pick a nickname to use, and join the channel for the program or distribution you’re having trouble with. Ask your question nicely, and if you don’t get a response immediately, keep your chat program open for a few hours–not everybody is watching the channel every minute.

What kinds of problems can you solve?
Let’s look at the same types of problems as before, and look at the resolutions:

What’s the key thing in this section? Addressing actual problems is within your control, when you’re working with open source. In the proprietary world, you’re entirely at the mercy of a single software vendor. If your problem is in the interaction between two different applications, you’re really stuck — there’s nothing you can do.

But in the open source world, there’s always something you can do. You can hire anybody with the skills to solve your problem, and fix it in the software itself. You don’t need any blessing from any single software vendor.

Open formats are perhaps more valuable than open source software, for most businesses. Because this is such a compelling advantage of open source software, many proprietary programs are beginning to open up their formats to allow other software to read them–their customers are demanding it.

How long will it take to solve your problem?
I’m not going to bother with a table for this one. The answer is nearly always “too long,” regardless of the operating system.

Actually, that’s not quite always the case–it depends on whether somebody has already solved the problem or not, as well as whether the solution is a fix or a workaround.

Many, many problems in Windows are not really fixes, they’re just workarounds. The only real “fix” for a virus-infected machine is the workaround of reinstalling your operating system. The only fix for lots of other minor issues that cause your system to slow down over time is to reboot. These are not fixes, they’re workarounds.

Real fixes take a lot longer, and need acknowledgment that the problem is real. Workarounds are band aids to get you through until there’s a real fix. And there are some real differences between the entire approaches of each operating system around these fixes.

Windows is chock-full of workarounds. Because Microsoft has gone to great lengths to maintain backwards compatibility of just about everything that’s been released for Windows since Windows 95, it’s full of workarounds to keep the old behavior. Rather than fixing behavior that might really be undesirable, they’ve had to patch it with workarounds because too many existing applications turned out to depend on that bad behavior. That, fundamentally, is why Windows is so big, bloated, slow, and painful to work with.

Apple suffers a different problem: changing their closed libraries too quickly for external developers to keep up. Each release, they break lots of things, and don’t always tell 3rd party developers ahead of time. This means the number of third party developers of Apple software is shrinking–they’ve managed to alienate quite a few. So their polish and high quality comes at the price of having a healthy thriving developer community outside the walls of Apple. There’s little transparency in this process, so developers outside Apple are always playing catch-up, and having to work around these changes in behavior.

Linux has its share of problems, too. Linux does not attempt to maintain “binary” compatibility between versions, though it does its best to maintain compatibility in source code. There is endless debate about the “correct” way to fix a problem, and competition between fixes. The challenge of this is that it can be hard for application developers to keep up, especially if they want to keep their source code closed and only ship software in binary form.

But the process is completely, utterly transparent. Anybody can see the progress on any fixes to any part of the system, and can jump in with their own solution at any time. It’s a true meritocracy, with those doing the actual work and providing the best solutions winning out over time.

The whole open source ecosystem is nimble enough to provide real fixes to technical problems, rather than just simple workarounds. If the source of the problem is design, it can take a long time to get the right design in place and resolve all the issues that changing the design causes in other applications. But when a bug gets fixed, it’s really fixed and usually doesn’t appear again.

In my early Linux system administration days, when I was first trying to set up a mail server with spam filtering, I ran across a really puzzling bug in Dspam, the software I was trying to get working. While all the other users of the software were getting great results, with Dspam catching 99%+ of all their spam, it was only catching about 70% of my spam after quite a bit of training.

I posted my results, and confusion, to the Dspam mailing list. The original developer of this software (which has thousands of users), Jonathan Zdziarski, responded that that did not sound right. He asked if he could log into my server and see what was wrong.

I created a test account for him, and logged in to the same screen so I could watch what he was doing. As I watched, he put debugging messages into his code, ran several tests, and within 10 minutes had identified the problem: the 19-digit number he put into the database (MySQL) was different than the 19-digit number he got out. It was a storage error in somebody else’s software causing the problem, a rounding error. He filed a bug in MySQL, and it was fixed within a month or two. But he also had a workaround for me: change the data type to a set of characters instead of an integer. Problem solved.

What’s extraordinary about the open source community is that this level of support happens all the time, every day, without charge, in hundreds, thousands of projects out there. People that can get to the bottom of a problem and fix it at the source, not just provide a workaround, are directly reachable and motivated to see their software work as well as possible. They’re not hidden away from the public behind a large corporation, unreachable with layers of clueless support script readers stuffed between you and them. Here are some rules for getting open source support directly from the projects:

• Before asking anybody, do your homework. Use Google, read the project FAQ, make some attempt to learn the basics without pestering people with questions they’ve already answered hundreds of times. Nine times out of ten, your problem has already been encountered and somebody has figured out a workaround.
• Limit the scope of your question to the fundamental problem. Get to the point. I’m obviously guilty of being tremendously long-winded at times, but unless your question is right at the top and asked directly, you’ll probably get ignored. Developers are busy, they don’t want to read a novel, but they’re usually happy to answer a question.
• Provide supporting details after asking your question. Many programs will create a log with lots of information that can help somebody diagnose your problem. Find what looks relevant in the logs. Specify what version of operating system, distribution, application, etc. Specify what you were trying to do, what you expected, and what really happened. But provide this stuff after asking your initial question–people aren’t going to wade through a long email to find your question.
• Contribute something. The easiest way you can contribute is by answering other people’s questions. The whole thing works because people help each other, and this help goes both ways. If you always ask questions and never answer anybody else’s, or provide any other sort of contribution, you’ll eventually start getting ignored.
• Always, always, be positive, respectful, and polite when asking your questions. Developers have a lot invested in their project, and insulting it won’t gain you any favors. Developers are under no obligation to help you either–you haven’t paid for it. Common courtesy is valuable. Complements are welcome.
• Be patient. Sometimes the person who has the answer to your question is away from the computer. Usually you’ll get your problem solved quicker than you would calling some tech support line, but there are times it’s going to take a while. If you don’t get any response in a reasonable period of time (judged by how active the list or forum is, reasonable could be a couple hours or a couple days), there are several likely reasons: You haven’t been specific enough in your question; you’re in the wrong forum (e.g. users when it’s a developer question); nobody else is trying to do what you’re doing (in which case you may need to hire someone with the right skills); the project is dead (it happens sometimes–find another one); or the developers are swamped (give them more time, or come up with a new scenario that sheds light on your problem in a different way).

That list describes how we get open source support at Freelock. Aside from a couple unsupported hardware devices, or issues with proprietary programs, we have yet to get stumped, in over 6 years of extremely heavy Linux and open source use. We’ve never paid a dime for this support, though we have provided help to many others in return.

Patrick asks,

Why not OpenSuSE, instead of Ubuntu?

At Freelock, we provide a maintenance service contract to manage Linux servers. For a fixed monthly fee, we provide monitoring, system updates, application updates, and our help recovering anything that goes wrong with an upgrade. We’re looking at adding disaster recovery to the mix, raising the price to cover the cost of backing up all of the data and providing varying service level agreements on how soon we will recover your machine from a total loss. But for our base price, we only support Ubuntu and CentOS, with a preference for Ubuntu. So Patrick asks, why not OpenSuSE? Here’s my reply:

Hi, Patrick,

There are literally hundreds of different distributions of Linux, and 7 or 8 that are very widely deployed in server environments. The reasons Ubuntu is our preferred distribution include:

• No separate version of operating system–Red Hat and Novell/SuSE keep some of their stuff for their commercial version
• Commercial company backing it (Canonical, LTD), support contracts available
• Strong community support, lots of friendly help available
• “Long Term Support” releases, with a commitment by Canonical to maintain security releases for 5 years on designated versions
• Superior upgrade path when a version reaches end-of-life
• Nice balance between cutting-edge versions of new software, while making sure they’re stable
• Easy package management, well-built packages

OpenSuse has no commercial support available, and no commitment on the part of Novell to provide long term support to their free versions–you have to buy SuSE Enterprise Linux to get that level of stability/support, and that’s not the same software–they bundle older versions in their enterprise distributions that often don’t have features we’re coding on top of…

The upgrade path is another nice feature–we’ve had very good success upgrading Ubuntu boxes in place, without having to install an entirely new system and migrate the data over.

We support CentOS as well, which is a free version of Red Hat Enterprise Linux, because some of the software our clients run are only available for Red Hat, so we have to… CentOS doesn’t have as good an upgrade path as Ubuntu, so we don’t include system upgrades at software end-of-life with that. We have supported SuSE boxes in the past, but I think we’re down to a single one running some legacy software. The main reason we discourage it is to streamline our processes–it’s much easier to administer a bunch of the same operating system, than having every box be a one-off system.

We’ve been doing this long enough to have to upgrade several servers because the OS has reached end-of-life and there are no more security releases available. Of the free distributions, the only ones we can deploy with confidence knowing we won’t have to upgrade for at least 3 or 4 years are Ubuntu, CentOS, and Debian. While there are many other solid distribution choices you could make, none of the others quite stack up to meet our needs as well as Ubuntu and CentOS.

This is a new feature we’re starting: Ask Freelock! Have a question about using open source in business? Drop us a line, ask us a question. We’ll do our best to answer.

Happy New Year! Here’s a quick story about why Linux is the future:

Windows screwup forces Ubuntu shift

Good introductory article on monitoring disk usage, with a nice little script to send a mail as filesystems approach their limit:

System Administrators Toolkit: Monitoring disk space and usage
.

Here’s a handy article for setting up/configuring a monitoring service on a web server: BobCares :: Outsourced Web Hosting Support :: Installing and Configuring Nagios

Wow. Here’s a great, lengthy article detailing all sorts of things about using Linux and open source software in small businesses. Linux: A Buyer’s Guide for SMEs–ZDNet UK.

Just stumbled upon a pretty cool web interface to a centralized backup system. You can manage a series of snapshot backups of all the computers on your network. It can automatically send a user an email if there’s a repeated problem of missed backups. And users can restore their own backups through the web site. BackupPC: Open Source Backup to disk.

I found this while reading an article in Linux Journal that also has some helpful hints for setting up a Linux-based backup system: Build a Home Terabyte Backup System Using Linux.

There are a number of ways to integrate Linux clients and servers into an Active Directory environment. This article discusses a few of them.

Flavio’s TechnoTalk has some good tips for securing a Linux server. Good advice for anyone running servers that can be reached from the Internet: Hardening Linux: a 10 step approach to a secure server.

The Win-tel duopoly is about to become obsolete? An interesting editorial on the Enterprise Linux I.T. site claims that IBM and Sony are working together on a completely new processor architecture that’s going to deliver supercomputer power to a single chip, putting a grid network into your desktop box. And with its extreme adaptabillity to different computing architecture, it’s Linux that’ll be used there first.

According to the editorial by Paul Murphy, Microsoft is too dependent on the Intel architecture to be able to make the transition, and will soon come to be seen as stalling out.

Linux Reality Outstrips Linux Myth.

Great background information on what I always considered to be “Windows Networking”. The author of Samba writes about the Myths about Samba.

A couple of interesting stories here. First of all, it takes about 4 minutes for an unpatched Windows machine connected to the Internet without a firewall to be compromised. The typical Linux machine takes more like 3 months, according to this VNU Net story: Linux fights off hackers.

A four year analysis of the Linux source code conducted by five Stanford University computer science researchers found a tiny fraction of the number of security holes and bugs, compared to similar studies of proprietary software. Read the Wired story, Linux: Fewer Bugs Than Rivals.

Novell has launched a marketing effort to counter the Microsoft “Get the Facts” campaign, spinning the studies to show a favorable total cost of ownership of using Linux. But we knew that already! NOVELL: Unbending the Truth

Books

Linux For Dummies, by Dee-Ann LeBlanc, John Wiley & Sons, 5th Edition, 2003
Red Hat Linux 9, by Michael Jang, Sybex, 2003
Running Linux, by Matt Welsh, Matthias Kalle Alheimer, and Lar Kaufman, O’Reilly & Associates, 4th edition, 2002

Software

Mandrake Linux

Red Hat Linux
The Fedora Project
SuSE Linux
Debian Linux
Microsoft Windows
Apple Mac OS X
Sun Solaris
Advanced Extranet Server
Shorewall Firewall
Prelude Intrusion Detection System
Webmin

Web Sites

• Easy URPMI, configure media sources for Mandrake