A colleague who sounds a bit upset at last week’s results twittered a link to a blog post that accuses us of being “sheeple”, and going down the path of socialism, and apparently the author believes this will cause our country to collapse. The tail end of the McCain/Palin campaign hurled the Socialism epithet as well. Other colleagues have been sending cartoons of trick-or-treaters collecting candy on behalf of their friends who are “too lazy” to trick-or-treat.

There must be some fundamental difference in the way people see the world. I hear these things, and several thoughts come to mind: (warning: long, rambling unedited rant ahead)

• We’re facing some awfully large problems. Sub-par health-care system. Failing education system. Crumbling transportation infrastructure built around gas-guzzling cars as the cost of oil goes up. Failure of the free market to regulate itself. It strikes me that these are all problems we need to solve together, not leave to the laissez-faire system we’ve had that have rigged the playing field towards the rich, redistributing our country’s wealth to the very top.
• Why does the word Socialism have such negative connotations for the right wing? Is it just because it was part of the USSR’s name? Most of Europe has socialist programs in place, resulting in much better health coverage and a social safety net to help reduce homelessness.
• The word “lazy” is used as a broad brush to dismiss the efforts of anybody who hasn’t reached some level of success, often by people who are struggling themselves. I’d argue that people at the bottom of the pay scale, working 2 minimum-wage jobs and barely scraping by, are anything but lazy. Opportunity is capricious, and not evenly distributed.
• Open Source provides a proof-of-concept that illustrates that we’re better off working together to solve our problems, rather than keeping our solutions close to our chest and not sharing with our neighbors.

Us versus them

Our country was founded with a common enemy. For many people, it seems that patriotism depends upon having an external enemy. First the Brits. More recently, the Nazis and Japan, and then the Soviet Union. Now, it’s terrorism, along with illegal immigrants.

But Franklin Roosevelt, at his inauguration in the midst of the Great Depression, famously said “the only thing we have to fear is fear itself—nameless, unreasoning, unjustified terror which paralyzes needed efforts to convert retreat into advance.” And indeed, that is exactly what we fear today–terror. We’ve declared war on it, to the detriment of our civil rights and our moral standing in the world.

I think many people have transferred their fear and uncertainty of their own economic circumstances to this external threat of terrorism. But our real enemy, both here and abroad, is poverty and an unwillingness to work together in the face of much larger looming disasters, such as peak oil, climate change, water and energy shortages, overpopulation, mass extinctions. Instead of dealing with these issues, we’re blaming others for our problems.

I fail to see how the problems we face in the world are all because of terrorists, Democrats, or lazy people. I would argue we have terrorists because we have poverty, and hugely imbalanced distribution of wealth. If we don’t deal with those issues, we’ll surely see more terrorism in the future.

The individualist end game

Time is running out on many fronts. The sheer size of our population is putting a strain on the natural systems of our planet on which we’re completely dependent. Yet we squabble on how to divide up the planet, and those who play the dirtiest have long been winning.

In economics, there’s a distinction made between rivalrous and non-rivalrous resources. Rivalrous resources are those that can only be used by one person at any given time. Non-rivalrous resources are generally things that can be used by many at the same time. For example, my backpack is rivalrous–if you take it from me, I don’t have it anymore. A song is non-rivalrous–I can sing “Happy Birthday” without taking that away from you. Or at least, I should be able to…

In our current economic system, we have made more and more things rivalrous, at least in our legal system. We have carved up our land and made the vast majority of it private. We have granted patents on our very genes. We have allowed corporations to claim ownership of drinking water and charge people for access. We have companies whose sole purpose is to own a patent portfolio and make money off the use of ideas, whether or not other people came up with the same idea independently.

If you’re an individualist, you might see these things as capitalism at work, and the way it should be. But this is really a short-sighted view. Taken to the ultimate result of more and more people fighting over less and less, and you end up with situations like Rwanda in the 90s. According to Jared Diamond, one of the biggest factors leading up to the genocide was population-pressure–the population density had reached a level not seen anywhere else in Africa. Sons were fighting over postage-stamp lots of land to have something they could grow food on, as fathers divided the land into ever smaller bits. There were no shared spaces left, no mental buffers, not enough land to support the people. The rate we’re exhausting our planet’s resources, we risk a similar fate.

We’ve been steadily turning our common wealth into private wealth, redistributing wealth to a tiny few, leaving not enough for everybody else. We’ve been spending resources like oil that have taken millions of years to accumulate in a few generations, polluting our seas and soils, shipping our waste anywhere out of sight, and seeing all of this as somebody else’s problems.

Our country was founded on the principles of working together in the face of a common enemy. Ben Franklin is full of relevant quotes: “We must hang together, gentlemen…else, we shall most assuredly hang separately.” The biggest problem is that while these big companies take, take, take from our environment and people all to maximize their own profits, they’re stealing the resources everybody else will need in the long run, and socializing the true costs of them doing business.

Scarcity versus abundance

Ironically, the way out of this mess is to change perspectives and realize we really are blessed with tremendous resources. Among the largest, most under-utilized resource we’ve got are millions of smart people locked up doing menial work as cogs in the machinery of these big corporations. We’ve largely moved slavery out of agriculture into minimum-wage jobs across the country, and lower-wage jobs abroad. The term “wage slave” is not far off the mark. Prices of everything has been spiraling up while wages have flat-lined. In the past generation, the cost of buying a house has gone up by a factor of ten, while wages have doubled. Yet more people have bought houses than ever before, and we now see that it was mostly on borrowed money they couldn’t possibly pay back.

You hear the right wing complaining about big government, and how liberal policies reward people for not working. Our policies say a lot about what we value, and just about every decision congress makes benefits some people while penalizing others. We need to get a lot more strategic about our policy-making, and align those policies to address the big problems we’re facing–a health care crisis, global warming, deforestation, peak oil, the financial crisis, and more. If our most abundant under-utilized resource is brainpower, let’s make some policy changes that rewards using those resources, while penalizing the use of the natural resources we’re quickly exhausting.

We need to find ways of penalizing companies that consume natural resources and burden our systems with waste. We need to find ways of rewarding smart, responsible people and companies who help reduce waste in the system.

So how do we make this happen?

Let’s go back to the basics. What does everybody need? Food and shelter, obviously. Today, health-care needs to be a right–we’re a wealthy enough country that it’s absurd people have to choose between food and medication they need to treat and prevent a health issue. Beyond that, people need opportunity, and motivation. For far too long, our policy has been to motivate through fear. Make people afraid they won’t have enough food or have a place to live, and they’ll do drudgery work to avoid that. But it’s hard to give creativity free reign when you’re trying to scrape together the basic things you need to survive. Too many of our people are in exactly that position. And we need their help solving the bigger problems.

In fact, some of our biggest problems are caused by exactly this issue–people not having adequate food or shelter. Terrorists come from places where the future is bleak, where people aren’t secure that they’ll have a safe place to live and enough to eat, let alone make any positive contributions. But there’s always an opportunity to be destructive if your situation is dire.

Now, as much as I’ve been railing against big business, I do think business is the answer. We need to re-align our policies to favor small businesses, and lots of them, to power our way through this mess. Small businesses need to be part of their communities–they can’t just take their jobs to the lowest bidder. Small businesses are much more entrenched in location, and able to make contributions to help their communities. Small businesses provide jobs, solve multiple problems, and are able to think about things beyond merely satisfying their shareholders’ greed–they are able to optimize their business to fit in their part of the economic ecosystem, rather than doing whatever it takes to maximize profits. And small businesses succeed by recognizing that others need to succeed too.

The current barriers to people creating small entrepreneurial businesses that solve real problems are many:

• A workforce trained to be wage slaves, rather than thinking, and making decisions, for themselves.
• Crushing costs of doing business: payroll taxes, health care, business operational overhead that is a high cost of doing business.
• Lack of knowledge/experience in creating a business. Very few people who start a business have done it before, and until you’ve started a business, it’s hard to understand the magnitude of the task.
• Money. It takes money to get started. It takes money to hire people, provide training, lease space, buy equipment, get insurance, handle bookkeeping and taxes, and obtain licenses, let alone buy the raw materials that become your product.
• Time management. If you’re a service business, you need to have enough billable hours to cover your costs and make a profit. If you’re a product business, you need to create or manage your products. You need to devote some of your time to marketing and sales before you make any money at all. You need to divide your time effectively between doing the stuff you get paid for, letting the world know you exist, developing the key relationships with partners and vendors and customers, and handling all the little stuff that needs to get done to keep the business running.

The bottom line is, nobody can do all of this alone. To succeed in business, you need help, and lots of it. You need customers who will spread the good word. You need vendors to help with all the operational aspects of your business. You need employees to do the actual work of your business. You need competitors to prove to the marketplace that your product or service is valuable.

So that’s a very long-winded way to get to the point that the vast majority of businesses do not have a monopoly on their product, and do not need “Intellectual Property” (IP) protection to be successful.

Venture Capitalists are always looking for IP: patents, copyrights, trade secrets, or some element that gives a company an advantage over every other potential competitor in the market. They need this because the venture system is based on home runs–they expect that out of 10 businesses, 7 will fail, 2 may break even, and 1 will be successful enough to cover the losses of the other businesses. They don’t know which business will be that home run, so they’re going to invest in the businesses they think stand the best chance.

I’ve spoken with hundreds of people who think their idea is unique, and they don’t want to tell you about it because you might steal it and make a ton of money–or at least be a competitor. The problem is, there are lots of good ideas, and many people come up with very similar ideas independently–the real shortage is the talent to execute those ideas, turn them into a viable business.

One of my biggest challenges as an early open source solution provider was having my potential customers take my solutions seriously. After all, if the software is free, how can it be any good? And if it avoids vendor lock-in, why am I the only one proposing it while they’ve got a dozen vendors pitching Microsoft or Intuit solutions?

While the idea behind my business is wide open, and there’s no prohibitive barrier to competitors setting up shop and doing exactly the same thing, our business is succeeding on execution, on the talent of my employees. It doesn’t matter what your idea is–what matters is whether you can bring it to market.

Systems thinking

To wrap this long meandering post up, we can no longer afford to run businesses that maximize a single factor (profit) at the expense of everything else (people, environment, waste, the future). We need to start optimizing for all critical factors. Our products and businesses need to account for the full impact of what we’re doing, and if it doesn’t make the world a better place in some small way, it shouldn’t continue to exist.

Businesses don’t exist in a vacuum. Businesses can serve a highly constructive role in our society, and help address all of the major challenges we face. But we need to adopt the Unix architecture, create them as small pieces loosely joined, not big monolithic monsters that crush everything in their path.

First of all, the Mac Break Weekly show apparently spends some time bashing the open source community, calling out Drupal, and how difficult it is to solve “simple” problems like uploading images for blog posts. In practically the same breath, the hosts claim that the open source community never has any innovation behind it. Irony drips:

• In the world of content management systems (CMSs), most of the innovation starts in open source projects these days, and Drupal is at the cutting edge of this with its powerful system of taxonomies, and hundreds of add-ons freely available.
• I can think of a grand total of 2 proprietary CMSs that have anywhere near as widespread use as most of the open source CMS systems. One has turned open source itself: Movable Type. The other is Sharepoint, and it’s widespread because Microsoft has shipped it out on lots of its server products.
• Complain about usability all you want, but name a proprietary product as powerful as Drupal, that’s easier to install, administer, and configure.
• The TWIT.tv site itself is running on Drupal.

So let’s talk a little about innovation. While Photoshop may still dominate the world of graphic design, but the lines aren’t so clear when it comes to animation. The Blender project recently released its second short animated film, Big Buck Bunny. While you might argue about the strength of its story, you cannot deny that the technical effects are as stunning as any major animated film coming out from the big studios. And it was created by 7 people in 9 months, using open source software. Even the big studios like Pixar, Dreamworks, and Industrial Light and Magic rely on open source software to deliver their magic, such as CinePaint, POVray, and several others.

On the subject of innovation, KDE4 is breaking new ground and stirring up controversy, laying a bedrock that promises the ability to do things beyond the standard “Desktop” paradigm that was invented over 30 years ago and we’ve all used ever since. Meanwhile, the GNOME team is working on creative ways to embed web applications into your desktop.

But the real innovations of the open source community are all a few layers deeper in the application stack, all the plumbing that powers the Internet. Microsoft itself borrowed its early networking stack from BSD, one of the earliest open source operating systems. Domain Name Service (DNS) and email were first implemented on the Internet using open source software (BIND and Sendmail).

The open source community tends to snicker whenever Apple claims to be innovative. Its core “innovations” were all invented somewhere else:

• The Mac Desktop interface borrowed heavily from Xerox PARC labs
• OS X uses BSD under the hood
• “Spaces” were in use in Unix systems for a decade before they arrived on the Mac
• The “Time Machine” functionality in Leopard is standard in many source code management tools

To its credit, Apple polishes these features better than anybody else, making them easier to find and use by normal people. But many, if not most of its innovations come from somebody else.

Read the previous post for more discussion about Linux support.

Dann and Linc had a quite spirited debate about the merits of having a company hire low-end tech support with scripts (Dann) versus having an experienced, savvy, tech professional able to really solve the user’s problem (Linc). Dann’s point was that it can easily be more cost-effective for both the support company and the end user to go straight to reinstalling a system if rebooting doesn’t solve the user’s problem, while Linc seemed to think a savvy tech person could get to the root of the issue much quicker, and brought up the point that there’s a cost to the frustration of users being put through the whole front-line support nightmare.

I’d suggest the situation is even more complex than that, but it also differs greatly between the open source projects and proprietary operating systems. First, let me make a bold statement:

Linux support is far better than Windows or Mac.

But it also has a completely different set of rules. Learn those rules, and you’ll be able to solve your problems more satisfactorily than it’s possible to do in the proprietary world. Let’s talk about these differences, looking specifically at a who, where, what, and how long.

Who can help with your problem?

Let’s take a look at who can help with your problem:

The bottom line here is that while you probably know fewer people personally who can help you with Linux, there are more options for commercial support, and you can reach people who can do more to solve your problem for free, than you can with either Mac or Windows.

The fundamental reason for this is that anybody can read the raw source code of any open source product out there, and with enough skill and talent, can solve your problem without needing to pay anybody for the right to do so. In the proprietary world, only one company can help beyond a certain point: for Windows problems, that’s Microsoft. For problems in an application, it’s the application developer.

So the next question is:

Where can you get help?
Again, let’s compare the options:

The main point of the table above is that in the proprietary world, the harder your problem is to solve, the fewer people can help you solve it. You quickly get down to one place to go, and if it’s in the operating system, it might be expensive or not possible to fix. In the open source world, it’s nearly the opposite case–it can be harder to find the simple quick answer to your question, but the harder and deeper your problem, the more places you can go to get help.

Where do you look for help first? This is the single stumbling block for most otherwise tech-savvy users new to Linux. To learn Windows or Mac you can take a class, talk to neighbors and friends, and find lots of very low-end help that way. For Linux, unless you’re friends with some hard-core geeks, you need to go online to find help. Once you’re there, it really depends on where your problem lies.

For just figuring out how to use a system, go to the forums for your distribution. The Ubuntu forums are a great place to ask general questions. Be specific about what you’re trying to do and you’re more likely to get help. Remember that this level of support is free, and people helping you are volunteering their time and knowledge. While you’re there, see if you can answer somebody else’s question–the more help you give, the more you’ll get in return.

The other fantastic place to go for help, especially for quick questions, is IRC. IRC is a system that provides chat rooms and instant messaging. Most open source projects with any community behind them have a chat room on irc.freenode.net. Install an IRC program like Chatzilla, Konversation, Pidgin, or any number of others, connect to irc.freenode.net, pick a nickname to use, and join the channel for the program or distribution you’re having trouble with. Ask your question nicely, and if you don’t get a response immediately, keep your chat program open for a few hours–not everybody is watching the channel every minute.

What kinds of problems can you solve?
Let’s look at the same types of problems as before, and look at the resolutions:

What’s the key thing in this section? Addressing actual problems is within your control, when you’re working with open source. In the proprietary world, you’re entirely at the mercy of a single software vendor. If your problem is in the interaction between two different applications, you’re really stuck — there’s nothing you can do.

But in the open source world, there’s always something you can do. You can hire anybody with the skills to solve your problem, and fix it in the software itself. You don’t need any blessing from any single software vendor.

Open formats are perhaps more valuable than open source software, for most businesses. Because this is such a compelling advantage of open source software, many proprietary programs are beginning to open up their formats to allow other software to read them–their customers are demanding it.

How long will it take to solve your problem?
I’m not going to bother with a table for this one. The answer is nearly always “too long,” regardless of the operating system.

Actually, that’s not quite always the case–it depends on whether somebody has already solved the problem or not, as well as whether the solution is a fix or a workaround.

Many, many problems in Windows are not really fixes, they’re just workarounds. The only real “fix” for a virus-infected machine is the workaround of reinstalling your operating system. The only fix for lots of other minor issues that cause your system to slow down over time is to reboot. These are not fixes, they’re workarounds.

Real fixes take a lot longer, and need acknowledgment that the problem is real. Workarounds are band aids to get you through until there’s a real fix. And there are some real differences between the entire approaches of each operating system around these fixes.

Windows is chock-full of workarounds. Because Microsoft has gone to great lengths to maintain backwards compatibility of just about everything that’s been released for Windows since Windows 95, it’s full of workarounds to keep the old behavior. Rather than fixing behavior that might really be undesirable, they’ve had to patch it with workarounds because too many existing applications turned out to depend on that bad behavior. That, fundamentally, is why Windows is so big, bloated, slow, and painful to work with.

Apple suffers a different problem: changing their closed libraries too quickly for external developers to keep up. Each release, they break lots of things, and don’t always tell 3rd party developers ahead of time. This means the number of third party developers of Apple software is shrinking–they’ve managed to alienate quite a few. So their polish and high quality comes at the price of having a healthy thriving developer community outside the walls of Apple. There’s little transparency in this process, so developers outside Apple are always playing catch-up, and having to work around these changes in behavior.

Linux has its share of problems, too. Linux does not attempt to maintain “binary” compatibility between versions, though it does its best to maintain compatibility in source code. There is endless debate about the “correct” way to fix a problem, and competition between fixes. The challenge of this is that it can be hard for application developers to keep up, especially if they want to keep their source code closed and only ship software in binary form.

But the process is completely, utterly transparent. Anybody can see the progress on any fixes to any part of the system, and can jump in with their own solution at any time. It’s a true meritocracy, with those doing the actual work and providing the best solutions winning out over time.

The whole open source ecosystem is nimble enough to provide real fixes to technical problems, rather than just simple workarounds. If the source of the problem is design, it can take a long time to get the right design in place and resolve all the issues that changing the design causes in other applications. But when a bug gets fixed, it’s really fixed and usually doesn’t appear again.

I posted my results, and confusion, to the Dspam mailing list. The original developer of this software (which has thousands of users), Jonathan Zdziarski, responded that that did not sound right. He asked if he could log into my server and see what was wrong.

I created a test account for him, and logged in to the same screen so I could watch what he was doing. As I watched, he put debugging messages into his code, ran several tests, and within 10 minutes had identified the problem: the 19-digit number he put into the database (MySQL) was different than the 19-digit number he got out. It was a storage error in somebody else’s software causing the problem, a rounding error. He filed a bug in MySQL, and it was fixed within a month or two. But he also had a workaround for me: change the data type to a set of characters instead of an integer. Problem solved.

• Before asking anybody, do your homework. Use Google, read the project FAQ, make some attempt to learn the basics without pestering people with questions they’ve already answered hundreds of times. Nine times out of ten, your problem has already been encountered and somebody has figured out a workaround.
• Limit the scope of your question to the fundamental problem. Get to the point. I’m obviously guilty of being tremendously long-winded at times, but unless your question is right at the top and asked directly, you’ll probably get ignored. Developers are busy, they don’t want to read a novel, but they’re usually happy to answer a question.
• Provide supporting details after asking your question. Many programs will create a log with lots of information that can help somebody diagnose your problem. Find what looks relevant in the logs. Specify what version of operating system, distribution, application, etc. Specify what you were trying to do, what you expected, and what really happened. But provide this stuff after asking your initial question–people aren’t going to wade through a long email to find your question.
• Contribute something. The easiest way you can contribute is by answering other people’s questions. The whole thing works because people help each other, and this help goes both ways. If you always ask questions and never answer anybody else’s, or provide any other sort of contribution, you’ll eventually start getting ignored.
• Always, always, be positive, respectful, and polite when asking your questions. Developers have a lot invested in their project, and insulting it won’t gain you any favors. Developers are under no obligation to help you either–you haven’t paid for it. Common courtesy is valuable. Complements are welcome.
• Be patient. Sometimes the person who has the answer to your question is away from the computer. Usually you’ll get your problem solved quicker than you would calling some tech support line, but there are times it’s going to take a while. If you don’t get any response in a reasonable period of time (judged by how active the list or forum is, reasonable could be a couple hours or a couple days), there are several likely reasons: You haven’t been specific enough in your question; you’re in the wrong forum (e.g. users when it’s a developer question); nobody else is trying to do what you’re doing (in which case you may need to hire someone with the right skills); the project is dead (it happens sometimes–find another one); or the developers are swamped (give them more time, or come up with a new scenario that sheds light on your problem in a different way).

That list describes how we get open source support at Freelock. Aside from a couple unsupported hardware devices, or issues with proprietary programs, we have yet to get stumped, in over 6 years of extremely heavy Linux and open source use. We’ve never paid a dime for this support, though we have provided help to many others in return.

For a long time, we’ve managed our custom code projects and business documents in a central repository, called Subversion (also known as svn). Subversion is relatively easy to understand–it’s like having a library of files you can check a copy out of, do some work on it, and then check it back in. Subversion is the librarian that tracks who has copies of what, and when they checked it out. So if Erik checks in changes to a brochure, and then Jill goes to submit changes to the same document, Subversion will say “hey wait a minute, that document has already been changed–you need to make sure you put Erik’s changes in your document before I’ll let you put in your document.”

This is great for managing conflicts between people working on a single team, or for code that is being developed in relative isolation from the rest of the world.

The problem is, we’re doing more than that–we’re taking code from various open source projects and either customizing it or building new applications on top of it. And so when the outside projects get updated, we need to manually update anything we’ve written that depends on that code. There is no longer a single repository where we control our code–there is our code library, plus another one for every project we use.

This makes managing add-ons for projects like Joomla or ZenCart quite challenging, because our add-ons get scattered throughout the filesystem to be able to hook into the right place. And if we have to touch a core file, we’re going to end up needing to re-implement our change with any update to that core file.

There are other issues we run into, managing our code and hosting, all of which take fairly time-consuming, manual intervention. Here’s the list:

• Since we host and provide security updates for Joomla, Word Press, Zen Cart, Drupal, and others, we need to upgrade dozens of installations any time there’s a new release that has a fix for a security vulnerability. With Joomla this has happened quite a lot, and every Joomla installation needs to be upgraded individually–and tested. And since each installation is slightly different, we can’t manage them easily within a single repository, while updating the underlying code.
• Templates, modules, components, blocks, themes, plugins, and whatever. Developing these types of add-ons are our bread-and-butter. But code for these often get scattered across an installation, making it quite difficult to manage just our add-ons while we develop them, or roll back to earlier versions if there’s a problem.
• The Dojo Toolkit, and builds. We’re doing a lot of development with Dojo right now, to add desktop-like functionality such as trees, sortable tables, right-click menus, animations, and lots of other really cool things. However, if you don’t “build” the code after you write it, it’s painfully slow in a web browser. And due to the nature of how Subversion works, you can’t easily store a built Dojo tree if you ever want to change it again. Which means you’d need to build it every place you deploy it. And on some computers, it can take a long time to build–on our demo server, one of our projects currently takes 8 minutes.
• As we get more directly involved with open source projects like LedgerSMB, we’re finding the need to change core files while we hack away at some particular feature. To do this, you create a branch of the code, work on your feature, and then merge your changes back into the “trunk.” If you don’t have access to save directly to the project repository, doing this gets a lot more complicated.

Git to the rescue. Git solves all of these issues. Read on for a technical discussion of how.

Managing lots of installations with Git
We haven’t yet started doing this, but for projects like Joomla and ZenCart, this is going to be really effective. It’s not quite as helpful for Drupal, which allows for a bunch of sites to be centrally managed out of the box–though even there it will help us recover quickly from a failed upgrade.

Here’s the basic approach:

1. Create a master git repository, either by pulling down the Subversion repository using git svn, or just unpacking tarballs of a release and importing into our git repository for the project.
2. Create a new git clone for each installation, and drop into each installation directory. Create a current branch for that repository, and rebase it to the current installed version.
3. Commit local modifications to the repository. Make sure the .git directory is getting backed up, and is not accessible through the web server.
4. When a new release is available, use git pull to update each local installation. Use git reset HEAD^ –hard to undo if there’s a problem.

Managing submodules, customizations
This we get pretty much for free by having dedicated repositories for particular problems. Managing add-ons we’d like to use elsewhere can still be challenging, but we can pull customizations from other repositories to assemble custom packages–it’s at least easier to manage these across installations.

The main thing to do is to make sure when you’re developing on a particular add-on, you’re always doing it in a branch dedicated to those changes. Then you can merge them into other projects more easily. It’s also possible to “cherry-pick” commits involving particular files, and pull those changes into a new branch if development ended up getting mixed in with different features.

Dojo Toolkit builds
We’re huge fans of the Dojo Toolkit. It makes developing really sophisticated browser-based applications much easier, not just providing a large set of active widgets but also a data abstraction layer and an encapsulation system for your own code.

Its biggest downside is that every feature you want to use means adding another file that the browser needs to load. And we use a lot of features… the base package loads about 20 files, and in our applications there are well over a hundred. This takes a long time to load, and isn’t very scalable.

The solution? Use the Dojo build process. A Java program can build all the features you want to use into a single compressed, stripped, javascript file. The difference in speed is incredible–applications suddenly load right up. And this build system can easily build your Javascript as well.

The problem is, this build system is not compatible with Subversion, because Subversion scatters its working copy data inside each directory, but the build system deletes the directories every time you build. So you can’t easily manage built code inside Subversion.

Git stores all of its repository information in the top level directory, so you can easily build and re-build Dojo and manage it without any issues.

Do local development on open source projects
Now here’s the traditional reason to use git: to allow you to manage changes to open source projects without needing write access to their code repositories. We simply set up git repositories to track their Subversion repository, and then we can use all the features of git to manage our customizations.

When the original project has new releases, we just pull them into our repository and git handles most of the merging for us. If we want to contribute back to the project, we can copy them into our Subversion tracking branch and commit them straight back to Subversion–or send a patch.

We’re starting to set up public git repositories for projects we work with extensively. You can browse them here: http://git.freelock.com. You can clone and pull updates from here: git://git.freelock.com/git/<projectname>.git. You can read more about how to manage non-standard Subversion repositories in git here.

If your reason for purchasing Office 2007 is #11, drop us an email and ask us how open source software can make your business run better.

This sounds quite similar to the challenges a small business faces. “The E-Myth Revisited,” by Michael Gerber, is essential reading for anyone wanting to start a small business, and some of the same ideas apply to building a successful open source project.

Basically, Gerber says you need to have 3 personalities in your business:

• The entrepreneur, the person with the vision to drive the business/project forward, always a step ahead working on what’s next
• The manager, somebody to make sure all the work gets done on schedule, delivered on time, and that everybody is working in the same direction with the same priorities
• The technician, who actually does the work

And small businesses, like open source projects, are almost always started by technicians, people who just know they can do a better job than anybody else, so they set out to do it themselves. The reason why so many businesses fail is because the founders spend all their time building something without setting enough overall direction or managing their cash flow. Sound familiar?

The key to creating both a successful business and an open source project is balancing out these 3 personalities and making sure all are represented to an appropriate degree.

Todd over at Napera asks,

Certainly the terms Microsoft has been offering companies since the EU decision in October 2007 are extremely reasonable. Given Microsoft’s new open protocol documentation and their patent pledge for open source developers, what’s not to like?

I haven’t seen any pundits or commentators in the US defending the EU decision. If they are, what are their substantive points?
Napera Networks » The EU crashes Microsoft’s party

Hmm. Let me see if I can shed some light…

First of all, Microsoft wants you to believe that open source developers are all a bunch of hobbyists creating code for no pay. Their recent pledge is to not sue open source developers developing non-commercial systems.

Boy, what a deal that is… Microsoft gets all this open source development for free, right? And they can still charge anybody who dares to compete with them commercially? Great deal–for Microsoft.

The reality is, a great number of open source developers engage in commercial activity. Many successful open source projects are backed by commercial companies. OpenOffice.org, JBoss, SugarCRM, MySQL, PHP, the list goes on. By definition, open source means you can use and redistribute software for any reason, including commercial activity–by excluding commercial entities from patent protection, Microsoft is making a marketing play without making any real concession.

Their real agenda here is to give the illusion of openness so they can get their OOXML format approved as an open standard, without giving up any control over the format. Never mind that they were part of the committee developing the ODF standard and didn’t bother to contribute. Microsoft has no interest in working with other companies to develop an open standard–they want to be proclaimed the standard so they can demand “reasonable royalties” from everyone else, and all the rest of this is just marketing spin to try to put one past the EU, who so far have managed to see through all this bull.

Secondly, Microsoft seems to think the only healthy software marketplace is one that revolves around it. It got where it is by using all sorts of unethical, hardball business practices to drive its competition out of business. This is classic anti-competitive behavior, and is the reason it’s been the target of so many actions.

Microsoft has built a large ecosystem of developers and service providers all gathered around the MS stack. Yet you look at the stack and there’s not all that much vibrant activity, at least not compared to the open source bazaar. Take content management systems as an example. In the Microsoft world, you’ve got Sharepoint. You’ve got a couple of ASP.net open source projects like Dot Net Nuke. And that’s about it.

On the open source side, you’ve got overwhelming choice, and some of them quite great. Instead of one size fits all, you can go shopping for just the features you need, and get it all for a great price–free, if you have the technical ability to make them work. Joomla, Drupal, Plone, MediaWiki, Typo3, Postnuke, Word Press, Serendipity, several hundred different options for you to choose from. Which looks like a more vibrant marketplace, with more competition and more choice?

Ah, but then you ask about standards–isn’t it great that Microsoft makes it so simple for you to do what you need to do? You don’t have to make all those choices. Soviet bakeries were so much better than American supermarkets, too, huh?

Oh, but wait. Who invented http, rss, the world wide web, email, the spreadsheet, the database, the word processor, or anything else we use our computers for? Hint: not Microsoft. And the first few things in that list were created in universities and other open source, collaborative environments.

Finally, to speak directly to the EU’s decision, how does letting Microsoft get away with their monopolistic, anti-competitive behavior help the EU in any way? By protecting their software industry from Microsoft, they’re fostering an environment that can lead to a much more competitive, fertile ground to grow their own software industry.

No other industry (other than possibly the utility companies and the railroads) have had a single company that so dominates the industry like Microsoft dominates software. How can that be a good thing? I know from personal experience a half a dozen companies that Microsoft crushed, and for the most part their technology with them. This mono-culture of software has led to all the trouble with spyware and botnets and through them, spam. I don’t think we’re better off with our Redmond overlords.

I for one was happy to hear about the EU’s decision, glad they were able to stand up against monopolistic practices and do something to actually help their software industry thrive.

My coding weakness, it turns out, is reliability. I’m terrible at handling errors, building test frameworks, doing unit testing. I find all of that stuff quite boring. But it’s essential to building a reliable application.

Reliability and security go hand in hand. In security, you’re looking at the attacks, and making sure your code is secure against them. In reliability, you’re identifying what each chunk of code expects to get, and then define how to handle exceptions, unexpected input. Done correctly, reliable code is secure. But it’s a total pain to do, and it takes a lot longer to get there.

One of the code samples I examined recently was set up in a completely class-driven way, though I would not call it object oriented because none of the classes extended other classes. It was a rather simple, flat collection of objects and helpers and interfaces. It was not powerful. My guess is, it is not fast. It did not look very customizable. But it was certainly clear, and every single method inspected every single parameter, making sure the input was valid. Calls to other objects had extensive error handling built-in — this application looked like it could not fail without notifying the programmer exactly where the failure was, with helpful feedback.

This is tedious work. I save it for the polishing phases of a project, focusing on getting things to work in the first place. But there’s a strong argument to be made for building reliability into each module from the start. It’s a very different style of programming, and takes a lot longer to get there, but the end result will inevitably be more secure, less buggy, and more able to account for every possible scenario–even if it handles a scenario by saying “I can’t do that yet.”

I think there’s a personality difference between these development styles. The artist figures out some innovative way of solving the problem, gets a proof-of-concept working brilliantly quickly, and cranks through code producing a huge amount in a short amount of time. The craftsman takes a slower, methodical approach, crafting each module individually, building unit tests to make sure it works correctly as he goes, and building a system piece by polished piece.

Successful projects need both. The artist/hacker provides vision, drive, and momentum. The craftsman makes sure the system can handle the load, and can prove it’s doing what it’s designed to do.

The 80/20 rule comes into play here. 80% of the features can be hacked together very quickly, in the first 20% of the project. To make the project stand the tests of time, handle everything that might be thrown at it, and act as a foundation for a business or a mission-critical part, you need the craftsman to do the remaining 80% of the work to finish the job and get that final 20% of the functionality complete.

So here’s a checklist for evaluating reliability of a project:

• Is the program broken up into discrete modules that can be completely tested one at a time?
• Are there unit tests built for each module, testing the output for normal and exceptional conditions?
• Is the input to each module validated and properly tested to handle all possible things that may be passed to it?
• Does the module handle non-normal input, and raise the appropriate errors?
• Are there regular tests of the software as a whole, and each module, to identify tests that fail, or regressions in the code?

The only way to ensure reliability is through rigorous testing. Some of the newer programming practices rely on test-driven development–first you define what a module does, then you develop a test for it, and then only after all that do you finally develop the module until it passes all the tests.

In a small business environment, this all may be too much overhead. 80% of an application may be enough, and at 20% of the cost, much more inline with the budget. But when you need something to be completely reliable, take a look at the testing framework, how much it covers, and how much of the application passes the tests.