While they’re fresh, let me jot them down here. WARNING: Extremely technical content ahead.

First of all, SOAP is supposed to stand for “Simple Object Access Protocol.” It’s anything but simple. There is a lot of SOAP software out there, but subtle implementation gotchas that can be quite difficult to figure out.

We chose the native PHP SoapServer in PHP 5.2 to implement the project, mainly because we’re a PHP shop, and a little smoke testing revealed it was quite quick to get set up and going. It turns out that it’s quite hard to debug. For its good points, it can read in a WSDL and automatically map methods to methods on a class, and it converts arrays, simple objects, or complex objects to a valid response object, and request objects into simple or complex objects on the incoming side.

Problems with PHP’s SOAP Server:

• No validation of incoming or outgoing documents.
• No warnings, exceptions, or errors if it can’t convert a document to fit the schema–it just dies.
• No debugging information about what it’s doing.
• No ability to manage namespaces, especially if they need to be copied from the SOAP envelope into the payload.
• Difficult to test.
• No access to the raw XML of either the request or the response.

Using PHP’s SoapServer is quite simple, except when things aren’t perfect…

Here’s what the code looks like for the simple case:


< ?php
$xml = $GLOBALS['HTTP_RAW_POST_DATA']; // or file_get_contents('php://input');
// make sure you have something to process, throw an error if $xml is empty
$soap = new SoapServer('http://path/to/your.wsdl');
$soap->setClass('mySoapClass');
$soap->handle($xml);
?>

That’s basically it. You declare methods on ‘mySoapClass’ that correspond to the SOAP methods. These handler methods receive a simple object as a parameter, and you can do whatever you need to do with that data. Then it needs to return some data structure that can be serialized to the expected type defined in the WSDL. The return data structure can be an array, a simple object, or an object of a class you define that can serialize appropriately.

Great. With this much, it took me about a day to have a working web service with 8 methods and a bunch of complex data objects. The problems started when people connected with different SOAP software.

The web service I was implementing defines a specific SOAP Fault document, so if I did run across a problem, I could simply throw an exception of that type. My wrapper object kindly passed the custom fields defined in the WSDL.

Problem #1: Validation

As I said before, there is none. If the SoapServer gets anything it doesn’t like, it doesn’t send any response at all. And since all you get inside your method handlers is an already-converted object, you don’t have any way to validate the response without using a global variable or a call to a singleton.

In our case, the project specified that we strip out the payload from the SOAP header and store the payload XML on the disk as a document for several methods, and do processing on other methods. Processing a SOAP request was not a problem. Storing a valid XML document was. Several methods just stored the XML on the disk, with another method retrieving it and returning it to the caller. The problem we had was that the Soap Response was more picky than the Soap Request–so documents that we loaded from the disk and returned as the response would fail with no explanation.

Our solution was to load the raw XML into a DOM Document, and validate it against the schema. This mostly worked, until we had to deal with a request generated from Jitterbit. More on that later.

The question is, what to validate? We weren’t supposed to store the entire SOAP envelope–just the payload. So how to extract it? The simple way was to grab the first child of the Body element, append it to the DOMDocument itself, remove the original root, and call the normalize() method. This did generate a warning, but not a fatal error, and did the right thing. Furthermore, we could also access the raw libxml validation specifics, by calling libxml_use_internal_errors(true), and then when a document fails to validate, using libxml_get_errors() and libxml_get_error() to grab the details.

Problem #2: Returning valid XML responses

The root of all of our problems in this project has to do with where namespaces are defined. One limitation of libxml appears to be that you can only point it to one schema for validation at a time. We can validate against the SOAP Schema, or our custom schema, but not both at the same time, unless one includes the other. So our validation options consist of:

1. Include the SOAP Schema in the custom schema, and validate the entire SOAP body, or
2. Extract the payload from the SOAP body, and validate only that against our schema.

#2 is clearly the correct way–we really don’t care about the SOAP envelope once we have the message. But the problem is, many SOAP clients put the namespace declarations on the SOAP Envelope, and not the payload root element. In fact, the XML generated by the PHP SoapServer class does this itself.

So our first task was to generate the proper XML Namespace declarations on our generated payloads.

To do this, we could no longer rely on the PHP SoapServer’s automatic conversion of simple objects or arrays to XML–we had to generate our own XML, and tell the SOAP server to use that instead. This turned out to be difficult to track down, so here’s the answer:


< ?php
class dataClass{
/* Serialize XML as desired here.
Omit the XML declaration, start with the root element
You can also simply build the XML as a string from object properties
*/
function toXml(){
$this->myDom->documentElement->setAttribute('xmlns','http://my.custom.namespace/version_1');
$xml = $this->myDom->saveXML();
preg_match('%< \?.*=\?>(.*)$%s',$xml,$match); //strip XML declaration
return $match[1];
}
}

// snip to end of actual SOAP handler method:
$out = new SOAPVar($data->toXML(),XSD_ANYXML);
return $out;
}

The SOAPVar object allows you to control the output of the SoapServer a bit better, with support for namespaces, raw XML, or casting objects in a certain way.

Problem #3: Jitterbit

Validating a clean payload without namespace prefixes seems to work fine. Our technique of moving the nodes around with the DOM seemed to keep most of the namespaces intact, so even if there was no xmlns declaration on the payload root, we could still validate just the payload effectively, and serialize/de-serialize without issue.

Except for Jitterbit.

Now, I loaded up Jitterbit yesterday, and it didn’t do this–this might be a problem with an earlier version. The problem is, Jitterbit is extremely verbose, specifying not just a namespace prefix on every element, but also an xsi:type. And even that’s not enough to break it–except that the value for its xsi:type also contained a namespace declaration. And if this namespace declaration was not the root namespace, suddenly our validation broke.

It broke for us on types declared as ns:token, ns:string, ns:integer — the simple types specified by XSD itself, which Jitterbit put into a namespace prefixed with ns: and declared on the SOAP Envelope.

For example, here’s the start of a problem document:


< ?xml version="1.0" encoding="UTF-8" standalone="no" ?>
xmlns:ns="http://www.w3.org/2001/XMLSchema"
xmlns:oi="http://ws.outdoorindustry.org/v1_2/"
xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
xmlns:ws="http://ws.outdoorindustry.org/v1_2/ws/"
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>

xmlns:tns1="http://ws.outdoorindustry.org/v1_2/"
xsi:type="tns1:PO">
06AB4EF9-6AF157E7-3F8441AB-AB499933
Preseason

999
Sample Vendor

The first validation error was on the VendorID, with xsi:type=”ns:token”. If I copied xmlns:ns=”http://www.w3.org/2001/XMLSchema” into the tns1:submitPO element, it validated fine. The PHP DOMDocument seems to be able to keep track of namespaces on elements and attribute names even after the envelope is gone. But not attribute values.

After hours of banging on this, we came up with 3 workarounds for this:

1. Completely regenerate the XML, after processing. To do this, we would need to create a custom data class for each incoming object, provide a classmap to the SoapServer, and then generate brand new XML out of the data object. This is perhaps the best approach, but I didn’t think of it until the project was over–I was thinking about writing out the data to the database and then loading our custom objects and serializing them as we do for our responses. The biggest drawback here is that we need to model the entire complexity of the request, as allowed in the schema. And this was a really complex object… lots of work to implement, when we’re only going to store this XML for passing to other systems.
2. Hack the XML to get the offending namespace into the stored document. This turned out to be easy to program, but uses lots of CPU resources–DOMDocuments are expensive to use. It’s also the most brittle approach, only catching this single case–if the namespace prefix changes, or a different required namespace is necessary, it’ll break. To do this, we created a new DOM Document, imported the root node of the payload, appended it to the document, and used setAttribute to set an “xmlns:ns” attribute on the root. This did not actually get the namespace recognized for validation, and normalizeDocument did not fix it–but creating a third DOMDocument, and doing $doc->loadXML($doc2->saveXML) did make the namespace recognized by the object so we could successfully validate.
3. Hack the XSD to validate the entire SOAP request. By including the SOAP schema in our custom schema (using xs:import), we could validate the raw SOAP request, then extract the payload and save it. The saved XML does not validate on its own, but we know it validates. So we can remove our validation check on the outgoing document, and as long as the other system does not explicitly validate the standalone XML, we’re okay.

Whew. Hope this helps somebody…

For a long time, we’ve managed our custom code projects and business documents in a central repository, called Subversion (also known as svn). Subversion is relatively easy to understand–it’s like having a library of files you can check a copy out of, do some work on it, and then check it back in. Subversion is the librarian that tracks who has copies of what, and when they checked it out. So if Erik checks in changes to a brochure, and then Jill goes to submit changes to the same document, Subversion will say “hey wait a minute, that document has already been changed–you need to make sure you put Erik’s changes in your document before I’ll let you put in your document.”

This is great for managing conflicts between people working on a single team, or for code that is being developed in relative isolation from the rest of the world.

The problem is, we’re doing more than that–we’re taking code from various open source projects and either customizing it or building new applications on top of it. And so when the outside projects get updated, we need to manually update anything we’ve written that depends on that code. There is no longer a single repository where we control our code–there is our code library, plus another one for every project we use.

This makes managing add-ons for projects like Joomla or ZenCart quite challenging, because our add-ons get scattered throughout the filesystem to be able to hook into the right place. And if we have to touch a core file, we’re going to end up needing to re-implement our change with any update to that core file.

There are other issues we run into, managing our code and hosting, all of which take fairly time-consuming, manual intervention. Here’s the list:

• Since we host and provide security updates for Joomla, Word Press, Zen Cart, Drupal, and others, we need to upgrade dozens of installations any time there’s a new release that has a fix for a security vulnerability. With Joomla this has happened quite a lot, and every Joomla installation needs to be upgraded individually–and tested. And since each installation is slightly different, we can’t manage them easily within a single repository, while updating the underlying code.
• Templates, modules, components, blocks, themes, plugins, and whatever. Developing these types of add-ons are our bread-and-butter. But code for these often get scattered across an installation, making it quite difficult to manage just our add-ons while we develop them, or roll back to earlier versions if there’s a problem.
• The Dojo Toolkit, and builds. We’re doing a lot of development with Dojo right now, to add desktop-like functionality such as trees, sortable tables, right-click menus, animations, and lots of other really cool things. However, if you don’t “build” the code after you write it, it’s painfully slow in a web browser. And due to the nature of how Subversion works, you can’t easily store a built Dojo tree if you ever want to change it again. Which means you’d need to build it every place you deploy it. And on some computers, it can take a long time to build–on our demo server, one of our projects currently takes 8 minutes.
• As we get more directly involved with open source projects like LedgerSMB, we’re finding the need to change core files while we hack away at some particular feature. To do this, you create a branch of the code, work on your feature, and then merge your changes back into the “trunk.” If you don’t have access to save directly to the project repository, doing this gets a lot more complicated.

Git to the rescue. Git solves all of these issues. Read on for a technical discussion of how.

Managing lots of installations with Git
We haven’t yet started doing this, but for projects like Joomla and ZenCart, this is going to be really effective. It’s not quite as helpful for Drupal, which allows for a bunch of sites to be centrally managed out of the box–though even there it will help us recover quickly from a failed upgrade.

Here’s the basic approach:

1. Create a master git repository, either by pulling down the Subversion repository using git svn, or just unpacking tarballs of a release and importing into our git repository for the project.
2. Create a new git clone for each installation, and drop into each installation directory. Create a current branch for that repository, and rebase it to the current installed version.
3. Commit local modifications to the repository. Make sure the .git directory is getting backed up, and is not accessible through the web server.
4. When a new release is available, use git pull to update each local installation. Use git reset HEAD^ –hard to undo if there’s a problem.

Managing submodules, customizations
This we get pretty much for free by having dedicated repositories for particular problems. Managing add-ons we’d like to use elsewhere can still be challenging, but we can pull customizations from other repositories to assemble custom packages–it’s at least easier to manage these across installations.

The main thing to do is to make sure when you’re developing on a particular add-on, you’re always doing it in a branch dedicated to those changes. Then you can merge them into other projects more easily. It’s also possible to “cherry-pick” commits involving particular files, and pull those changes into a new branch if development ended up getting mixed in with different features.

Dojo Toolkit builds
We’re huge fans of the Dojo Toolkit. It makes developing really sophisticated browser-based applications much easier, not just providing a large set of active widgets but also a data abstraction layer and an encapsulation system for your own code.

Its biggest downside is that every feature you want to use means adding another file that the browser needs to load. And we use a lot of features… the base package loads about 20 files, and in our applications there are well over a hundred. This takes a long time to load, and isn’t very scalable.

The solution? Use the Dojo build process. A Java program can build all the features you want to use into a single compressed, stripped, javascript file. The difference in speed is incredible–applications suddenly load right up. And this build system can easily build your Javascript as well.

The problem is, this build system is not compatible with Subversion, because Subversion scatters its working copy data inside each directory, but the build system deletes the directories every time you build. So you can’t easily manage built code inside Subversion.

Git stores all of its repository information in the top level directory, so you can easily build and re-build Dojo and manage it without any issues.

Do local development on open source projects
Now here’s the traditional reason to use git: to allow you to manage changes to open source projects without needing write access to their code repositories. We simply set up git repositories to track their Subversion repository, and then we can use all the features of git to manage our customizations.

When the original project has new releases, we just pull them into our repository and git handles most of the merging for us. If we want to contribute back to the project, we can copy them into our Subversion tracking branch and commit them straight back to Subversion–or send a patch.

We’re starting to set up public git repositories for projects we work with extensively. You can browse them here: http://git.freelock.com. You can clone and pull updates from here: git://git.freelock.com/git/<projectname>.git. You can read more about how to manage non-standard Subversion repositories in git here.

What follows is a very technical discussion–if you’re a business reader, you should probably skip this post…

HTTP Auth is a specific mechanism for handling authentication. HTTP Auth is built into Apache and IIS, and so the server can handle authentication purely through configuration, offering many different back ends for storing the data. Browsers also handle HTTP Auth natively, popping up a normal login box whenever it gets a Basic Authentication request from the server. But this login box is ugly, and doesn’t provide a friendly experience to allow people to create an account, get a password resent, or anything–it falls back to a basic error page. You can, of course, customize the error page, but not necessarily help people with the password login itself.

There are several benefits to using HTTP Auth, though. First of all, other applications on the same server can accept the same credentials, allowing you to sign in once and access multiple applications without having to log into each one. Secondly, you can set up stronger authentication methods, such as client-side certificates. Also, you can configure the server to protect large parts of a web site very easily, reducing exposure to information disclosure.

So how do you make a sign-in form on a web application set http auth? Browsers do not allow you to access these settings via script. You can use an XmlHttpRequest object to set authentication, but only after the proper challenge has been sent from the server. The biggest problem is, if the server sends this challenge twice in a row, your browser will intercept the second request and pop up the ugly password prompt. So designing a form that keeps this login prompt from popping up under most circumstances is quite the challenge.

The gist of the issue is that while you can open an XmlHttpRequest object with a user and password for http authentication, the browser will only actually use those credentials after the server has rejected a request. The process looks like this:

1. Your script creates and sends an XmlHttpRequest with http auth username and password.
2. The browser submits the request to the server, without sending the username and password.
3. The server responds with 401 requires authentication, and a WWW-Authenticate header specifying a realm.
4. The browser looks in its cache to see if it already has http auth set for that domain and realm. If it does, it sends those credentials, NOT THE ONES you specified in your XmlHttpRequest. If it does not have those credentials, only then will it set http auth to what your script asked for.
5. The server responds. Generally, if the username or password are incorrect, the server will repeat the 401 response, and WWW-Authenticate.
6. The browser gets its second 401 in a row, and pops up its password box. Your script never gets a chance to intercept this. So if the stored http auth credentials are wrong, or the user mistypes the password, their browser takes over and you get a password prompt.

How do you handle this situation? It turns out you need to engage in some trickery on both the client and the server.

Here’s a basic flow of how you need to handle this, from both the server and the client perspective:

1. First, collect the credentials from the user, and create your request as outlined above.
2. Browser sends request without credentials.
3. Server responds with 401 and WWW-Authenticate.
4. Browser sends cached credentials, if they exist, or your credentials if not.
5. If credentials are accepted, server allows log in and responds with 200. If credentials are not accepted, server returns an error code OTHER THAN 401, and does not send a WWW-Authenticate:
   1. We use 403 not authorized for a credential failure here. You might also use 400 Bad Request.
   2. Because the response was something other than 401, your browser caches the bad credentials.
   3. XmlHttpRequest status reflects the error condition.
   4. Your script checks the result for the error your server has returned. Now comes the crucial part:
   5. Your script submits a new request with different credentials to some server location that will return successfully. For example, we call a login method on our application, passing username “public” and password “?”.
   6. The browser sends the new credentials and submits the request.
   7. The server returns 200.
   8. The browser updates its http auth cached credentials with the new bogus ones.
6. Now you can present an error to the user, and ask for new credentials.

The key to the above process is that if the browser gets two 401 responses without having a 200 somewhere between, it will pop up its password box and there’s nothing you can do about it. So the key is to use a different error code to indicate bad credentials, and do an intervening request that will return 200 so that you can re-authenticate.

Logging Out
You cannot really log out of HTTP Auth. But you can change the credentials to a known bad user. That’s a key technique we use to effectively log out of an application, and we re-use this method to reset after bad credentials.

On the server
I’m very much still in development with this. You can see the server side code for Project Auriga logins here.

In this system, we do set a cookie after successful login, to keep from having to check credentials again. This script also allows for cookie-only logins without using http auth. The important bits:

• action=logout: if this is called, the script always returns successfully. This allows the client script to provide new bogus credentials. It passes a username of “public” to log out completely.
• action=httpauth: if this is called, and there are no http auth credentials or the http auth username is “public”, return a 401 and WWW-Authenticate. This is always the first request from a browser, and triggers the browser to re-request with the credentials.
• action=httpauth, with http auth username set, and it’s not “public”: The second or later requests, we never want to return a 401 or the browser will pop up its password prompt. So we return 403 (or 400) if the credentials are bad, or allow the script to continue processing if its good. In this case, our authenticate method returns true if credentials are good, false if the user is not found, and throws an exception if the credentials are bad.

That’s basically what you need to do on the server side. Now for the client.

Client-side logins
We’re using the Dojo Toolkit extensively in Project Auriga, so the login functions are using dojo.xhr* requests to wrap the XmlHttpRequest objects and provide convenient callback functions. You can see our login code here. Key items:

• auriga.login is called by the login form. Note that if this is the first time to this page, the dojo.xhrPost actually happens twice: first time with no credentials, and the second time with them. If the second post is accepted, auriga.login_complete is called. If the second post returns any kind of error, auriga.login_err is called.
• auriga.login_complete is easy… it just redirects to wherever the server response designates.
• auriga.login_err is the real trick here. If it detects the error code we’ve chosen for bad passwords, it immediately calls the server logout method, to get a good response so the next time the browser gets a 401, it won’t immediately pop up the password box.

You can see the code in action on our demo server.

Other notes

• Actually doing single sign-on is hard. We’re trying out different strategies for detecting whether a user already has http auth set, by calling our login method once on page load, but haven’t gotten that figured out. In our current script, just clicking Login with the form blank but authenticated elsewhere on the same domain and Realm, will log you in with your existing credentials.
• Because your browser stores credentials based on the domain and the realm together, all applications that you set to share these items must accept the same credentials. If you have a different password on a different system on the same server, you must set a different realm, or logging into one will log you out of the other.
• If you want to require http auth, but not Javascript, I suggest submitting something different to the server using Javascript to identify this type of request. Perhaps show your form only when Javascript is available, and when it’s not, have a link to a protected page to let your browser go ahead and show the password dialog.
• Using http auth can actually allow users to disable cookies, if your application is RESTful. In Project Auriga, the session login script supports either–the client pages and logins work with either a cookie or http auth. The login process attempts to set http auth and a session cookie. On subsequent attempts, it uses the cookie to avoid re-authenticating every request.
• Finally, a note on security: Basic Authentication provides no protection against passwords being sniffed over the network. If you need a secure login, be sure the server conversation uses SSL–otherwise neighbors on your wireless network can easily sniff out your password. HTTP Auth does not make your application more secure–it just makes it easier to share authentication with other resources on the same server.