X-Cart: Slo-Cart to Go-Cart!

Now having worked as Technical Director for a web application vendor, for 7 special years, I can’t remember the number of times people used to ’get tough’ with us about the performance of our web application under load. The fact was, that with any web application, heavy load is always going to mean that some sort of ’special’ & ’Not-so-off-the-shelf’ configurations would be called for. And these special configurations would obviously cost time and money to develop and tailor to the customers requirements, and hence the customers disgust at discovering a painful side effect of success in Web 2.0 life.

This of course is a problem which curses any person running off the shelf web products, content management systems, shopping carts etc. Suddenly your site gets a little on the busy side, and your server turns to treacle. It’s natural to want to blame the server guys. And of course the server guys blame the software guys. Sound familiar?

And so we get a call from someone having (what we like to refer to as) ’success’ challenges with a shopping cart package called ’X-Cart’. Basically they found it slow. So slow, that for large periods of time, even viewing the homepage (forget buying) involved making tea. Lots of it. The server people say the servers fine. PHP on the server is fine. What is going on?

After a quick pop into the sickly server in question, we could see that the issues were a little to do with the Application, a little to the Server - but a LOT to do with how requests were being handled between the two. In short, we are looking at a car crash - in a traffic jam.

So we bench X-Cart up in the lab environment. s per usual, we wanted to make this a control test, which is basically load testing the application with absolutely no tweaking of the core services such as PHP & MySQL. We are testing it right off the shelf.  As expected, the ’off the shelf’ test of X-Cart didn’t impress.

As you can see in Fig.1 we’re looking at some pretty ’laggy’ scores here. Our ’rush test’ which subjected the applications to almost DOS levels of traffic (100 users with 2 new users every two seconds) managed 4 pages a second, 106 hits / second, and average throughput of 191kb/s and the all important average page response time of just over 13 seconds!    The really telling stat though is looking at the server CPU usage % whilst the test is running:     Fig.2 XCART 100 USER LOAD TEST CPU USE - VANILLA SETUP     After 60 user sessions, XCart is taking up pretty much all the server CPU.   Before X-Cart customers go up in arms, please remember this is a very extreme test, and as stated, we’re testing an un-tweaked setup here.   With our control test in the bag, next was the fun bit. Basically, slapping on some nice go faster stripes, and lets not forget a big cylinder of Nitrus Oxide eh?   Initial packet inspection tests revealed that indeed there was a traffic problem. PHP was talking a little longer to process requests that our virtual users were happy with. This was causing the requests to back up, and well - our stressed out Cart finds itself in a - err a ’race condition’.   Sorry about that one.   How to fix? Well we need to apply some a good old V8 Traffic Management system in front of this Cart server. We’ll use it to offload and tune the requests so PHP doesn’t get it’s knickers in a twist, perhaps we’ll cache some resource intensive requests, and rate limit others. After just an hour of tinkering under the hood, we too our new hot rod for a test drive. Bring on the load test from Hell!     Fig.3 XCART 100 USER LOAD TEST - MANAGED SETUP     The results were nothing short of amazing, and just goes to show that, without changing a single server configuration, or application setting, your application can be transformed into a sleek and shifty web application, ready to get slapped in the Facebook by a popular linkage. Basically we are looking at a staggering 400% increase in performance - and the average page response time is down to a highly responsive 78ms.   And that’s how we turned a Slo-Cart into a GoGoGo-Cart which even Lewis Hamilton would be proud of.... OK I will stop now, honest....