Implementing asynchronous functionality in Magento

ECommerce is a small thing, right?  Nobody’s doing it and it’s so simple that everyone who does it is doing it right.  When that Cyber Monday hits, nobody panics; sites stay up, they’re able to handle the load and nobody gets yelled at, right?

OK, maybe 20 years ago.

PHP eCommerce had humble beginnings.  Very humble beginnings.  And when those humble beginnings started to show there was a company that seized upon that opportunity.  The result is an eCommerce platform called Magento.  If you are reading this, it is likely that you know about Magento.  Maybe you use it, maybe you don’t.  But you probably have an opinion.  Whether you like the software or not that is the sign of a strong ecosystem.

I was asked to submit for the Magento Imagine conference in Los Angeles.  The topic of my talk was based off of a talk I did at ZendCon 2010 called “Do You Queue?”  That talk was about some general scalability considerations along with an example of a library that I wrote which allows you to utilize the Zend Server Job Queue to do easy asynchronous execution.  The talk I gave at Imagine was the same talk.

Except that it required a lot more code.  The reason for this is because I took the code that I wrote for ZendCon and created an abstraction layer that directly integrated with Magento without any changes in the core code.  What this means is that ANYONE who had any need for asynchronous execution (doing stuff outside of the inline code) can take that code and bake it into their own Magento installation.

That code is available on GitHub, the links for which I will provide in a moment.

There are three extensions (and a fourth library that they are based off of) that I wrote which can take you from simply implementing this asynchronous processing to actually doing Ajax-based payment processing.

On a very simplified level, the way the Zend Server Job Queue works is that you can tell the Job Queue to execute a URL asynchronously from the source request.  In other words if you have something that needs to execute some complex, or long running code, you can do so by simply calling a URL where that logic resides.

Which is cool, but I prefer more elegant constructions.  Maybe, just maybe, there’s a shortcut (and a gold star for you if you get the movie reference).

What I did was build a library that allows you to take this URL-based approach and change it to an object-based approach.  There are a couple of classes to be aware of, which are all part of the library which you can download from GitHub (https://github.com/kschroeder/ZendServer-JobQueue-Job-API).  They are based on PHP 5.3.

• Manager 
  • Handles connecting to the queue and passing results back and forth
• JobAbstract 
  • Abstract class that a job would be based off of
• Response 
  • The response from the manager when a job is queued.  Contains the server name and job number

The only two classes you need to be concerned about is the JobAbstract and Response.  Any job needs to extend the JobAbstract class.  This is sort of a “gateway” class that both takes the input and provides the result of the job, typically via a getter and a setter.  To see an example of this, download the Job API code and look at the class in the folder jobs/org/eschrade/job/GetRemoteLinks.php.

To execute a job, simple instantiate the job, provide whatever data it needs and call the execute method.

1
2
3
4
5
6

use orgeschradejobGetRemoteLinks;
$job = new GetRemoteLinks(
      'http://www.eschrade.com/'
);
 
$response = $job->execute();

That method returns a response which provides the server name and job ID. It is used later on when you check to see if the job has completed.  That is done with some very simple code as well.

1
2
3
4
5
6

use com\zend\jobqueue\Manager;
 
$mgr = new Manager();
if (($job = $mgr->getCompletedJob($response)) !== null) {
     // do stuff
}

If the manager returns a null value it means that the job has not finished executing.  If it has completed then it will return the instance of the original object back to you so you can use it to retrieve the results.

I don’t want to spend too much time on the details of this so you can see this working by downloading Zend Server, and getting a 30 day trial license and downloading the library code.  It should be a very quick install for you so you can see it working.  On Linux machines it should work out of the box, though Windows machines may require you to set a named queue (zend_jobqueue.named_queues) which matches your hostname to the value of zend_jobqueue.default_binding.  In my case, the value is LAP-KEVIN:tcp://127.0.0.1:10085 for zend_jobqueue.named_queues.

That whole introduction is to bring you to a place where you can get a minimal view of how the Magento extension I built works.  I would recommend understanding how the base code works before diving into the Magento portion.

There are two primary Magento extensions that I built that utilize this.  The first is the abstraction layer that implements my prior job API.  The second is an example, called Async_Payment, which intercepts payment requests and does them asynchronously.

The Job Queue layer is an extension called Zendserver_Jobqueue and is available on GitHub (https://github.com/kschroeder/Magento-ZendServer-JobQueue).  Once installed it will require you to provide an entry point URL for the location where the jobs will actually execute.  This can either be the local machine, a remote machine or a load balancer.  It is set in the regular configuration GUI in Magento.  My value is http://mage.local/jobqueue/, since it uses the regular router.  If you have custom routing you may need to change that.  The URL needs to call Zendserver_Jobqueue_IndexController::indexAction() which is where the Job Queue manager is invoked.

If you look in the controller code you will also see a quick example that shows how this works.  There is a sample job that is provided called Zendserver_Jobqueue_Job_Nots.  What it does take a Boolean value and nots it, providing the result for later.  The job extends Zendserver_Jobqueue_JobAbstract which, in turn, extends comzendjobqueueJobAbstract.

As a side note, as of the version of Magento that I had when writing this, which was off of the 1.5 development brach, did not support PHP 5.3 namespaces so I needed to build a mechanism that included the Zend Framework autoloader, which does.  My understanding is that this is an issue that will be fixed shortly.

The next extension is the one called Async_Payment.  What it does is use an observer to redirect payment requests to the controller in Async_Payment.  The way this is done is via configuration under the ASYNCHRONOUS PAYMENT category.  That shows the different payment methods, but adds another tab called Asynchronous Settings.  The setting here allows you to turn the asynchronous processing on and off (handled in the observer).  What you need to do to make it work is give it the view templates to watch for.  When it sees that one of those templates (comma separated) is being rendered it appends some JavaScript that overwrites some of the functionality of the one-click checkout method to redirect the payment request to the Async_Payment controller.  Still following me?  My value is checkout/onepage.phtml.  So when that view is being rendered the extension will know that it needs to inject some JavaScript into the view to take hold of the payment request.

The final payment request is redirected to the Async_Payment_IndexController class.  What it does is take the data being submitted, which is exactly the same as the normal payment request, and passes it into a job, which is then executed in Async_Payment_IndexController::taskexecAction().  Then the browser will call Async_Payment_IndexController::oneclickpingAction() to check the queue manager to see if the asynchronous payment has been completed.

The asynchronous payment job is actually quite simple.  It pretends to be a browser and does an HTTP request to the original payment URL and returns the result.  Then the next time the browser calls oneclickpingAction() the raw result is returned to the browser, interpreting it as it would have been a normal request and you’re on your way.

Where to go from here?  First; download Zed Server.  There’s a 30 day trial license that you can use to try this stuff out.  Second; download the simplified Job Queue library.   Run the unit tests and debug the code.  That’s the best way for you to understand what’s actually going on.  After that, download and install the Magento extensions.  I think it is critical to work in this order, especially if you’re a coder.  Jumping straight into the Magento extensions will probably end up confusing you without the basic job queuing mechanism being properly understood.

Have fun, and drop me a line on email (kevin @ zend) or on Twitter.