Magento Testing Framework

Magento Testing Framework

For some time I’ve been working at edrone on creating an effective testing environment. It should give us the possibility of conducting regression tests for our edroneCRM module on different versions of the e-commerce platform Magento, which is used by our customers.

Magento is available in several dozen versions, which the clients may freely install and update. If we want to provide correct functioning and full service for each version, we need to have free access to each of them. Multitude of versions of the application in combination with the number of available operating systems, patches etc. quickly leads to an explosion of combinations. We have decided to solve this through virtualization.

Another important thing for us was to make the whole process automatic. Preparing a specific version for tests required installing it, configuring it, and finally deploying our module.

Since here at edrone we’ve been using the Amazon Web Servicesplatform from the very beginning, we believed that this would also be the best place for our test environment. We considered preparing a base image and a series of EBS disks each containing separate version of Magento. Indeed, an instance with the LAMP environment allows us to connect drives fairly quickly. The whole process would then consist only of switching appropriate drives/images depending on our needs.

But there were some problems with that. The first and basic of them, which manifested during the installation of EBS for our instance, was the synchronization of files with the database. Another relevant problem was the cost of this approach. Amazon charges a fee for every newly started instance and we want to start them often.. It is also worth to consider the efficiency of such a configuration. Although switching instances does not take much time (roughly 2–3 minutes) this process is tricky to automate. This would go against the premise of making the whole process fast and reliable.

We started looking for an alternative solution. It turned out that there is one that does not have any of the above shortcomings. Moreover, it works within a single EBS instance/drive.

Docker in service of testing

Docker is an open-source platform based on containers, created for purpose of building, providing and running applications — and, as we’re about to find out — also testing them.

The whole workload, which was initially to be done by taking advantage of multiple EBS drives, has been shifted to Docker images. Each of them is built deriving from a base image containing a configured LAMP environment, ready for installing Magento. This allows us to create new images very quickly.

Docker allows for easy redirection of TCP ports to the inside of the container. We were particularly interested in two of them: HTTP and SSH, which would allow us to connect independently with each container. This spawned the idea to make each version of Magento have its own dedicated ports under which it could be accessed. For example, Magento version 1.9.1.0 would be available at HTTP port 1910, and SSH port 21910.

Container management is done from the browser-level. It is possible toquickly launch the required version or turn it off from an intuitive panel on the website. Therefore container is ready to use within a few seconds.

The automation of the whole process is provided through Ansibleplaybooks. This allows us to prepare a new version in less than 15 minutes, including configuration and installation of edroneCRM.

All images created through this method are saved in a private repository, making them fully operational and ready for future use.


Testing in practice

Our tests are highly repeatable. They consist of carrying out the largest possible number of scenarios, and then checking if the module is gathering the desired information. We chose Selenium as the driver for these tests. It lets us easily write new scenarios (which can then be, to a large extent, simply “clicked through”) and then customize them.

The instance on which we launch the testing environment is t2.small with 16 GB of disk space. With this setup we can launch up to 10 parallel Docker containers. The costs of maintenance consist of: $0.026 per hour of operation of EC2 instance plus the costs associated with data stored (LAMP and Magento images) on EBS, that is $0.05 per month for each GB used. The monthly cost is within $15.

The environment is launched daily through Skeddly — the tool supporting our AWS account as far scheduling is concerned.


All of these tools create a consistent, effective system. Although our product is still in development, Magento Testing Framework has already proved to be useful with edrone.

Docker has shown great utility potential. Thanks to it we no longer have to worry about the costs, which are now mostly associated only with launching the instance. In the future we are planning to use it not only as a testing bed, but also for other solutions in our company.

Daniel Skoczny

Daniel Skoczny

view all post

By Daniele Zedda • 18 February

← PREV POST

By Daniele Zedda • 18 February

NEXT POST → 34
Share on