Some of our previous articles have dealt with behavioural testing to some degree. This post will attempt to reason through some of the design choices I’ve made in using Behat in eZ sites.
Much like other content management frameworks, eZ Publish and eZ Platform have their idiosyncrasies which make the creation of ‘plain’ behavioural tests somewhat different than in the case of simpler sites.
There are many variables and considerations, but little in the way of best practices in Behat for eZ sites. This post will attempt to reason through some of the design choices I’ve made in utilising behavioural tests in recent projects.
Defining siteaccesses and environments
Given that the various siteaccesses of a site can amount to very different ‘sites’ running on multiple backends, it was necessary to find a way to test individual (siteaccess-specific) features in addition to features common to all siteaccesses, such as Solr search. There can be fifty or more siteaccesses, and we might be testing a development environment in addition to staging.
In order to keep the amount of custom configuration minimal, I make use of inheritance with the Mink configuration in behat.yml.
default: extensions: Behat\MinkExtension: base_url: 'http://com.testsite.local'
This is the default profile, from which settings are inherited for any configuration keys omitted in other profiles. The idea is to make multiple profiles, one for each combination of siteaccess and Symfony environment. If this seems tedious, hold on.
There is only a single profile per siteaccess/environment combination, and only a single suite to keep it all simple. All that is necessary to define the dev profile for the testsite_uk siteaccess is:
dev@testsite_uk: extensions: Behat\MinkExtension: base_url: 'http://uk.testsite.local'
One neat thing is that this enables us to run tests for individual environments from the Behat console like this:
php bin/behat --profile dev@testsite_uk
I’ve omitted this from the previous section, but other settings can be inherited from the default profile, which includes Selenium configuration:
default: extensions: Behat\MinkExtension: base_url: 'http://com.testsite.local' goutte: ~ selenium2: wd_host: 'http://localhost:4444/wd/hub'
I recommend using the most recent PhantomJS version, as I’ve experienced some JS events not triggering inexplicably in versions as late as 1.9.
After installing all its dependencies, I prefer to keep PhantomJS in my project’s bin/ folder. To run PhantomJS with the above setup, it is enough to type:
Notice this is the same port used in the configuration. In most cases one would probably prefer to run the program in the background and just log the output, but this is useful for initial testing.
╳ Could not open connection: Curl error thrown for http POST to http://localhost:4444/wd/hub/session with params: ╳ ╳ Failed to connect to localhost port 4444: Connection refused (Behat\Mink\Exception\DriverException)
In that case, double-check your port configuration in behat.yml, make sure PhantomJS is actually running, and running with the desired port. The example configuration assumes you are running both PhantomJS and Behat on the same machine, so there is no need for port forwarding, firewall exceptions or other shenanigans.
Changing state and test data
There are many approaches to using test data:
Encapsulate state changes in transactions to be rolled back after testing
Perhaps the most obvious approach - we have an initial state of the database, run our tests, then roll back any changes, such as created users, products, baskets, deleted content, new versions or translations, to return to the initial state. Unfortunately, this does not pan out well in practice. Given the number and complexity of inter-table relations in eZ, there are too many unknowns, such as filesystem changes, hidden flags, cache interaction etc. to be certain that the entirety of initial application and database state is preserved over a transaction rollback.
Test against generated fixture data
This approach works fine for smaller applications with all or most tables already bound to an ORM, such as Doctrine. Mapping the dozens of mutually referenced tables needed to get a basic variant of the site up and running would probably be too time-consuming.
A combination of generated fixture data and testing with a new copy of the database
The idea is this - we create special ‘test objects‘, such as ‘known users’ to test logging in as an existing user, shop offers which never expire for testing offers etc. However, to save time, I’ve opted for creating this fixture data using a Symfony command utilising the eZ API, not a fixture generation tool such as DoctrineFixturesBundle. In order to give these test objects a plausible environment (in this context, ‘plausible’ means that it should be as likely to trigger test failures as the production environment), the database is copied to the test environment. Then it is populated with the test objects, so tests can be run.
To get a ‘clean’ state, we have to repeat the entire process, erasing all changes made to the database. However, the database only needs to be re-imported if major changes occur, such as the addition of new tables, changes in field definitions etc. This approach is far from ideal, but at the time of the writing, it seems to be the only feasible option.
Limiting test scope
The Symfony2Extension for Behat allows us to use the Symfony container and eZ Publish repository in a somewhat limited manner. Its limitations have to do with all interaction happening outside the normal request-response cycle. Tests utilising the Symfony2Extension directly interact with the backend and can handle more advanced features, such as various clean-ups or rollbacks after running scenarios. However, as stated previously, state is tampered with in often unpredictable ways, which can even result in test-only bugs.
After some consideration, we’ve opted for using ‘dumb’ tests instead. This means that state is changed only via frontend input - all the tests can ‘do’ are clicks and keystrokes on some URI.
Feature tagging and test objects
The ‘default’ test suite which is applied to all profiles looks like this
suites: default: paths: [ src/Acme/Bundle/TestBundle/features/ ] contexts: - Acme\Bundle\TestBundle\Context\GeneralContext: defaultSiteaccess: test_com existingUser: 'email@example.com' existingPassword: 'dummypassword'
Only a single context and only a single feature path are used in order to keep the configuration minimal. By default, all tests underneath all test suite paths are run when you run Behat with the --profile parameter. In my implementation, the tests are divided into ‘common’ and ‘siteaccess-specific’ ones. The common tests should be run for all siteaccesses, but the siteaccess-specific ones should only be run for the siteaccesses they refer to. The solution is tagging the features/scenarios with @common or with a siteaccess such as @test_com. This allows us to run:
php bin/behat --profile dev@test_com --tags common,test_com
A cool side-effect of the way tags are handled in Behat is that ‘test_com’ will be ignored if no features have been tagged with it, instead of throwing an error. This allows us to batch tests for all siteaccesses using the same formula!
This is just my personal experience, but getting the configuration for Behat 'right' in a CMF context has quite a few pitfalls. In terms of best practices for Behat, with little to go on apart from Sylius, there is certainly room for improvement, so any questions and/or suggestions are welcome.