An approach to using Behat in eZ sites

Much like other content management frameworks, eZ Publish and eZ Platform have their idiosyncrasies which make the creation of ‘plain’ behavioural tests somewhat different than in the case of simpler sites.

There are many variables and considerations, but little in the way of best practices in Behat for eZ sites. This post will attempt to reason through some of the design choices I’ve made in utilising behavioural tests in recent projects.

Defining siteaccesses and environments

Given that the various siteaccesses of a site can amount to very different ‘sites’ running on multiple backends, it was necessary to find a way to test individual (siteaccess-specific) features in addition to features common to all siteaccesses, such as Solr search. There can be fifty or more siteaccesses, and we might be testing a development environment in addition to staging.

In order to keep the amount of custom configuration minimal, I make use of inheritance with the Mink configuration in behat.yml.

 default:
   extensions:
       Behat\MinkExtension:
           base_url: 'http://com.testsite.local'

This is the default profile, from which settings are inherited for any configuration keys omitted in other profiles. The idea is to make multiple profiles, one for each combination of siteaccess and Symfony environment. If this seems tedious, hold on.

There is only a single profile per siteaccess/environment combination, and only a single suite to keep it all simple. All that is necessary to define the dev profile for the testsite_uk siteaccess is:

 dev@testsite_uk:
   extensions:
       Behat\MinkExtension:
           base_url: 'http://uk.testsite.local'

One neat thing is that this enables us to run tests for individual environments from the Behat console like this:

 php bin/behat --profile dev@testsite_uk

Headless testing

I’ve omitted this from the previous section, but other settings can be inherited from the default profile, which includes Selenium configuration:

 default:
   extensions:
       Behat\MinkExtension:
           base_url: 'http://com.testsite.local'
           goutte: ~
           selenium2:
               wd_host: 'http://localhost:4444/wd/hub'

Goutte is used as the test driver whenever possible (by omitting the @javascript tag before scenarios in feature files). All other tests use PhantomJS. I’ve experimented with some other drivers, but PhantomJS turned out the most reliable and bug-free one of the bunch.

In the simplest terms, PhantomJS is a browser which can be run from the console, and it supports Javascript. Goutte (which is basically CURL) works perfectly until you get to a feature using onClick events or similar functionality.

I recommend using the most recent PhantomJS version, as I’ve experienced some JS events not triggering inexplicably in versions as late as 1.9.

After installing all its dependencies, I prefer to keep PhantomJS in my project’s bin/ folder. To run PhantomJS with the above setup, it is enough to type:

 bin/phantomjs --webdriver=4444

Notice this is the same port used in the configuration. In most cases one would probably prefer to run the program in the background and just log the output, but this is useful for initial testing.

While this process is running, javascript-enabled tests should pass. If there is an issue with using the driver, this is the most likely error message:

  ╳ Could not open connection: Curl error thrown for http POST to http://localhost:4444/wd/hub/session with params:
  ╳  
  ╳  Failed to connect to localhost port 4444: Connection refused (Behat\Mink\Exception\DriverException) 

In that case, double-check your port configuration in behat.yml, make sure PhantomJS is actually running, and running with the desired port. The example configuration assumes you are running both PhantomJS and Behat on the same machine, so there is no need for port forwarding, firewall exceptions or other shenanigans.

Changing state and test data

There are many approaches to using test data:

Encapsulate state changes in transactions to be rolled back after testing

Perhaps the most obvious approach - we have an initial state of the database, run our tests, then roll back any changes, such as created users, products, baskets, deleted content, new versions or translations, to return to the initial state. Unfortunately, this does not pan out well in practice. Given the number and complexity of inter-table relations in eZ, there are too many unknowns, such as filesystem changes, hidden flags, cache interaction etc. to be certain that the entirety of initial application and database state is preserved over a transaction rollback.

Test against generated fixture data

This approach works fine for smaller applications with all or most tables already bound to an ORM, such as Doctrine. Mapping the dozens of mutually referenced tables needed to get a basic variant of the site up and running would probably be too time-consuming.

A combination of generated fixture data and testing with a new copy of the database

The idea is this - we create special ‘test objects‘, such as ‘known users’ to test logging in as an existing user, shop offers which never expire for testing offers etc. However, to save time, I’ve opted for creating this fixture data using a Symfony command utilising the eZ API, not a fixture generation tool such as DoctrineFixturesBundle. In order to give these test objects a plausible environment (in this context, ‘plausible’ means that it should be as likely to trigger test failures as the production environment), the database is copied to the test environment. Then it is populated with the test objects, so tests can be run.

To get a ‘clean’ state, we have to repeat the entire process, erasing all changes made to the database. However, the database only needs to be re-imported if major changes occur, such as the addition of new tables, changes in field definitions etc. This approach is far from ideal, but at the time of the writing, it seems to be the only feasible option.

Limiting test scope

The Symfony2Extension for Behat allows us to use the Symfony container and eZ Publish repository in a somewhat limited manner. Its limitations have to do with all interaction happening outside the normal request-response cycle. Tests utilising the Symfony2Extension directly interact with the backend and can handle more advanced features, such as various clean-ups or rollbacks after running scenarios. However, as stated previously, state is tampered with in often unpredictable ways, which can even result in test-only bugs.

After some consideration, we’ve opted for using ‘dumb’ tests instead. This means that state is changed only via frontend input - all the tests can ‘do’ are clicks and keystrokes on some URI.

Feature tagging and test objects

The ‘default’ test suite which is applied to all profiles looks like this

 suites:
   default:
       paths: [ src/Acme/Bundle/TestBundle/features/ ]
       contexts:
         - Acme\Bundle\TestBundle\Context\GeneralContext:
             defaultSiteaccess: test_com
             existingUser: '[email protected]'
             existingPassword: 'dummypassword'

Only a single context and only a single feature path are used in order to keep the configuration minimal. By default, all tests underneath all test suite paths are run when you run Behat with the --profile parameter. In my implementation, the tests are divided into ‘common’ and ‘siteaccess-specific’ ones. The common tests should be run for all siteaccesses, but the siteaccess-specific ones should only be run for the siteaccesses they refer to. The solution is tagging the features/scenarios with @common or with a siteaccess such as @test_com. This allows us to run:

 php bin/behat --profile dev@test_com --tags common,test_com

A cool side-effect of the way tags are handled in Behat is that ‘test_com’ will be ignored if no features have been tagged with it, instead of throwing an error. This allows us to batch tests for all siteaccesses using the same formula!

Conclusion

This is just my personal experience, but getting the configuration for Behat 'right' in a CMF context has quite a few pitfalls. In terms of best practices for Behat, with little to go on apart from Sylius, there is certainly room for improvement, so any questions and/or suggestions are welcome.