eZ Platform Content and Location filtering API

The state of the search

Until now, there were no options to perform the filtering without a search engine. There are several services that allowed the developers to perform the search. First of all, there was the SearchService that was provided as the part of the Repository layer and was distributed with the eZ Platform by default. Then we have 3rd party solutions like Find and Filter services, distributed as part of the Site API. Those services provide high-level access to Search API that is not dependent on the configured search engine. 

That being said, the aforementioned services, we can configure the following backends:

• legacy - uses database search
• solr - uses Solr
• elasticsearch -  uses Elastic, but it’s only available in the Enterprise

Let’s check the Search and Filter services more in detail.

The SearchService

The SearchService is the default Search API provided by the eZ Platform. It is powerful enough for a variety of cases, allowing full-text search and standard querying using built-in Criterion and Sort Clause value objects. As already mentioned, it supports the different search backends through a standardised API. That being said, we can easily select the search backend only by choosing the search engine in the project configuration. Any additional changes on the code are not needed. The only drawback is that configured search backends are being used throughout the whole project. For example, if the Solr search engine is configured, it is going to be used as the main and the only search backend on which SearchService depends. While Solr search engine provides more features and more performance than the Legacy search engine, it’s a separate system that needs to be synchronised with changes in the database, which is acceptable for search pages that contain complex search logic or requires some sort of full-text search. We can name two examples where this approach falls short. The first problem is asynchronous and delayed indexing. This means that changed data is not immediately visible through a search engine. There are places in the core where SearchService is used, and that place is LocationService, for example. Methods deleteLocation() and moveSubtree() can have unwanted results if the data is not indexed when the methods are used. 

The second problem is limited search functionality without a search engine. There was no option to perform any kind of search until now. The only option available was loading by ID. Another issue, for example, was the case when the editor updates the content. However, still content is not available on the frontend because the Solr index was not up to date when the HTTP cache generation occurred. This forces the editor to either clear the HTTP cache manually or wait until the cache expires.

Added sugar - Site API FilterService

A few years back, our Petar developed the FilterService to solve the problem of delayed indexing and Solr dependence. Initially, it started as an idea to solve those problems directly on the eZ Platform a few years back by this pull request. Meanwhile, it has been implemented as part of our Site API layer. The purpose of the FilterService is to find Content and Locations by using eZ Platform’s Repository Search API, but with the difference that it will always use the Legacy search engine. It works as a drop-in replacement for a SearchService, supporting the same Search API, which means it uses the same Query, Criterion and Sort Clause objects. It supports the same Criterion and Sort Clauses as a SearchService. FilterService gives you access to search that is always up to date because it uses Legacy search engine that works directly with a database. There is only a minimal difference between FilterService and SearchService, the method naming. The only difference between FilterService and SearchService is in the service and method you use to perform the search. The two new methods introduced:

• FilterService::filterContent()
• FilterService::filterLocations()

This solved the problem for us. That's our default for search now. It's a search service that always works with Legacy search engine and also takes current user and permissions into account. 

Pragmatic Repository filtering

The Repository filtering has some resemblance to the FilterService. It will always use the database to filter the Content or Locations but without the search engine backend and the indexed data. That is where all the similarities stop. Repository filtering does not expose the new service for filtering but rather adds new methods on the existing ContentService and LocationService:

• ContentService::find()
• LocationService::find()

Both methods accept the instance of the Filter value object, instead of the Query or LocationQuery as we used to. The cool thing is that Filter exposes the fluent interface, which makes building the filter criteria easy and readable, which we will demonstrate in the practical examples in the later chapters. The other difference from the SearchService or FilterService methods is that the Repository filtering methods return either the ContentList or LocationList and not the SearchResult object. Which, for example, means that we can’t use existing Pagerfanta pagers, but new ones need to be implemented. We will cover this topic later.

The Repository filtering supports most of the available Criterion and Sort Clause value objects. It is worth noting that Criterion and Sort Clauses work in the same way as with the use of the search engine, but internally are being handled by different services, not by the existing Criterion and Sort Clause visitors, but CriterionQueryBuilder and SortClauseQueryBuilder respectively. So have this in mind when implementing your custom Criteria or Sort Clauses.

So the Criterion or Sort Clause value objects that are supported by the Repository Filtering implement the following:

• eZ\Publish\SPI\Repository\Values\Filter\FilteringCriterion - in case of criterion
• eZ\Publish\SPI\Repository\Values\Filter\FilteringSortClause - in case of sort clause

The whole Repository filtering thing is quite performant as it is not building the SQL statements with the use of subqueries. This comes at a cost though, the Field criterion is not supported due to its complexity. The most problematic is the language handling when field filtering is used.

It is worth mentioning that both methods are SiteAccess-aware, which means that they will take the context, like languages of the current SiteAccess when performing filtering. 

Content filtering

Let’s start with the first example, the Content filtering API. All our practical examples are based on Netgen's eZ Platform 3 based Media Site. For the first example, we'll be building a simple page with the listing of fifteen latest articles. The idea is to create a custom controller that will hold the filtering logic, and a template to display the results. Start with adding a new FilterContentController to our project. It will contain only one public method, the filterContent(). Here we want to put the filtering logic. First, instantiate the Filter object. The first Criterion object should be added via withCriterion() method. Others should add calling the andWithCriterion() or orWithCriterion() method. Behind the scenes, those two methods will either add a new Criterion using a LogicalAnd or LogicalOr operator, respectively. Any Sort Clauses are added via the withSortClause() method. When our filter criteria are set, we can call the sliceBy() to set the limit and offset. Filter limit must be specified, as the Repository filtering doesn’t set the default limit like it is the case with a search service. 

Some eagle-eyed readers might notice the use of Location-related Criterion objects. This is fine with Repository filtering, and it is not going to throw an exception. So basically, we can reuse our filter configuration with either Content or Location filtering. We can’t use the FullText criterion, for example, as it is not supported. With everything put together, our controller looks like this:

<?php

declare(strict_types=1);

namespace App\Controller;

use eZ\Bundle\EzPublishCoreBundle\Controller;
use eZ\Publish\API\Repository\ContentService;
use eZ\Publish\API\Repository\Values\Content\Query;
use eZ\Publish\API\Repository\Values\Content\Query\Criterion;
use eZ\Publish\API\Repository\Values\Content\Query\SortClause;
use eZ\Publish\API\Repository\Values\Filter\Filter;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\Annotation\Route;

final class FilterContentController extends Controller
{
    /**
    * @Route("/example/filter/content", name="app_filter_content")
    *
    * @param \eZ\Publish\API\Repository\ContentService $contentService
    *
    * @return \Symfony\Component\HttpFoundation\Response
    */
    public function filterContent(ContentService $contentService): Response
    {
        $rootLocation = $this->getRootLocation();

        $filter = new Filter();
        $filter->withCriterion(new Criterion\ContentTypeIdentifier('ng_article'));
        $filter->andWithCriterion(new Criterion\Subtree($rootLocation->pathString));
        $filter->andWithCriterion(
            new Criterion\Location\IsMainLocation(
                Criterion\Location\IsMainLocation::MAIN
            )
        );
        $filter->withSortClause(new SortClause\DatePublished(Query::SORT_DESC));
        $filter->sliceBy(15, 0);

        $results = $contentService->find($filter);

        return $this->render( 
            '@ezdesign/filter/content.html.twig',
            [
                'results' => $results,
            ]
        );
    }
}

Registering the controller as service is not required, as it is going to be automatically done for us. The same thing goes with controller method dependency, the eZ\Publish\API\Repository\ContentService. If you are not familiar with concepts of autowiring, please take a look at this blog post.

As for the template, we will create a content.html.twig template in template/themes/app/filter directory. It reuses some styling from the Media Site and contains a simple for loop that iterates over results and renders a search view type for every content item. So the template will look like this:

{% extends nglayouts.layoutTemplate %}

{% block pre_content %}
    <header class="full-page-header full-article-header">
        <div class="container">
            <h1 class="full-page-title">Latest articles</h1>
            <div class="full-page-info">
                Here you can find the fresh stuff.
            </div>
        </div>
    </header>

    <div class="full-search-results">
        <div class="container">
            <div class="row">
                <div class="col-xs-12">
                    <div id="search-result">
                        {% for content in results.iterator %}
                            {% include '@ezdesign/parts/ng_view_content.html.twig' with {
                                content: content,
                                view_type: 'search'
                            } only %}
                        {% endfor %}
                    </div>
                </div>
            </div>
        </div>
    </div>
{% endblock %} 

When everything is put together, we will have a lovely article listing based on some predefined criteria.

It is time to move on to the Location filtering example.

Location filtering 

Again we will work on the article listing, but this time involving the Location filtering and pagination. Currently, there is no provided Pagerfanta adapter out of the box, so we need to build one. Let’s start with a custom controller. Create a new FilterLocationController controller with filterLocations() method. This time we need an eZ\Publish\API\Repository\LocationService. Filtering logic is very similar; we want only articles, sorted by date published. We can see how easy and natural is to write down the filtering logic with the fluent interface. But this time we won’t call the sliceBy() method manually, as the FilterLocationAdapter will do it for us. We're going to write down the adapter implementation in a moment.

And finally, let’s attach this method to the /example/filter/location route. To sum everything up, this is how our controller looks like:

<?php

declare(strict_types=1);

namespace App\Controller;

use App\Pagerfanta\FilterLocationAdapter;
use eZ\Bundle\EzPublishCoreBundle\Controller;
use eZ\Publish\API\Repository\LocationService;
use eZ\Publish\API\Repository\Values\Content\Query;
use eZ\Publish\API\Repository\Values\Content\Query\Criterion;
use eZ\Publish\API\Repository\Values\Content\Query\SortClause;
use eZ\Publish\API\Repository\Values\Filter\Filter;
use Pagerfanta\Pagerfanta;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\Annotation\Route;

final class FilterLocationController extends Controller
{
    /**
    * @Route("/example/filter/location", name="app_filter_location")
    *
    * @param \Symfony\Component\HttpFoundation\Request $request
    * @param \eZ\Publish\API\Repository\LocationService $locationService
    *
    * @return \Symfony\Component\HttpFoundation\Response
    */
    public function filterContent(Request $request, LocationService $locationService): Response
    {
        $rootLocation = $this->getRootLocation();

        $filter = new Filter();
        $filter->withCriterion(new Criterion\ContentTypeIdentifier('ng_article'));
        $filter->andWithCriterion(new Criterion\Subtree($rootLocation->pathString));
        $filter->withSortClause(new SortClause\DatePublished(Query::SORT_DESC));

        $pager = new Pagerfanta(
           new FilterLocationAdapter(
               $filter,
               $locationService
           )
        );

        $pager->setNormalizeOutOfRangePages(true);
        $pager->setMaxPerPage((int) 12);

        $currentPage = $request->query->getInt('page', 1);
        $pager->setCurrentPage($currentPage > 0 ? $currentPage : 1);

        return $this->render(
            '@ezdesign/filter/location.html.twig',
            [
                'pager' => $pager,
            ]
        );
    }
} 

To create a custom adapter for the Pagerfanta pagination library we need to implement the Pagerfanta\Adapter\AdapterInterface and provide implementation for getNBResults() and getSlice() methods. Let’s add a new FilterLocationAdapter class to App\Pagerfanta namespace and write down the implementations for the required methods:

<?php

declare(strict_types=1);

namespace App\Pagerfanta;

use eZ\Publish\API\Repository\LocationService;
use eZ\Publish\API\Repository\Values\Filter\Filter;
use Pagerfanta\Adapter\AdapterInterface;

final class FilterLocationAdapter implements AdapterInterface
{
    /**
    * @var \eZ\Publish\API\Repository\Values\Filter\Filter
    */
    private $filter;

    /**
    * @var \eZ\Publish\API\Repository\LocationService
    */
    private $locationService;

    public function __construct(Filter $filter, LocationService $locationService)
    {
        $this->filter = $filter;
        $this->locationService = $locationService;
    }

    public function getNbResults()
    {
        if (!isset($this->nbResults)) {
            $filter = $this->filter;

            $filter->sliceBy(0, 0);

           $this->nbResults = $this->locationService
               ->find($filter)
               ->totalCount;
        }

        return $this->nbResults;
    }

    public function getSlice($offset, $length)
    {
        $filter = clone $this->filter;

        $filter->sliceBy($length, $offset);

        return $this->locationService
            ->find($filter);
    }
}

All fine except one thing. Our adapter is automatically registered as a service by Symfony's dependency injection container. To check which services are available in the container, run the debug:container command. If we scroll back up to the start of the command output, we can see our App\Pagerfanta\FilterLocationAdapter. We don’t need the FilterLocationAdapter to be registered as the Symfony service, as we are going to instantiate it manually on-demand in our controller. To exclude our adapter from the container, we need to update the main config/services.yml file. Just add our Pagerfanta namespace to the exclude option:

services:
    _defaults:
    autowire: true # Automatically injects dependencies in your services.
    autoconfigure: true # Automatically registers your services as commands, event subscribers, etc.

    App\:
        resource: '../src/*'
        exclude: '../src/{DependencyInjection,Entity,Migrations,Tests,Pagerfanta,Kernel.php}'

At this point, our backend part is done. You can duplicate the previous template and add the logic for Pagerfanta pagination: 

{% extends nglayouts.layoutTemplate %}

{% block pre_content %}
    <header class="full-page-header full-article-header">
        <div class="container">
            <h1 class="full-page-title">Latest news and articles</h1>
            <div class="full-page-info">
                Here you can find the fresh stuff.
            </div>
        </div>
    </header>

    <div class="full-search-results">
        <div class="container">
            <div class="row">
                <div class="col-xs-12">
                    {% if pager.haveToPaginate() %}
                        {{ pagerfanta(pager, 'ngsite') }}
                    {% endif %}

                    <div id="search-result">
                        {% for location in pager.iterator %}
                            {% include '@ezdesign/parts/ng_view_content.html.twig' with {
                                location: location,
                                view_type: 'search'
                            } only %}
                        {% endfor %}
                    </div>

                    {% if pager.haveToPaginate() %}
                        {{ pagerfanta(pager, 'ngsite') }}
                    {% endif %}
                </div>
            </div>
        </div>
    </div>
{% endblock %}

Let’s check the final result by pointing our browser of choice to /example/filter/location URL.

You can found the complete examples in this repository.

Conclusion

In this blog post, we explored a new option of content and location filtering, but without involving the use of search engines. Being not dependent on the search engine configuration, providing the support for most of the Criterions and Sort Clauses, exposing the fluent interface, this new API will help developers to implement less complicated searches in a breeze. On the other hand, we have an option to use the FilterService, which is like a search service that always works with the Legacy search engine and supports the full range of Criterions and SortClauses.