Hello pytest-play!

pytest-play is a rec&play (rec not yet available) pytest plugin that let you execute a set of actions and assertions using commands serialized in JSON format. It tries to make test automation more affordable for non programmers or non Python programmers for browser, functional, API, integration or system testing thanks to its pluggable architecture and third party plugins that let you interact with the most common databases and systems.

In addition it provides also some facilitations for writing browser UI actions (e.g., implicit waits before interacting with an input element. The Cypress framework for me was a great source of inspiration) and asynchronous checks (e.g., wait until a certain condition is true).

You can use pytest-play programmatically (e.g., use the pytest-play engine as a library for pytest-play standalone scenarios or using the pytest-play API implementing BDD steps).

Starting from pytest-play>1.4.x it was introduced a new experimental feature that let you use pytest-play as a framework creating Python-free automated tests based on a JSON based serialization format for actions and assertions (in the next future the more user friendly YAML format will be supported).

So now depending on your needs and skills you can choose to use pytest-play as a library or as a framework.

In this article I'm going to show how to implement a Plone CMS based login test using the python-free approach without having to write any line of Python code.

What is pytest-play and why it exists

In this section I'm going to add more information about the pytest-play approach and other considerations: if you want to see now how to implement our Python-free automated login test jump to the next section!

Hyper specialized tool problems

There are many commercial products or tools that offer solutions for API testing only, browser testing only. Sometimes hyper specialized tools might fit your needs (e.g., a content management system based web application) but sometimes they are not helpful for other distributed applications.

For example an API-only platform is not effective for testing a CQRS based application. It is not useful testing only HTTP 200 OK response, you should test that all the expected commands are generated on the event store (e.g., Cassandra) or other side effects.

Another example for an IoT applications and UI/browser only testing platforms. You cannot test reactive web apps only with a browser, you should control also simulated device activities (e.g., MQTT, queues, API) for messages/alarms/reports) or any other web based interactions performed by other users (e.g., HTTP calls); you might need to check asynchronously the expected results on web sockets instead of using a real browser implementing when some actions are performed.

What is pytest-play

In other words pytest-play is an open source testing solution based on the pytest framework that let you:

• write actions and cross assertions against different protocols and test levels in the same scenario (e.g., check HTTP response and database assertions)
• minimize the development of Selenium-based asynchronous wait functions before interacting with input elements thanks to implicit wait functions that let you interact with web elements only when they are ready. You just focus on user actions, you are more productive and you reduce the chance of writing bad or not robust asynchronous wait functions
• implement polling-based asynchronous waiter commands based on custom expressions when needed

using a serialization format (JSON at this time of writing, YAML in the next future) that should be more affordable for non technical testers, non programmers or programmers with no Python knowledge.

Potentially you will be able to share and execute a new scenario not yet included in your test library copying and pasting a pytest-play JSON to a Jenkins build with parameters form like the following one (see the PLAY textarea):

From http://davidemoro.blogspot.it/2018/03/test-automation-python-pytest-jenkins.html

In addition if you are a technical user you can extend it writing your own plugins, you can provide the integration with external tools (e.g., test management tools, software metrics engines, etc), you can decide the test abstraction depending on deadlines/skills/strategy (e.g., use plain json files, a programmatic approach based on json scenarios or BDD steps based on pytest-play).

What pytest-play is not

For example pytest-play doesn't provide a test scenario recorder but it enforces users to understand what they are doing.

It requires a very very little programming knowledge for writing some assertions using simple code expressions but with a little training activity it is still affordable by non programmers (you don't have to learn a programming language, just some basic assertions).

It is not feature complete but it is free software.

If you want to know more in this previous article I've talked about:

• test automation framework thoughts plus a comprehensive, detailed look at using pytest and Jenkins for automated testing.

A pytest-play example: parametrized login (featuring Plone CMS)

In this example we'll see how to write and execute pure json pytest-play scenarios with test data decoupled by the test implementation and test parametrization. I'm using the available online Plone 5 demo site kindly hosted by Andreas Jung (www.zopyx.com).

The project is available here:

• https://github.com/davidemoro/pytest-play-plone-example

The tests could be launched this way as a normal pytest project once you installed pytest and the dependencies (there is a requirements.txt file, see the above link):

$ pytest --variables env-ALPHA.yml --splinter-webdriver firefox --splinter-screenshot-dir /tmp -x

Where the you can have multiple environment/variable files. E.g., env-ALPHA.yml containing the alpha base url and any other variables:

pytest-play:
  base_url: https://plone-demo.info

Our login test_login.json scenario contains (as you can see there are NO asynchronous waits because they are not needed for basic examples so you can focus on actions and assertions thanks to implicit waits):

{
    "steps": [
        {
            "comment": "visit base url",
            "type": "get",
            "url": "$base_url"
        },
        {
            "comment": "click on login link",
            "locator": {
                "type": "id",
                "value": "personaltools-login"
            },
            "type": "clickElement"
        },
        {
            "comment": "provide a username",
            "locator": {
                "type": "id",
                "value": "__ac_name"
            },
            "text": "$username",
            "type": "setElementText"
        },
        {
            "comment": "provide a password",
            "locator": {
                "type": "id",
                "value": "__ac_password"
            },
            "text": "$password",
            "type": "setElementText"
        },
        {
            "comment": "click on login submit button",
            "locator": {
                "type": "css",
                "value": ".pattern-modal-buttons > input[name=submit]"
            },
            "type": "clickElement"
        },
        {
            "comment": "wait for page loaded",
            "locator": {
                "type": "css",
                "value": ".icon-user"
            },
            "type": "waitForElementVisible"
        }
    ]
} 

Plus an optional test scenario metadata file test_login.ini that contains pytest keyword and decoupled test data:

[pytest]
markers =
    login
test_data =
    {"username": "siteadmin", "password": "siteadmin"}
    {"username": "editor", "password": "editor"}
    {"username": "reader", "password": "reader"}

Thanks to the metadata file you have just one scenario and it will be executed 3 times (as many times as test data rows)!

Et voilà, let's see it in action out scenario without having to write any line of Python code:
 

There is only a warning I have to remove but it worked and we got exactly 3 different test runs for our login scenario as expected!

pytest-play status

pytest-play should be still considered experimental software and many features needs to be implemented or refactored:

• yaml instead of json. YAML will become the primary configuration format (it should be more user friendly as suggested by some users)
• API should not be considered stable until future 2.x version
• improve API testing when using pure json scenarios registering functions (e.g., invoke a function returning a valid authentication bearer for authenticated API testing)
• implement some python requests library features not yet implemented in play_requests (e.g., cookies)
• refactor parametrization and templating (Jinja?)
• implement additional Selenium actions (e.g., right clicks, upload files, etc)
• implement other cool Cypress ideas enabling non expert testers in writing more robust Selenium scenarios
• add page object abstraction in pytest-play based Selenium scenarios with new commands that let you interact with page regions and interact with complex UI widgets
• ownership change, waiting for pytest-dev core developers approval. Probably soon the ownership will change from davidemoro/pytest-play to pytest-dev/pytest-play once the approvation iter will finish

PyCon Nove @ Florence

If you are going to attending next PyCon Nove in Florence don't miss the following pytest-play talk presented by Serena Martinetti:

• https://www.pycon.it/conference/talks/integration-tests-ready-to-use-with-pytest-play

Do you like pytest-play?

Tweets about pytest-play happens on @davidemoro.
Positive or negative feedback is always appreciated. If you find interesting the concepts behind pytest-play let me know with a tweet, add a new pytest-play adapter and/or add a GitHub star if you liked it:

Star

Updates

• [20190222] pytest-play 2.0.0 released, no more backwards compatible with 1.x format. Using YAML syntax instead of JSON now and many other improvements and new features. See https://davidemoro.blogspot.com/2019/02/api-rest-testing-pytest-play-yaml-chuck-norris.html