Test automation framework thoughts and examples with Python, pytest and Jenkins

In this article I'll share some personal thoughts about Test Automation Frameworks; you can take inspiration from them if you are going to evaluate different test automation platforms or assess your current test automation solution (or solutions).

Despite it is a generic article about test automation, you'll find many examples explaining how to address some common needs using the Python based test framework named pytest and the Jenkins automation server: use the information contained here just as a comparison and feel free to comment sharing alternative methods or ideas coming from different worlds.

It contains references to some well (or less) known pytest plugins or testing libraries too.

Before talking about automation and test automation framework features and characteristics let me introduce the most important test automation goal you should always keep in mind.

Test automation goals: ROI

You invest in automation for a future return of investment.
Simpler approaches let you start more quickly but in the long term they don't perform well in terms of ROI and vice versa. In addition the initial complexity due to a higher level of abstraction may produce better results in the medium or long term: better ROI and some benefits for non technical testers too. Have a look at the test automation engineer ISTQB certification syllabus for more information:

• https://www.istqb.org/downloads/category/48-advanced-level-test-automation-engineer-documents.html

So what I mean is that test automation is not easy: it doesn't mean just recording some actions or write some automated test procedures because how you decide to automate things affects the ROI. Your test automation strategy should consider your tester technical skills now and future evolutions, considerations about how to improve your system testability (is your software testable?), good test design and architecture/system/domain knowledge. In other words be aware of vendors selling "silver bullet" solutions promising smooth test automation for everyone, especially rec&play solutions: there are no silver bullets.

Test automation solution features and characteristics

A test automation solution should be enough generic and flexible, otherwise there is the risk of having to adopt different and maybe incompatible tools for different kind of tests. Try to imagine the mess of having the following situation: one tool or commercial service for browser based tests only based on rec&play, one tool for API testing only, performance test frameworks that doesn't let you reuse existing scenarios, one tool for BDD only scenarios, different Jenkins jobs with different settings for each different tool, no test management tool integration, etc. A unique solution, if possible, would be better: something that let you choose the level of abstraction and that doesn't force you. Something that let you start simple and that follow your future needs and the skill evolution of your testers.
That's one of the reasons why I prefer pytest over an hyper specialized solution like behave for example: if you combine pytest+pytest-bdd you can write BDD scenarios too and you are not forced to use a BDD only capable test framework (without having the pytest flexibility and tons of additional plugins).

And now, after this preamble, an unordered list of features or characteristics that you may consider for your test automation solution software selection:

• fine grained test selection mechanism that allows to be very selective when you have to choose which tests you are going to launch
• parametrization
• high reuse
• test execution logs easy to read and analyze
• easy target environment switch
• block on first failure
• repeat your tests for a given amount of times
• repeat your tests until a failure occurs
• support parallel executions
• provide integration with third party software like test management tools
• integration with cloud services or browser grids
• execute tests in debug mode or with different log verbosity
• support random tests execution order (the order should be reproducible if some problems occur thanks to a random seed if needed)
• versioning support
• integration with external metrics engine collectors
• support different levels of abstraction (e.g., keyword driven testing, BDD, etc)
• rerun last failed
• integration with platforms that let you test against a large combination of OS and browsers if needed
• are you able to extend your solution writing or installing third party plugins?

Typically a test automation engineer will be able to drive automated test runs using the framework command line interface (CLI) during test development but you'll find out very soon that you need an automation server for long running tests, scheduled builds, CI and here it comes Jenkins. Jenkins could be used by non technical testers for launching test runs or initialize an environment with some test data.

Jenkins

What is Jenkins? From the Jenkins website:

Continuous Integration and Continuous Delivery. As an extensible automation server, Jenkins can be used as a simple CI server or turned into the continuous delivery hub for any project.

So thanks to Jenkins everyone can launch a parametrized automated test session just using a browser: no command line and nothing installed on your personal computer. So more power to non technical users thanks to Jenkins!

With Jenkins you can easily schedule recurrent automatic test runs, start remotely via external software some parametrized test runs, implement a CI and many other things. In addition as we will see Jenkins is quite easy to configure and manage thanks to through the web configuration and/or Jenkins pipelines.

Basically Jenkins is very good at starting builds and generally jobs. In this case Jenkins will be in charge of launching our parametrized automated test runs.

And now let's talk a little bit of Python and the pytest test framework.

Python for testing

I don't know if there are some articles talking about statistics on the net about the correlation between Test Automation Engineer job offers and the Python programming language, with a comparison between other programming languages. If you find a similar resource share with me please!

My personal feeling observing for a while many Test Automation Engineer job offers (or any similar QA job with some automation flavor) is that the Python word is very common. Most of times is one of the nice to have requirements and other times is mandatory.

Let's see why the programming language of choice for many QA departments is Python, even for companies that are not using Python for building their product or solutions.

Why Python for testing

Why Python is becoming so popular for test automation? Probably because it is more affordable for people with no or little programming knowledge compared to other languages. In addition the Python community is very supportive and friendly especially with new comers, so if you are planning to attend any Python conference be prepared to fall in love with this fantastic community and make new friends (friends, not only connections!). For example at this time of writing you are still in time for attending PyCon Nove 2018 in the beautiful Florence (even better if you like history, good wine, good food and meet great people): 

• https://www.pycon.it

You can just compare the most classical hello world, for example with Java:

public class HelloWorld {

    public static void main(String[] args) {

        System.out.println("Hello, World!");

    }

}

and compare it with the Python version now:

print("Hello, World!")

Do you see any difference? If you are trying to explain to a non programmer how to print a line in the terminal window with Java you'll have to introduce public, static, void, class, System, installing a runtime environment choosing from different version, installing an IDE, running javac, etc and only at the end you will be able to see something printed on the screen. With Python, most of times it comes preinstalled in many distributions, you just focus on what to need to do. Requirements: a text editor and Python installed. If you are not experienced you start with a simple approach and later you can progressively learn more advanced testing approaches.

And what about test assertions? Compare for example a Javascript based assertions:

expect(b).not.toEqual(c);

with the Python version:

assert b != c

So no expect(a).not.toBeLessThan(b), expect(c >= d).toBeTruthy() or expect(e).toBeLessThan(f): with Python you just say assert a >= 0 so nothing to remember for assertions!

Python is a big fat and very powerful programming language but it follows a "pay only for what you eat" approach.

Why pytest

If Python is the language of your choice you should consider the pytest framework and its high quality community plugins and I think it is a good starting point for building your own test automation solution.

The pytest framework (https://docs.pytest.org/en/latest/) makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries.

Most important pytest features:

• simple assertions instead of inventing assertion APIs (.not.toEqual or self.assert*)
• auto discovery test modules and functions
• effective CLI for controlling what is going to be executed or skipped using expressions
• fixtures, easy to manage fixtures lifecycle for long-lived test resources and parametrized features make it easy and funny implementing what you found hard and boring with other frameworks
• fixtures as function arguments, a dependency injection mechanism for test resources
• overriding fixtures at various levels
• framework customizations thanks to pluggable hooks
• very large third party plugins ecosystem

I strongly suggest to have a look at the pytest documentation but I'd like to make some examples showing something about fixtures, code reuse, test parametrization and improved maintainability of your tests. If you are not a technical reader you can skip this section.

I'm trying to explain fixtures with practical examples based on answers and questions:

• When should be created a new instance of our test resource?
  You can do that with the fixture scope (session, module, class, function or more advanced options like autouse). Session means that your test resource will live for the entire session, module/class for all the tests contained in that module or class, with function you'll have an always fresh instance of your test resource for each test
• How can I determine some teardown actions at the end of the test resource life?
  You can add a sort of fixture finalizer after the yield line that will be invoked at the end of our test resource lifecycle. For example you can close a connection, wipe out some data, etc.
• How can I execute all my existing tests using that fixture as many as your fixture configurations?
  You can do that with params. For example you can reuse all your existing tests verifying the integration with different real databases, smtp servers. Or if you have the web application offering the same features deployed with a different look&feel for different brands you can reuse all your existing functional UI tests thanks to pytest's fixture parametrization and a page objects pattern where for different look&feel I don't mean only different CSS but different UI components (e.g. completely different datetime widgets or navigation menu), components disposition in page, etc.
• How can I decouple test implementation and test data? Thanks to parametrize you can decouple them and write just one time your test implementation. Your test will be executed as many times as your different test data

Here you can see an example of fixture parametrization (the test_smtp will be executed twice because you have 2 different fixture configurations):

import pytest
import smtplib

@pytest.fixture(scope="module",
                        params=["smtp1.com", "smtp2.org"])
def smtp(request):
    smtp = smtplib.SMTP(request.param, 587, timeout=5)
    yield smtp
    print("finalizing %s" % smtp)
    smtp.close()

def test_smtp(smtp):
    # use smtp fixture (e.g., smtp.sendmail(...))
    # and make some assertions.
    # The same test will be executed twice (2 different params)
    ...

 And now an example of test parametrization:

import pytest
@pytest.mark.parametrize("test_input,expected", [
    ("3+5", 8),
    ("2+4", 6),
    ("6*9", 42), ])
def test_eval(test_input, expected):
    assert eval(test_input) == expected

For more info see:

• https://docs.pytest.org 
• https://docs.pytest.org/en/latest/fixture.html
• https://docs.pytest.org/en/latest/fixture.html#parametrizing-fixtures
• https://docs.pytest.org/en/latest/parametrize.html

This is only pytest, as we will see there are many pytest plugins that extend the pytest core features.

Pytest plugins

There are hundreds of pytest plugins, the ones I am using more frequently are:

• pytest-bdd, BDD library for the pytest runner
• pytest-variables, plugin for pytest that provides variables to tests/fixtures as a dictionary via a file specified on the command line
• pytest-html, plugin for generating HTML reports for pytest results
• pytest-selenium, plugin for running Selenium with pytest
• pytest-splinter, a pytest-selenium alternative based on Splinter. pPytest splinter and selenium integration for anyone interested in browser interaction in tests
• pytest-xdist, a py.test plugin for test parallelization, distributed testing and loop-on-failures testing modes
• pytest-testrail, pytest plugin for creating TestRail runs and adding results on the TestRail test management tool
• pytest-randomly, a pytest plugin to randomly order tests and control random seed (but there are different random order plugins if you search for "pytest random")
• pytest-repeat, plugin for pytest that makes it easy to repeat a single test, or multiple tests, a specific number of times. You can repeat a test or group of tests until a failure occurs
• pytest-play, an experimental rec&play pytest plugin that let you execute a set of actions and assertions using commands serialized in JSON format. Makes test automation more affordable for non programmers or non Python programmers for browser, functional, API, integration or system testing thanks to its pluggable architecture and many plugins that let you interact with the most common databases and systems. It provides also some facilitations for writing browser UI actions (e.g., implicit waits before interacting with an input element) and asynchronous checks (e.g., wait until a certain condition is true)

Python libraries for testing:

• PyPOM, python page object model for Selenium or Splinter 
• pypom_form, a PyPOM abstraction that extends the page object model applied to forms thanks to declarative form schemas

Scaffolding tools:

• cookiecutter-qa, generates a test automation project ready to be integrated with Jenkins and with the test management tool TestRail that provides working hello world examples. It is shipped with all the above plugins and it provides examples based on raw splinter/selenium calls, a BDD example and a pytest-play example 
• cookiecutter-performance, generate a tox based environment based on Taurus bzt for performance test. BlazeMeter ready for distributed/cloud performance tests. Thanks to the bzt/taurus pytest executor you will be able to reuse all your pytest based automated tests for performance tests

Pytest + Jenkins together

We've discussed about Python, pytest and Jenkins main ingredients for our cocktail recipe (shaken, not stirred). Optional ingredients: integration with external test management tools and selenium grid providers.

Thanks to pytest and its plugins you have a rich command line interface (CLI); with Jenkins you can schedule automated builds, setup a CI, let not technical users or other stakeholders executing parametrized test runs or building test always fresh test data on the fly for manual testing, etc. You just need a browser, nothing installed on your computer.

Here you can see how our recipe looks like:

Now lets comment all our features provided by the Jenkins "build with parameters" graphical interface, explaining option by option when and why they are useful.

Target environment (ENVIRONMENT)

In this article we are not talking about regular unit tests, the basis for your testing pyramid. Instead we are talking about system, functional, API, integration, performance tests to be launched against a particular instance of an integrated system (e.g., dev, alpha or beta environments).

You know, unit tests are good they are not sufficient: it is important to verify if the integrated system (sometimes different complex systems developed by different teams under the same or third party organizations) works fine as it is supposed to do. It is important because it might happen that 100% unit tested systems doesn't play well after the integration for many different reasons. So with unit tests you take care about your code quality, with higher test levels you take care about your product quality. Thanks to these tests you can confirm an expected product behavior or criticize your product.

So thanks to the ENVIRONMENT option you will be able to choose one of the target environments. It is important to be able to reuse all your tests and launch them against different environments without having to change your testware code. Under the hood the pytest launcher will be able to switch between different environments thanks to the pytest-variables parametrization using the --variables command line option, where each available option in the ENVIRONMENT select element is bound to a variables files (e.g., DEV.yml, ALPHA.yml, etc) containing what the testware needs to know about the target environment.

Generally speaking you should be able to reuse your tests without any modification thanks to a parametrization mechanism.If your test framework doesn't let you change target environment and it forces you to modify your code, change framework.

Browser settings (BROWSER)

This option makes sense only if you are going to launch browser based tests otherwise it will be ignored for other type of tests (e.g., API or integration tests).

You should be able to select a particular version of browser (latest or a specific version) if any of your tests require a real browser (not needed for API tests just for making one example) and preferably you should be able to integrate with a cloud system that allows you to use any combination of real browsers and OS systems (not only a minimal subset of versions and only Firefox and Chrome like several test platforms online do). Thanks to the BROWSER option you can choose which browser and version use for your browser based tests. Under the hood the pytest launcher will use the --variables command line option provided by the pytest-variables plugin, where each option is bound to a file containing the browser type, version and capabilities (e.g., FIREFOX.yml, FIREFOX-xy.yml, etc). Thanks to pytest, or any other code based testing framework, you will be able to combine browser interactions with non browser actions or assertions.

A lot of big fat warnings about rec&play online platforms for browser testing or if you want to implement your testing strategy using only or too many browser based tests. You shouldn't consider only if they provide a wide range of OS and versions, the most common browsers. They should let you perform also non browser based actions or assertions (interaction with queues, database interaction, http POST/PUT/etc calls, etc). What I mean is that sometimes only a browser is not sufficient for testing your system: it might be good for a CMS but if you are testing an IoT platform you don't have enough control and you will write completely useless tests or low value tests (e.g., pure UI checks instead of testing reactive side effects depending on eternal triggers, reports, device activity simulations causing some effects on the web platform under test, etc).

In addition be aware that some browser based online testing platforms doesn't use Selenium for their browser automation engine under the hood. For example during a software selection I found an online platform using some Javascript injection for implementing user actions interaction inside the browser and this might be very dangerous. For example let's consider a login page that takes a while before the input elements become ready for accepting the user input when some conditions are met. If for some reasons a bug will never unlock the disabled login form behind a spinner icon, your users won't be able to login to that platform. Using Selenium you'll get a failing result in case of failure due to a timeout error (the test will wait for elements won't never be ready to interact with and after few seconds it will raise an exception) and it's absolutely correct. Using that platform the test was green because under the hood the input element interaction was implemented using DOM actions with the final result of having all your users stuck: how can you trust such platform?

OS settings (OS)

This option is useful for browser based tests too. Many Selenium grid vendors provide real browser on real OS systems and you can choose the desired combination of versions.

Resolution settings (RESOLUTION)

Same for the above options, many vendor solutions let you choose the desired screen resolution for automated browser based testing sessions.

Select tests by names expressions (KEYWORDS)

Pytest let you select the tests you are going to launch selecting a subset of tests that matches a pattern language based on test and module names.

For example I find very useful to add the test management tool reference in test names, this way you will be able to launch exactly just that test:

c93466

Or for example all test names containing the login word but not c92411:

login and not c92411

Or if you organize your tests in different modules you can just specify the folder name and you'll select all the tests that live under that module:

api

Under the hood the pytest command will be launched with -k "EXPRESSION", for example

-k "c93466"

It is used in combination with markers, a sort of test tags.

Select tests to be executed by tag expressions (MARKERS)

Markers can be used alone or in conjunction with keyword expressions. They are a sort of tag expression that let you select just the minimum set of tests for your test run.

Under the hood the pytest launcher uses the command line syntax -m "EXPRESSION".

For example you can see a marker expression that selects all tests marked with the edit tag excluding the ones marked with CANBusProfileEdit:

edit and not CANBusProfileEdit

Or execute only edit negative tests: 

edit and negative

Or all integration tests

integration

It's up to you creating granular keywords for features and all you need for select your tests (e.g., functional, integration, fast, negative, ci, etc).

Test management tool integration (TESTRAIL_ENABLE)

All my tests are decorated with the test case identifier provided by the test management tool, in my company we are using TestRail.

If this option is enabled the test results of executed tests will be reported in the test management tool.

Implemented using the pytest-testrail plugin.

Enable debug mode (DEBUG)

The debug mode enables verbose logging.

In addition for browser based tests open selenium grid sessions activating debug capabilities options (https://www.browserstack.com/automate/capabilities). For example verbose browser console logs, video recordings, screenshots for each step, etc. In my company we are using a local installation of Zalenium and BrowserStack automate.

Block on first failure (BLOCK_FIRST_FAILURE)

This option is very useful for the following needs:

• a new build was deployed and you want to stop on the very first failure for a subset of sanity/smoke tests
• you are launching repeated, long running, parallel tests and you want to block on first failure

The first usage let you gain confidence with a new build and you want to stop on the very first failure for analyzing what happened.

The second usage is very helpful for:

• random problems (playing with number of repeated executions, random order and parallelism you can increase the probability of reproducing a random problem in less time)
• memory leaks
• testing system robustness, you can stimulate your system running some integration tests sequentially and then augment the parallelism level until your local computer is able to sustain the load. For example launching 24+ parallel integration tests on a simple laptop with pytest running on virtual machine is still fine. If you need something more heavy you can use distribuited pytest-xdist sessions or scale more with BlazeMeter

As you can imagine you may combine this option with COUNT, PARALLEL_SESSIONS, RANDOM_ENABLE and DEBUG depending on your needs. You can test your tests robustness too.

Under the hood implemented using the pytest's -x option.

Parallel test executions (PARALLEL_SESSIONS)

Under the hood implemented with pytest-xdist's command line option called -n NUM and let you execute your tests with the desired parallelism level.

pytest-xdist is very powerful and provides more advanced options and network distributed executions. See https://github.com/pytest-dev/pytest-xdist for further options.

Switch from different selenium grid providers (SELENIUM_GRID_URL)

For browser based testing by default your tests will be launched on a remote grid URL. If you don't touch this option the default grid will be used (a local Zalenium or any other provider) but in case of need you can easily switch provider without having to change nothing in your testware.

If you want you can save money maintaining and using a local Zalenium as default option; Zalenium can be configured as a selenium grid router that will dispatch capabilities that it is not able to satisfy. This way you will be able to save money and augment a little bit the parallelism level without having to change plan.

Repeat test execution for a given amount of times (COUNT)

Already discussed before, often used in conjunction with BLOCK_FIRST_FAILURE (pytest core -x option)

If you are trying to diagnose an intermittent failure, it can be useful to run the same test or group of tests over and over again until you get a failure. You can use py.test's -x option in conjunction with pytest-repeat to force the test runner to stop at the first failure.

Based on pytest-repeat's --count=COUNT command line option.

Enable random test ordering execution (RANDOM_ENABLE)

This option enables random test execution order.

At the moment I'm using the pytest-randomly plugin but there are 3 or 4 similar alternatives I have to try out.

By randomly ordering the tests, the risk of surprising inter-test dependencies is reduced.

Specify a random seed (RANDOM_SEED)

If you get a failure executing a random test, it should be possible to reproduce systematically rerunning the same tests order with the same test data.

Always from the pytest-randomly readme:

By resetting the random seed to a repeatable number for each test, tests can create data based on random numbers and yet remain repeatable, for example factory boy’s fuzzy values. This is good for ensuring that tests specify the data they need and that the tested system is not affected by any data that is filled in randomly due to not being specified.

Play option (PLAY)

This option will be discussed in a dedicated blog post I am going to write.

Basically you are able to paste a JSON serialization of actions and assertions and the pytest runner will be able to execute your test procedure.

You need just a computer with a browser for running any test (API, integration, system, UI, etc). You can paste how to reproduce a bug on a JIRA bug and everyone will be able to paste it on the Jenkins build with parameters form.

See pytest-play for further information.

If you are going to attending next Pycon in Florence don't miss the following pytest-play talk presented by Serena Martinetti:

• https://www.pycon.it/conference/talks/integration-tests-ready-to-use-with-pytest-play

UPDATES:

• [20180424] pytest-play introduction http://davidemoro.blogspot.it/2018/04/hello-pytest-play.html
• [20180222] pytest-play 2.0.0 released, no more compatible with 1.x format (using YAML instead of JSON and other improvements). See a detailed article showing how to use pytest-play for API REST testing using plain YAML files https://davidemoro.blogspot.com/2019/02/api-rest-testing-pytest-play-yaml-chuck-norris.html

How to create a pytest project

If you are a little bit curious about how to install pytest or create a pytest runner with Jenkins you can have a look at the following scaffolding tool:

• cookiecutter-qa

It provides a hello world example that let you start with the test technique more suitable for you: plain selenium scripts, BDD or pytest-play JSON test procedures. If you want you can install page objects library. So you can create a QA project in minutes.

Your QA project will be shipped with a Jenkinsfile file that requires a tox-py36 docker executor that provides a python3.6 environment with tox already installed; unfortunately tox-py36 is not yet public so you should implement it by your own at the moment.
Once you provide a tox-py36 docker executor the Jenkinsfile will create for you the build with parameters Jenkins form for you automatically on the very first Jenkins build for your project.

Conclusions

I hope you'll find some useful information in this article: nice to have features for test frameworks or platform, a little bit of curiosity for the Python world or  new pytest plugin you never heard about.

Feedback and contributions are always welcome.

Tweets about test automation and new articles happens here:

• @davidemoro

Combinatorial testing: ACTS

Last year I discovered the ACTS tool thanks to Rex Black's "Pairwise Testing with ACTS" webinar:

•  https://rbcs-us.com/blog/pairwise-testing-with-acts-available-now/

and I find it very helpful when you have to deal with non trivial systems, you fear to miss something of important or there is a combinatorial explosion.

ACTS let you create a model for your system defining parameters, boundary values, different types of inputs, valid and invalid data, relations between parameters and different generation algorithms, base choices (e.g., more important values you want to be used most often).

More information from the ACTS used guide:

ACTS is a test generation tool for constructing t-way combinatorial test sets.
Currently, it supports t-way test set generation with t ranging from 1 to 6.
Combinatorial testing has been shown very effective in detecting faults that are caused by unexpected interactions between different contributing factors.

Always from the ACTS user guide:

A system is specified by a set of parameters and their values. A test set is a t-way test set if it satisfies the following property: Given any t parameters (out of all the parameters) of a system, every combination of values of these t parameters is covered in at least one test in the test set.
Currently, ACTS supports t-way test set generation for 1 = t = 6. Empirical studies show that t being up to 6 is sufficient for most practical applications. A special form of 1-way testing, called base-choice testing, is implemented in ACTS. Base-choice testing requires that every parameter value be covered at least once and in a test in which all the other values are base choices. Each parameter has one or more values designated as base choices. Informally, base choices are "more important" values, e.g., default values, or values that are used most often in operation.

So ACTS helps you to drastically reduce the testing combinations for moderate or very large systems. The most important ACTS output is the test set generation (e.g., a calc/Excel export file). The coverage function from the statistics tab is interesting too.



Since we have finite time and infinite test combinations, we have to be very
selective and ACTS is a valid help.

I find that the ACTS test cases export sometimes is not readable and you loose the base choice info, invalid data and you might be lost in combinations. I think it is a good idea adding some simple formulas and conditional formatting for improving the readability of the export, see base choices, distinguish between positive and negative scenarios (e.g., yellow), emphasize when you get the same output from different paths, visualize different shapes of test cases and convey more meaningful information about your data depending on your domain.

This way you will be able to detect more easily what are the most important cases and prioritize them (e.g., feature confirmation and guessing the test cases with a higher probability of failure) for both positive and negative cases to be as effective as possible even if you have very little time for testing.

Obviously before creating the ACTS project you should already have an idea of what are the most important scenarios and all the possible ways for obtaining a certain result but thanks to ACTS you can have a visual double check and see if you missed something of important or other edge cases.

When you identify the most interesting rows you can add a description note explaining why this particular scenario is important, prioritize all your test cases and add all the needed information and adding new scenarios or edge cases. The file is ready to be imported to a test management tool now and you have also an ordered execution plan based on priority and risk.

And if you don't have enough time for testing the exported combinations you might evaluate if you have sufficient information to realize if the most relevant cases are covered, the level of confidence, coverage and you can evaluate if the residual risk is acceptable.

This test set could be also used for automating your scenarios. If you use for example a keyword driven testing pattern or more abstract patterns you can implement just one time the needed logics and with little time you will cover all your hundreds of combinations because your test framework let you decouple the test data from the test implementation. In following articles we'll discuss about pytest fixtures/parametrization, pytest-bdd and other test frameworks. Stay tuned!

Links:

• https://www.nist.gov/programs-projects/automated-combinatorial-testing-software-acts
• https://rbcs-us.com/blog/pairwise-testing-with-acts-available-now/
• https://www.istqb.org/downloads/category/48-advanced-level-test-automation-engineer-documents.html

Introducing substancek. A Kotti project

Let me introduce substancek, a Kotti (http://kotti.pylonsproject.org) project.

What it is substancek?

substancek is:

• Kotti (web application framework) + decoupled admin interface

It is only an additional layer upon the following opinionated stack:

• kotti_backend, a plugin that turns Kotti to a private admin interface (backend). See https://github.com/Kotti/kotti_backend
• Kotti, web development framework based on Pylons/Pyramid with workflows support, content types, security). See http://kotti.pylonsproject.org
• Pylons/Pyramid, one of the most flexible Python web frameworks. See http://www.pylonsproject.org
• SQLAlchemy, the Python SQL Toolkit and Object Relational Mapper. See http://www.sqlalchemy.org
• Python. See https://www.python.org

with the following motto:

"""(even) better development experience and complete frontend freedom"""

and introduces (or better promotes) the concept of private admin area (backend) decoupled from the public side (frontend) of your web applications built with Kotti.

In other words it is a set of technologies addressed under the substancek brand that let you extend Kotti in order to use it just as a private backend administration area for your application data.

So you are still using plain Kotti with an additional package (at least kotti_backend depending on what you need).

If you want to know more I've discussed here benefits and why frontend decoupled from the backend pattern. See http://davidemoro.blogspot.it/2015/07/kotti-cms-successful-story-part-1.html

substancek name

Tribute to:

• substanced (http://www.substanced.net). Carefully designed, well documented Python web development framework based on Pyramid/ZODB. It shares the concept of management views decoupled from the retail views for each published resource
• kotti (http://kotti.pylonsproject.org). The Kotti framework

When substancek is for you

Any project of any size (from micro to XXL) involving content management that needs:

• RDBMS. Kotti is an opinionated framework opposed to plain Pyramid. The persistence layer is managed by SQLAlchemy (http://www.sqlalchemy.org)
• user friendly and production ready admin/editing interface for content producers so they can immediately start adding content to your web application with copy/cut & paste, clean urls, etc
• rapid development. Pyramid and Kotti are easy to learn and let you become productive quickly and speed up your development. Even more with the decoupled admin interface.
• stability. Pyramid and Kotti exist since many years and they are solid rock solutions production ready
• frontend freedom. The admin interface comes for free while it is completely up to you add what your application needs (retail views). Since they are completely two different applications there are no css/js conflicts and you can integrate your preferred frontend tool chain without get influenced by how the admin interface is built.
• flexibility. Pyramid won't hold you back when your application is small and won't get in your way when your application becomes large.
  Even if you start small, using Pyramid you anticipate what customers will need later avoiding expensive rewrites and this can make the difference. In other words this also means successful projects in the long term. Same for Kotti. See What makes Pyramid unique.

So if you project needs (or in future iterations) one or more:

• complex security policies
• workflows
• hierarchical data support
• or even intranet/extranet like collaboration areas

you might consider even more substancek (kotti_backend + Kotti + Pyramid + SQLAlchemy).

For example:

• very small applications. For example a just one view public json endpoint for published news-like resources consumed by a third party app with an admin interface for editing contents
• heavy Javascript based applications with modern frontend tools (eg: SPA + REST) with a decoupled admin interface
• content management solutions
• blog
• ecommerce
• intranets
• large CMS-ish applications

Note well: if you don't need workflows don't be scared because there is no overkill. You can use a one state workflow or no workflow at all for example. No hierarchical data? Use not nestable resources and so on. If it comes out later that you need them it will be quite easy converting your code.

Alternatives

You can use plain Kotti, without the substancek's kotti_backend setup. Or if you prefer noSQL try the excellent substanced (substanced + Pyramid + ZODB). Both solutions are lightweight, tested, well documented and easy to learn. Alternatively if you need really a minimal and unopinionated solution you might use plain Pyramid.

Do you need something more? You might consider to use Plone (https://plone.org) as a framework.

Anyway the good news is that Python is plenty of good options.

substancek architecture details

As already told you the private admin area (backend) and the rest of the application (frontend) are two complete different applications with different settings, different views and shared authentication.

Assuming you are going to use PasteDeploy to run your application, let's consider the following configuration files setup:

• backend-dev.ini, configures the private admin interface based on Kotti thanks to the kotti_backend plugin
• frontend-dev.ini, configures your application you are developing (a full CMS frontend implementation or a microapp with just one retail view).
• development.ini (optional), mount the backend-dev.ini and frontend-dev.ini applications in the same process (/admin for the private admin interface and / for your application). Alternatively you can run the frontend and backend using two processes waiting for requests on different ports and play with rewrite rules.

backend-dev.ini

[app:kotti]
use = egg:kotti

...
pyramid.includes =
    pyramid_debugtoolbar
    pyramid_tm
    kotti_backend.views.override_root_view


kotti.configurators = kotti_tinymce.kotti_configure
    kotti_backend.kotti_configure

kotti.use_workflow = kotti_backend:workflows/simple_backend.zcml

kotti_backend.goto_frontend = 1 

This is a normal Kotti setup with:

• enabled the (optional) root view override for our admin interface. In other words the default root view will be @@contents instead of the standard Kotti's hello page (see pyramid.includes)
• added kotti_backend to kotti.configurators
• override the Kotti's default workflow with the one provided by kotti_backend (kotti.use_workflow). Playing with the additional pview permission you can decide your resources visibility on the frontend. See the workflow definition here https://github.com/Kotti/kotti_backend/blob/master/kotti_backend/workflows/simple_backend.zcml
• enable a "Goto frontend" link (kotti_backend.goto_frontend) for easy switching from admin interface and frontend 

See more options on the kotti_backend's README file:

• https://github.com/Kotti/kotti_backend

frontend-dev.ini

[app:main]
use = egg:Kotti
...

kotti.use_workflow = kotti_backend:workflows/simple_backend.zcml

kotti.configurators =
    your_package.kotti_configure

kotti.base_includes =
    kotti
    kotti.views

On the frontend configuration file we share the same workflow in use on the admin interface (kotti.use_workflow).

One of the most important configuration is the kotti.base_includes override: here we decide what will be loaded on our application. We omit all the Kotti views loaded by default in the standard setup and we load what we want to include where:

• kotti, loads the kotti "core"
• kotti.views (optional), load some view discriminators and utils defined by Kotti if you need them
• your_plugin.your_includes, load your includes registering the views needed by your application

The kotti.configurators typically auto includes your package and tell what should be included in your application (pyramid.includes). See the Kotti documentation for more info.

In other words:

"what is not loaded, it doesn't exist"

so the final result is that there is nothing exposed on the frontend except what you decide to load and extreme control. You can register just only one view application or configure a complex setup for a CMS-like application: it's up to you registering only the views your application needs and no more. This way you can use completely different frontend frameworks, different versions of Javascript libraries, you have no css/js conflicts and no need to hide unneeded things and you decide which resources will be published on the frontend.

See also another advance usage pattern "Using Kotti as a library" http://kotti.readthedocs.org/en/latest/developing/advanced/as-a-library.html

development.ini

# See http://pythonpaste.org/deploy/#paste-composite-factory
[composite:main]
use = egg:Paste#urlmap
/ = config:frontend-dev.ini
/admin = config:backend-dev.ini

[server:main]
use = egg:waitress#main
host = 127.0.0.1
port = 5000

The (optional) development.ini shows how to configure a composite application with different mount points. You can change /admin with /cms or /whateveryouwant depending on your needs.

Examples

You can checkout the https://github.com/substancek/substancek_cms_theme package if you want to see in action a (quite complex) example.

I'm going to provide more and simpler examples (eg: a pretend micro application), see the roadmap.

What are the substancek related packages

Here you can see the whole substancek ecosystem:

• kotti_backend, generic package that turns Kotti to a private admin area. This is the keystone for every substancek like project
• CMS-like applications
• substancek_cms (roadmap), a Kotti CMS distribution with an alternative frontend theme based on SASS, html/templates minification and assets optimization based on Yeoman tools. It will be based on the kotti_project experiment available here https://github.com/davidemoro/kotti_project
• substancek_cms_theme, an example public side CMS implementation based on Kotti built with SASS and html/templates minification
• kotti_actions, link actions backend implementation for header, footer and main navigation links
• kotti_boxes, portlets and box backend implementation
• kotti_es, elastic search support (to be refactored)
• common utilities
• pyramid_html_minifier, generic Pyramid package that introduces Chameleon templates minification with no overhead. Required by substancek_cms_theme.
• build_commands, generic Python package that defines a set of initialization commands on install (eg: python setup.py npm/bower/gulp) for frontend stuff. Required by substancek_cms_theme.

Who's using substancek technologies

MIP - graduate school of business

The MIP (Politecnico di Milano graduate school of business - www.mip.polimi.it/en) uses substancek technology inside for the private admin interface. This approach is so flexible that let you use Kotti as a private admin content management area and even implement your public views using other frameworks or non Python languages (for example PHP+Symfony2).

See:

• http://davidemoro.blogspot.com/2015/07/kotti-cms-successful-story-part-1.html
• http://davidemoro.blogspot.com/2015/07/kotti-cms-frontend-decoupled-backend-part-2.html

substancek_cms_theme

This is a work in progress opinionated CMS frontend implementation that reuses existing Kotti templates and logics.

The look and feel is the same you get with a standard Kotti installation but it shows how to integrate and distribute a Python package integrated with a Yeoman setup (http://yeoman.io) that provides:

• production vs development setup. In production mode will be used assets and templates from the dist
• Chameleon templates minification with no overhead thanks to pyramid_html_minifier when you are in "production" mode
• SASS
• image/assets optimization
• build commands for npm/bower/gulp initialization thanks to the generic package build_commands

Once installed you'll see the admin interface visiting http://localhost:5000/cms.
See the code here: https://github.com/substancek/substancek_cms_theme

Next steps

If you want to contribute there is a lot to do:

• contributions to the Kotti core (for example @@contents pagination, REST, etc and new ideas)
• creation of the substancek_cms package that puts together the substancek_cms_theme default theme plus the most useful third party Kotti plugins like news, events, etc with decoupled admin interface and Vagrant/Ansible automated setup based on kotti_project for easy installation/evaluation
• creation of simple pretend packages, add more examples (microapp, blog)
• a toolbar available on the frontend just of editor users for improved usability (the same shown in the MIP case history) implemented with a Pyramid tween
• more work on substancek_cms_theme, implement advanced features shown in the MIP case history like link managers and portlets and refactor the Yeoman folder removing what is not strictly needed
• scaffolding, help people create new projects with decoupled admin interface using Kotti as a framework
• substancek dedicated github page

Contributions, feedback or pings like "hey, I'm going to use Pyramid/Kotti for the my next project" will be very appreciated!

Documentation

• http://kotti.readthedocs.org/en/latest
• http://pylons.readthedocs.org/en/latest

All Kotti posts published by @davidemoro

• http://davidemoro.blogspot.it/search/label/kotti

Twitter links

• https://twitter.com/pylons
• https://twitter.com/KottiCMS
• https://twitter.com/davidemoro

Kotti CMS - frontend decoupled from backend. How we did it (part 2)

In the previous article http://davidemoro.blogspot.it/2015/07/kotti-cms-successful-story-part-1.html we have seen that:

• decoupled public website from the private content management is cool
• Python and Pyramid is cool
• don't use PHP please :)

No we will see:

• how to build a pure Python Kotti based setup with a private content management area decoupled from the public website (with tips, links, technical details and screenshots)

Here you can see some screenshots, implementation details and links.

Project setup

The installation folder is a package that contains all the application-specific settings, database configuration, which packages your project will need and where they lives.

From https://pypi.python.org/pypi/mr.developer.

The installation folder is a "one command install" meta package:

• replicable
• under version control + git flow
• documented + change list for each plugin and the project itself
• with a tag based deploy
• based on Python pip/requirements.txt
• batteries included
• exceptions logging on file (with logrotate) enabled by default
• automated deploy

so let the computer works for us and have fun.
See:

• http://davidemoro.blogspot.com/2015/04/pip-for-buildout-folks.html
• https://github.com/davidemoro/kotti_project
• http://davidemoro.blogspot.com/2015/03/pyramid-exceptions-logging.html

Populators

Populators are functions with no arguments that get called on system startup, they may then make automatic changes to the database like content initialization.

Populators are very important because when you install the project folder during development or on the very first production instance you'll find all the most important contents and sections by default. Things will be created automatically if the database is empty, so you don't obtain a blank site on the very first install.

Populators are also good for improving the first impact of the end users (I mean editors) with the platform because they see all the main sections already there.

See:

• http://kotti.readthedocs.org/en/latest/developing/basic/configuration.html#kotti-populators

Private backend area

Turning Kotti CMS into a private content administration is quite easy:

• http://davidemoro.blogspot.com/2015/02/kotti-cms-intranet.html
• http://davidemoro.blogspot.com/2015/02/kotti-cms-workflow-reference.html
• http://kotti.readthedocs.org/en/latest/developing/basic/security.html

Later I've created a generic package that does all the things for you (kotti_backend):

• https://github.com/Kotti/kotti_backend

so things are even easier now (install kotti_backend, done).

Multilingual

kotti_multilingual is your friend.

Goto frontend link, translation management and link to the technical documentation online based on Sphinx

See:

• https://github.com/Kotti/kotti_multilingual 
• http://davidemoro.blogspot.com/2015/04/kotti-multilingual.html

Elastic search

kotti_es provides ElasticSearch integration for fulltext search. This plugin needs more love and and a complete refactor (it was built in a hurry and I'm not yet satisfied) but it proved there are no known issue after months of intensive usage.
Probably things will change, hope other guys with the same needs will contribute.

See:

• https://github.com/Kotti/kotti_es
• http://davidemoro.blogspot.com/2015/05/kotti-cms-elasticsearch-integration.html

Main navigation and header/footer links

You can use the kotti_actions plugin if you want to implement footer, header links or even nested main navigation menus. Obviously kotti_actions is ment to be used with a decoupled frontend.

As you can see a custom colour can be assigned to courses, navigation links, sections and every kind of object thanks to the json annotations column provided by default by Kotti. So you can add arbitrary fields.

How the multilevel menu looks like on the public website

See:

• https://github.com/Kotti/kotti_actions
• http://davidemoro.blogspot.com/2015/02/kotti-cms-annotations-store-arbitrary-data.html

Portlets

The main layout based on box managers for portlets

The kotti_boxes is your friend. This plugin is ment to be used with a decoupled frontend. And it was quite quick implementing portlets because we didn't need to customize the private backend area.

You can define different page layouts for each resource type (home page, news, courses, etc) and show boxes in well defined areas (box managers), for example LeftBoxManager, AboveFooterBoxManager and so on.

So box and box managers are just non publishable contents and you can:

Banner portlets with links

• copy/paste them
• assign workflow with different security policies to box and box managers
• assign different views
• share edit permission to certain box or box managers to particular users or groups
• prevent certain type of boxes to be included in some areas (for example: banner box image only addable to the left box manager).

See:

• https://github.com/Kotti/kotti_boxes

Editor toolbar

As you can see if you are logged in the frontend will show an editor toolbar with:

• link to the backend version of the page
• object information (state, type of object, etc)

Info and links to the backend, edit and folder contents

or see exactly the website as an anonymous user (very common customer request):

Anonymous view

You can also add more features, for example direct edit links for images or portlets or live edit features.

Talking about a pure Python solution, you might implement this feature with a Pyramid Tween (I hope I'll have enough spare time to do that. Anyone would want to contribute? We are very welcome, contact me!):

• http://docs.pylonsproject.org/projects/pyramid//en/latest/glossary.html#term-tween
• http://docs.pylonsproject.org/projects/pyramid//en/latest/narr/hooks.html#registering-tweens

Course types (custom content types)

The course view with portlets and collapsable paragraphs

They are a sort of rich documents with an image attachment column and integrated with an external ecommerce site. When you add a course type there is an event that initializes automatically subobjects and the main image attachement by default, so less work for editors.

In addition all image content types and image attachments are addable or editable just by allowed users thank to custom roles and global or local permission sharing.

Collapsable paragraphs are implemented with custom content types not directly reachable on the frontend.

There are a lot of fields on this content type, so they are grouped together using fieldsets.
Editors can also create a base private course model and then copy and paste it when new courses should be added.

Sometimes you want to prevent the addability on the site root for particular object types, this way things will remain always tidy (why you should add a course on the very root of the site?).

See:

• http://davidemoro.blogspot.com/2015/02/kotti-cms-how-to-create-new-content-image.html
• http://davidemoro.blogspot.com/2015/03/kotti-cms-events-insert-subobjects.html
• http://davidemoro.blogspot.com/2015/05/kotti-cms-how-to-add-new-roles.html
• http://davidemoro.blogspot.com/2015/01/kotti-avoid-types-addable-in-content.html
• http://kotti.readthedocs.org/en/latest/developing/basic/security.html

Windows and MySQL issues and tips

Kotti can be installed on Windows but I strongly suggest to adopt a Unix-like server with Postgresql instead of MySQLas storage layer: 

• http://davidemoro.blogspot.com/2015/03/pyramid-mysql-windows-good-ugly-bad.html
• http://davidemoro.blogspot.com/2015/01/how-to-install-kotti-cms-on-windows.html
• http://davidemoro.blogspot.com/2015/03/pyramid-exceptions-logging.html 

Tests

All the software is tested. Very happy with the py.test framework.

See:

• http://pytest.org/latest/
• http://davidemoro.blogspot.it/2015/02/python-mock-library-for-ninja-testing.html

Other third party Kotti plugins

I've used the following third party plugins that can be used on a standard Kotti environment:

• kotti_calendar for events (very good solution!)
• kotti_newsitem for news
• kotti_navigation, add a configurable navigation on our backend
• kotti_link, link types

See also the full list of available plugins:

• https://pypi.python.org/pypi?%3Aaction=search&term=kotti&submit=search

Photoes and credits

All the screenshots shown in this article are taken from the "MIP Politecnico di Milano's graduate school of business" website:

• http://www.mip.polimi.it

So the MIP's website backend is powered by Pylons/Pyramid and Kotti CMS, I'll write a non-technical case study soon. In the mean time many thanks to:

• MIP
• Simona Strepparola, Head of Communication
• Gabriele Bedani, Microsoft Sysadmin
• all the MIP staff
• Truelab 
• Riccardo Bonini, project manager
• Ruben Barilani, web developer
• Caio Ceccon, web developer (Faculty and Staff implementation). Python developer that joined us for only 5 days but enough to become good friends! He never touched a Pyramid/Pylons or Kotti application but he was able to be productive in a couple of days, this proves the Pyramid/Pylons developer friendliness
• Davide Moro, it's me
• Andrea Sironi, creative director

Results

You can consider Kotti as a very good, secure, flexible, battle tested and easy to approach solution for important customers.

All Kotti posts published by @davidemoro

• http://davidemoro.blogspot.it/search/label/kotti

Next steps

Reading this article you should find all the bricks you need if you want to implement a public website decoupled from its backend with Kotti.

Now I'm assembling the above bricks in order to provide an "easy to install" solution with the same pattern I've described here. This is my roadmap:

• kotti_backend (https://github.com/Kotti/kotti_backend). Already published on PyPI, ready to be used. Turns Kotti CMS into a private content administration area
• kotti_frontend (https://github.com/Kotti/kotti_frontend). Public website solution completely decoupled by the Kotti backend with modern frontend tools (I want to obtain something like http://davidemoro.blogspot.it/2014/09/plone-angularjs-yeoman-starter.html or http://davidemoro.blogspot.it/2014/09/pyramid-starter-seed-yeomam-part-1.html). Optional support for blank sheet themes if you need more flexibility. It is just an incomplete but quite promising prototype, it is still under development.
  UPDATE 20150824: no more kotti_frontend. See http://davidemoro.blogspot.com/2015/08/introducing-substancek-kotti-project.html and https://github.com/substancek/substancek_cms_theme
• a new pyramid tween implementing the editor toolbar available on the public website (TODO)
• a meta-package "one command" installer based (a modified version of https://github.com/davidemoro/kotti_project). Still in TODO
• installable themes and themes generation (scaffolding)
• a Kotti distribution that provides all the most useful third party plugins, just if you need a CMS. Now you have to install all external components manually.
• REST interface support (included in standard Kotti's roadmap), very useful if you want to built heavy Javascript based applications 

It can be considered a good starting point for:

• CMS-ish application
• a generic framework for building generic web apps or single page web applications (remember, Kotti is not only a CMS, it could be considered as a framework)

So stay tuned and if you like this work please consider to contribute with

• code
• testing
• issue reporting
• Github stars
• spreading the word 

or why not sponsorships!

And if you want to know more about Kotti and you are attending +EuroPython Conference 2015 in Bilbao don't miss the Andreas Kaiser's talk "Standing on the Shoulders of Giants: The Kotti Web Application Framework". I'll join the sprint (remotely) at the end of EuroPython so see you on IRC (http://webchat.freenode.net/?channels=kotti). If you want to stay tuned follow https://twitter.com/KottiCMS.

Kotti CMS - a successful story (part 1)

Yet another Kotti CMS (http://kotti.pylonsproject.org) article.

We'll see in this blog post:

• advantages of the public website decoupled (frontend) from the private content administration area (backend) pattern

Furthermore in this article you might find traces of:

• PHP vs Python comparison
• Symfony2 vs Pylons/Pyramid
• Plone vs Kotti
• Python evangelism

But before starting let's explain what is the background of this true story and how Kotti CMS helped a lot.

Update 20150718

Part 2 is out! See:

• frontend decoupled from backend. How we did it (part 2)

The challenge

That were the requirements for an important project I joined last January:

• build a new CMS from scratch (yeah, I know, it doesn't make sense reinventing the wheel: see How to choose your CMS)
• based on PHP/Symfony
• with only two (2) developers
• end user friendly
• through the web customization for visual sections by editors/administrators (home page text introduction, main navigation, header and footer links, boxes, etc).
• with an external ecommerce integration
• custom logics and behaviours
• custom security based on roles, workflow, permission sharing
• configurable portlets system (manageable columns)
• custom navigation link managers
• multilingual
• elasticsearch integration
• nested urls (folders hierarchy)
• stable

... with a not postponable 3 months deadline!

Obviously there was a previous huge problem at commercial/estimation level and Peppa Pig would have said the famous "THIS IS IMPOSSIBLE!" because build a new CMS is an incredibly enormous task... but let's see why we are still alive and how it was possible to meet these impossible requirements with hard work, a bit of (italian) inventiveness and the Python/Pyramid based Kotti CMS.

1 - Does anyone already found a solution to a similar problem?

Not listed here:

• http://www.chucknorrisfacts.com
• http://macgyver.wikia.com/wiki/List_of_problems_solved_by_MacGyver

but I think the chucknorrisfacts guys should update their site. Obviously Chuck Norris can solve this problem, he can also prevent it with a roundhouse kick.

2 - Switch technology to Plone attempt

Now I'm serious :)

The first thought was: try to change the technology stack switching to Plone (http://plone.org) but the Windows/PHP/Symfony/MySQL stack was not an option, so no Plone for this project.

3 - Have a look at existing PHP-based CMS solutions

Second step, check if there is a decent CMS solution built with PHP with this requirements:

• secure
• easy to extend and maintain
• with a good and modern codebase
• with an intuitive backend interface for editors

and we were not able to find any existing solution matching our requirements criteria.
In addition introducing a new framework usually means more time and a lot of pain if you have to customize almost everything if the framework it is not built with flexibility in mind.

4 - try out Sonata + Symfony2 from scratch

We tried to setup a Symfony2 project powered by Sonata and we had a look at to existing early stage "CMS"s Sonata based but it was clear that it wasn't the right tool for building a real content hierarchy aware CMS usable by end users. Trust me, CMS is another sort of thing: you cannot call CMS the ugly version of the Django admin or something that is not suitable for end users. Anyway, Sonata is a quite good solution if you are going to write a data administration area for your custom Symfony2 application. See http://symfony.com/doc/1.0/cmf/cookbook/creating_cms_using_cmf_and_sonata.html and http://demo.sonata-project.org/.

5 - The solution

It was clear that it was impossible for us building a heavy customized CMS with almost "enterprise" requirements based on PHP for our team composition, experience, tight deadlines, etc.

So the solution was: why not adopting a good CMS based on a relational database storage powered by Python (there are very good options) and use it as a content administration backend area with a decoupled frontend built with Symfony2/PHP?

We had a look to Django and Pyramid CMS solutions and Kotti seemed the best option:

• very good code base (and its core it is damn small)
• powered by Pylons/Pyramid (quite small, flexible, easy to extend and approachable Python web framework with url traversing capabilities for free)
• clean user interface
• powered by SQLAlchemy
• very very flexible if you need to turn the whole system inside out
• many good concepts available in Kotti or Pyramid/Pylons are largely inspired by the Zope/Plone world (two decades of successful web applications and content management experience there)
• it is not only a CMS but a web development framework

This way we were able to be extremely efficient and take advantage of our expertise from the very first day:

• 1 PHP developer on the PHP-Symfony frontend side
• 1 Python developer on the Python Kotti CMS backend

As we will see that was our "winning" solution but I don't consider it optimal: this pattern itself is clean, powerful and I think I'll adopt the same technique in the next future in pure Python. Effectively there were some (well known) problems with our setup:

• PHP is not Python
• Symfony2 is not Pylons/Pyramid (security, routing, traversal, view declarations with discriminators)
• Kotti is not Plone (very good solution but it is still quite minimal), so I had to implement the missing parts we need. The good news is that Kotti and Pyramid you are very productive
• but... Pylons/Pyramid is Pylons/Pyramid! Very happy and damn productive, I really enjoyed programming with this framework. Very impressed.

PHP is not Python

Concepts like decorators not available as PHP builtin: cool things like decorators are only comments that are "compiled" later. And it is more hard implementing new "decorators" and test them compared to Python.

It seems that you cannot set to a class an instance of another class (our Kotti's type_info for example).

Doctrine or Propel works fine if you are adopting the Docrine or Propel pattern. If you need something of different, no way. We wrote a custom abstraction on the top of the pattern used by Kotti and SQLAlchemy for types inheritance with joined tables (!!). Last but not least, SQLAlchemy provides things like:

• association_proxy (used in Kotti tags). See http://docs.sqlalchemy.org/en/latest/orm/extensions/associationproxy.html
• other techniques used by kotti_multilingual used for independent fields. See https://github.com/Kotti/kotti_multilingual/blob/master/kotti_multilingual/sqla.py

and it feels quite magic because with PHP solutions the same patterns are not replicable so easily.

Bit operations math for cookies shared auth among Symfony2 frontend and Kotti CMS backend.

Generally it was a bit frustrating having to write a lot of code what you get for free with Python with just 2 lines of code.

Symfony2 is not Pylons/Pyramid

What is Symfony2: it is one of the better frameworks available in the PHP world, probably the most promising. It is not a monolithic framework, you can also use standalone symfony components as well in your non-Symfony PHP projects (many famous existing PHP projects are switching to Symfony). Symfony2 plays well with other existing components like ORMs and it tries to bring innovation to PHP.
It promotes best development practise, testing, modularity, extensibility. It has its own dependency injection system and it let you write templates with a quite good template system named Twig.

Anyway, quite good ideas with concepts stolen from other non-PHP framework, in particular dependency injection plus a configuration hooks if you want to replace existing components without having to modify an existing application (but no component adapter patterns). So with Symfony you can write good code, despite PHP.

Yeah, Symfony2 is quite good and it tries to bring innovation to the PHP world: very good. But if you feel innovative and you like so much innovation why not switching to other technologies that were already more innovative 20 years ago?! And now things are even better. Programming languages like Python and web frameworks like Pylons/Pyramid are on another planet: they really seem built by aliens compared to other PHP frameworks.

For example with Python you have:

• package managers like pip that exists since many years
• ORM with extreme flexibility (SQLAlchemy is on another planet too). Doctrine or Propel works fine just if you follow the Doctrine/Propel way, if you need alternative patterns they might not follow your needs with a lot of pain
• less security vulnerabilities (see https://plone.org/products/plone/security/overview)
• easy security setup for your applications
• component adapter patterns
• easy debugging (PDB)
• routes and above all concepts like traversal (see  http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/muchadoabouttraversal.html)
  
• clean syntax and builtin concepts like decorators and context managers
• no random byte code cache corruption
• more productivity

Installation, setup, deploys and installation management seems more simpler too with Python in my opinion. If you setup your projects with pip you can deploy your applications with a couple of commands:

$ git fetch
$ git checkout tags/0.1.5
$ pip install -r requirements.txt
$ restart your application

or if you have complex requirements you can have a look at buildout.
See:

• http://davidemoro.blogspot.com/2015/04/pip-for-buildout-folks.html

So it is no more time "with PHP you just copy and paste the code" if you are doing things seriously as you can see: http://symfony.com/doc/current/cookbook/deployment/tools.html
Probably complex is the wrong word, it would be better verbose. For verbose I mean: composer install with a lot of options and dump-autoload, clear the Symfony2 prod cache avoiding cache corruption issues and install assets, cache warmup, be sure there are right permissions for writeable directories, etc.
So the "just copy and paste the code" myth (#1) with PHP is not true anymore and I don't like it.

And the very first PHP initial setup on your computer is not so easy. I think for a newbie it is much more easy a different one or two commands approach: install requirements and launch one command.
If you have to start with a PHP/Symfony2 setup just for developing is quite a long and error prone task. That's why my very first experiment with PHP-Symfony2 was: adopt Vagrant/Ansible for environment setup and provisioning. If you need a Vagrant/Ansible example you can have a look at here: https://github.com/davidemoro/symfony-vagrant-ansible.

Another false myth (#2): the most used technology it is better than others: no.

Yet another false myth (#3): the most used technology is more secure: no, absolutely not. See again https://plone.org/products/plone/security/overview.

Let me show other things you can do better with Python:

• "byte code cache" corruption issues
• missing traversal concept
• views registration and override
• debugging
• security configuration

The idea of a byte code cache (http://symfony.com/doc/current/book/performance.html) is to remove the need to constantly recompile the PHP source code for improved performance. There shouldn't be any downside but we experienced random problems with cache corruption (not funny when it happens in production).

Symfony2: traversal... what?! You need to implement it, while in Pylons/Pyramid comes for free. See http://docs.pylonsproject.org/projects/pyramid//en/latest/narr/muchadoabouttraversal.html

Add or customize views. Just one example: register or override the default view of a Document only if its parent is a Course if damn easy with Pylons/Pyramid (just register a new view with a discriminator, one line). With other frameworks you'll need to write a lot of not generic code. See http://docs.pylonsproject.org/projects/pyramid//en/latest/narr/viewconfig.html

Debugging is a pain with PHP. Debugging things with PHP and/or Symfony is a bit frustrating if you had previous experience with the Python debugger (pdb) or with the pyramid debug toolbar and its interactive through the web exception shell. If something goes wrong you can see what's the problem interactively or why a variable is "None" without leaving the browser. See:

• http://docs.pylonsproject.org/projects/pyramid-debugtoolbar/en/latest/index.html#the-toolbar

I didn't liked at all the security configuration of the Symfony2 framework with its "firewall" concept: the official doc itself admits it might be confusing and tough to set up just "because security is complex!" (see https://symfony.com/doc/current/book/security.html). Things can be complex but if your API lead people to be confused, probably you have a bad design problem.
Anyway it is more easy working with Pylons/Pyramid if you are going to write applications with complex security requirements (even workflow-based security thanks to the third party repoze.workflow). See http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/security.html

I hope you don't consider the above personal thoughts as not constructive rants: Symfony2 is a good and promising PHP framework and if you are interested in building web applications with PHP it is definitively a very good choice. I have been working with Symfony for 6 months and this is just my opinion compared to my previous programming experience with Zope, Plone, Grok, BFG, Pylons/Pyramid, Django and a bit of Nodejs/Express (KeystoneJS, SailsJS, etc).

So the previous Symfony2/PHP vs Python/Pyramid/Python comparison boils down to: if you are a curious web developer I think it really worth leaving your "comfort zone" and have a look at something of new. It doesn't matter if you are going to use Python (Django, Pyramid/Pylons, Flask, morepath, etc), Ruby, NodeJS or whatever else: you'll learn a lot of useful new things for sure and innovate your work!

Kotti CMS is not Plone

Kotti is very lightweight framework and its core is damn small because:

• a lot of things are demanded to the SQLAlchemy layer for storage, queries, indexing, etc.
• simpler stack compared to Plone/Zope/CMF/Archetypes/Dexterity/etc (just Kotti, Pylons/Pyramid, SQLAlchemy + a list of well known and widely adopted third party libraries. So if you spend time learning Pylons/Pyramid, you can easily switch to other frameworks). For example if you have to write event handlers with Kotti you'll notice that things are more simpler (no event handlers hell or having to manage duplicated events)
• tries to keeps things simple
• new programmers with no Pylons/Pyramid background are productive after a couple of days

and probably in next releases its core will shrink again.

It is very easy to start with: just a couple of days and you will be ok because it is very lightweight.

So programmers with no Pylons/Pyramid nor SQLAlchemy knowledge (or even not Python developers) will be productive very soon with Kotti. Obviously if you have a previous SQLAlchemy, Plone, Pylons/Pyramid or any other Python frameworks background it is better but it is not required at all.

I think Kotti is very good if you have to write a custom relational database based application with CMS-ish features without having to remove all the things provided by Plone that you don't need at all: with Kotti or Pylons/Pyramid you only pay for what you eat. But Kotti is not only good for CMS-ish applications: it a framework that does very well as a backend for heavy Javascript single page web applications.

Obviously Kotti is not Plone:

• community (the Plone's community is much more large)
• Plone has a lot of third party plugins
• no auto generated add/edit forms (not a problem because it is very simple to create or customize forms thanks to Colander/Deform. So no z3c.form there!)
• no portlets (topic covered in next blog post)
• no link actions (eg: portal_actions, topic covered in next blog post)
• missing permalinks and not breakable links
• no collections
• no PloneFormGen-like plugins (but we used a very good form builder solution available online as a service with integrated CRM)
• etc

but if you need a feature not provided by Kotti or other third party plugins don't worry because you will be productive very soon, even more if you adopt a frontend decoupled from the backend pattern.

Why frontend decoupled from the backend pattern

A couple of definitions about my frontend and backend concepts:

• frontend. It is our public website, where anonymous users browse the contents of the website. In our case it is built with PHP/Symfony2 but obviously you can also do all things with Python alone (pure Python is the best option)
• backend. It is our private content management area, built with Kotti CMS, Pylons/Pyramid and Python

This pattern is so flexible that let you implement the public area with a completely different technology. This way you can choose the best solution for the content management private area (for example: Python) and keep control of the frontend with your in-house developers (for example: PHP).
If you adopt a separation among frontend and backend your work will be even more agile:

• you can start with a "blank sheet" theme (topic covered in next articles)
• you don't waste time removing or hiding features or unneeded views, just implement what you need. Probably guys with experience with big fat frameworks understand what I mean.
• no CSS or Javascript conflicts with the backend. You won't be influenced at all how to integrate your usual frontend toolchain with a framework with its own tools or opinions because they are two completely different applications. So there is absolute frontend freedom, just choose the development stack you prefer
• you can develop new complex features like portlets (box shown in views) without having to touch at all the backend interface because the backend is just a (raw) backend area! They are just another type of content (not publishable on the frontend but rendered in views). So you can adopt workflows on portlets, you can copy/cut and paste them, use different views on portlets, assign custom views, create custom addability rules based on portlet types, etc. This way you can develop complex features like portlets in a fraction of time!
• since the frontend and the backend are served on the same domain, there is shared authentication so you can implement toolbar, live edit or view site as anonymous for editors
• less SEO headaches for non-publishable objects (portlets, tabs, collapsable sections, etc), you can store in the administration area objects that won't be published by the frontend at all. It is very handy implement these things like regular contents because you inherit workflows, same navigation/editing interface, etc. For example you can manage collapsable paragraphs as regular content type objects on the backend (yoursite.com/cms/document/paragraph1) but they will be rendered on the document view yoursite.com/document (if you try to access yoursite.com/document/paragraph1 from the frontend you'll get a 404 NotFound). Since they are not published on the public website you don't have to protect urls, no need to redirect to the main parent content, etc: all these things requires a lot of extra work if you want to take care of this kind of details
• if you use the same technology for the public website and the private area, you can reuse code already defined on the backend (for example you can reuse the breadcrumbs callable view with a completely template)

And more:

• faster. You can build the frontend with less security checks (just expose public data with less complexity due to assertion about local roles, groups, etc)
• more secure. You'll keep the administration area (backend) completely private and the frontend in our case built with PHP with a low privilege database user (readonly)
• you can rebuild the frontend without having to touch the administration area (backend)
• frontend easy replaceable
• different fulltext search policy for frontend vs frontend. On the backend there is the standard search results (you search for a text contained on a collapsable document and you'll land on that collapsable document on the backend, just what you expect from the editor perspective). If you search the same word on the public website you'll land on the parent document of the collapsable document (good for visitors) 

Photoes (credits)

A-team photoes from http://iqtell.com/2014/05/10-reasons-why-the-a-team-is-better-than-your-team/

All Kotti posts published by @davidemoro

• http://davidemoro.blogspot.it/search/label/kotti

End of part 1

In this article we have seen that:

• decoupled is cool!
• Python is cool!

Instead in next blog posts I'll talk about:

• how to build a pure Python Kotti based setup case study with a private content management area decoupled from the public website (with tips, links and screenshots, etc).

Update 20150718

Published part 2! See:

• frontend decoupled from backend. How we did it (part 2)