tag:blogger.com,1999:blog-48213361138465983542024-02-19T12:37:30.288+01:00Davide MoroPython, Javascript, AngularJS, Pyramid, Plone and... moreDavide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.comBlogger50125tag:blogger.com,1999:blog-4821336113846598354.post-13462199444963640482019-04-17T23:45:00.001+02:002019-04-18T08:37:53.827+02:00Testing metrics thoughts and examples: how to turn lights on and off through MQTT with pytest-playIn this article I'll share some personal thoughts about <b>test metrics</b> and talk about some technologies and tools playing around a real example: how to turn lights on and off through MQTT collecting test metrics.<br />
<br />
By the way the considerations contained in this article are valid for any system, technology, test strategy and test tools so you can easily integrate your existing automated tests with <i>statsd</i> with a couple of lines of code in any language.<br />
<br />
I will use the <a href="https://github.com/pytest-dev/pytest-play">pytest-play</a> tool in this example so that even non programmers should be able to play with automation collecting metrics because this tool is based on YAML (this way no classes, functions, threads, imports, no compilation, etc) and if Docker is already no installation is needed. You'll need only a bit of command line knowledge and traces of <a href="https://www.python.org/">Python</a> expressions like <i>variables["count"] > 0</i>.<br />
<br />
Anyway... yes, you can drive telematics/IoT devices with MQTT using <a href="https://github.com/pytest-dev/pytest-play">pytest-play</a> collecting and visualizing metrics thanks to:<br />
<ul>
<li><a href="https://github.com/statsd/statsd">statsd</a>, a "Daemon for easy but powerful stats aggregation"</li>
<li><a href="https://graphiteapp.org/">Graphite</a>, a <i>statsd</i> compatible "Make it easy to store and graph metrics" solution</li>
</ul>
or any other <i>statsd</i> capable monitoring engine.<br />
<br />
In our example we will see step by step how to:<br />
<ul>
<li>send a command to a device through MQTT (e.g., turn on a fridge light)</li>
<li>make assertions against the expected asynchronous response sent back by the device through MQTT (e.g., report light on/off status. In our case we expect a light on status)</li>
<li>collect key performance externally observable metrics on a JUnit compatible report file and optionally feed a <i>statsd</i> external metrics/monitoring engine (e.g, track how much time was needed for a round trip command/feedback on the MQTT broker)</li>
</ul>
using MQTT and <a href="https://github.com/pytest-dev/pytest-play">pytest-play</a>, using YAML files.<br />
<h3>
Why test metrics?</h3>
"Because we can" (cit. <i>Big Band Theory</i>, <i>Series 01 Episode 09 - The Cooper-Hofstadter Polarization</i>):<br />
<blockquote class="tr_bq">
<i><b>Sheldon:</b> Someone in Sezchuan province, China is using his computer to turn our lights on and off.<br /><b>Penny:</b> Huh, well that’s handy. Um, here's a question... why?!<br /><b>All together:</b> Because we can!</i></blockquote>
If the "<i>Because we can</i>" answer doesn't convince your boss, there are several advantages that let you react proactively before something of not expected happens. And to be proactive you need knowledge of you system under test thanks to measurable metrics that let you:<br />
<ul>
<li>know how your system behaves (and confirm where bottlenecks are located)</li>
<ul>
<li>in standard conditions</li>
<li>under load</li>
<li>under stress</li>
<li>long running</li>
<li>peak response</li>
<li>with a big fat databases</li>
<li>simulating a small percentage of bad requests</li>
<li>or any other sensible scenario that needs to be covered</li>
</ul>
<li>know under which conditions your users will perceive</li>
<ul>
<li>no performance deterioration</li>
<li>a performance deterioration</li>
<li>a critical performance deterioration</li>
<li>system stuck</li>
</ul>
<li>understand how much time is available before first/critical/blocking performance deterioration will met considering users/application growth trends</li>
</ul>
so that you can be <b>proactive</b> and:<br />
<ul>
<li>keep your stakeholders informed very valuable information</li>
<li>improve your system performance before something of bad will happen</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGLC5ON6ll3-r8wJLp8chguHEHxUCInXE41zrTBlxxeZLPRk9UCbcYivZkLFssjqW5VVCRXRT5m5Mtkx96KTPgwAtswCtWeL3SynJadfvKJeo2nMjZ3VIqSeWVpXcpeT34MlgYCv6N/s1600/release.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="308" data-original-width="586" height="336" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGLC5ON6ll3-r8wJLp8chguHEHxUCInXE41zrTBlxxeZLPRk9UCbcYivZkLFssjqW5VVCRXRT5m5Mtkx96KTPgwAtswCtWeL3SynJadfvKJeo2nMjZ3VIqSeWVpXcpeT34MlgYCv6N/s640/release.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Ouch! The effects of a bad release in action</td></tr>
</tbody></table>
In addition you can:</div>
<ul>
<li>anticipate test automation failures due to timeouts, maybe you already experienced a test always passing and one day it will start sporadically to exceed your maximum timeout</li>
<li>choose more carefully timeouts if there are no specific requirements</li>
<li>avoid false alarms like a generic "<i>today the system seems slower</i>". If there is a confirmed problem you might say instead: "<i>compared to previous measurements, the system response is 0.7 s slower today. Systematically.</i>"</li>
<li>find corner cases. You might notice that the average response time is always pretty the same or slightly higher because there is a particular scenario that systematically produces a hard to discover response time peak compared to similar requests and that might create some integration problems if other components are not robust</li>
<li>avoid retesting response times with previous versions comparing to the actual built, because everything has been already tracked</li>
</ul>
What should you measure? Everything of valuable for you:.<br />
<ul>
<li>API response times</li>
<li>time needed for an asynchronous observable effect will happen</li>
<li>metrics from a user/business perspective (e.g., it is more important for users API response times, browser first paint or how when she/he can start using a web page?)</li>
<li>metadata (browser, versions, etc). Metadata formats non compatible with <i>statsd</i> might be tracked on custom JUnit XML reports</li>
<li>pass/skip/error/etc rates</li>
<li>deploys</li>
<li>etc</li>
</ul>
<h2>
<span style="font-weight: normal;">Some information about statsd/Graphite and MQTT</span></h2>
<h3>
statsd/Graphite</h3>
<h3>
<span style="font-size: small; font-weight: 400;">Very very interesting readings about <i>statsd</i> and the </span><span style="font-size: small;">measure everything approach</span><span style="font-size: small; font-weight: 400;">:</span></h3>
<div>
<br />
<ul>
<li><a href="https://codeascraft.com/2011/02/15/measure-anything-measure-everything/">https://codeascraft.com/2011/02/15/measure-anything-measure-everything/</a></li>
<li><a href="https://codeascraft.com/2010/12/08/track-every-release/">https://codeascraft.com/2010/12/08/track-every-release/</a></li>
</ul>
</div>
<div>
If you are not familiar with statsd and Graphite you can install it (<i>root</i>/<i>root</i> by default):</div>
<h3>
<blockquote class="tr_bq" style="font-size: medium; font-weight: 400;">
<i>docker run -d\<br /> --name graphite\<br /> --restart=always\<br /> -p 80:80\<br /> -p 2003-2004:2003-2004\<br /> -p 2023-2024:2023-2024\<br /> -p 8125:8125/udp\<br /> -p 8126:8126\<br /> graphiteapp/graphite-statsd</i></blockquote>
</h3>
and play with it sending fake metrics using <i>nc</i>:<br />
<blockquote class="tr_bq">
<i>echo -n "my.metric:320|ms" | nc -u -w0 127.0.0.1 8125</i></blockquote>
you'll find a new metric aggregations available:<br />
<blockquote class="tr_bq">
<i>stats.timers.$KEY.mean<br />stats.timers.$KEY.mean_$PCT<br />stats.timers.$KEY.upper_$PCT<br />stats.timers.$KEY.sum_$PCT<br />...</i></blockquote>
where:<br />
<br />
<ul>
<li><i>$KEY</i> is <i>my.metric</i> in this example (so metric keys are hierarchical for a better organization!)</li>
<li><i>$PCT</i> is the percentile (e.g., <i>stats.timers.my.metric.upper_90</i>)</li>
</ul>
<div>
More info, options, configurations and metric types here:</div>
<div>
<ul>
<li><a href="https://github.com/statsd/statsd">https://github.com/statsd/statsd</a></li>
<li><a href="https://graphiteapp.org/">https://graphiteapp.org</a></li>
</ul>
</div>
<br />
<h3>
What is MQTT?</h3>
From <a href="http://mqtt.org/">http://mqtt.org/</a>:<br />
<blockquote>
<i>MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol.<br />It was designed as an extremely lightweight publish/subscribe messaging transport.<br />It is useful for connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium.</i></blockquote>
MQTT is the standard de facto for smarthome/IoT/telematics/embedded devices communications, even on low performance embedded devices, and it<br />
is available on many cloud infrastructures.<br />
<br />
Every actor can publish a message for a certain topic and every actor can subscribe to a set of topics, so you get a message for every message of interest.<br />
<br />
Topics are hierarchical so that you can subscribe to a very specific or wide range of topics coming from devices or sensors (e.g., <i>/house1/room1/temp</i>, <i>/house1/room1/humidity</i> or all messages related to <i>/house1/room1/</i> etc).<br />
<br />
For example in a telematics application every device will listen to any command or configuration sent by a server component through a MQTT broker (e.g., <i>project1/DEVICE_SN/cmd</i>);<br />
server will be notified for any device response or communication subscribing to a particular topic (e.g., <i>project1/DEVICE_SN/data</i>).<br />
So:<br />
<ul>
<li>you send commands to a particular device publishing messages on <i>foo/bar/DEVICE_SN/cmd</i> </li>
<li>you expect responses subscribing to <i>foo/bar/DEVICE_SN/data.</i></li>
</ul>
If you are not confident with MQTT you can install the mosquitto utility and play with the <i>mosquitto_sub</i> and <i>mosquitto_pub</i> commands using with the public broker <a href="http://iot.eclipse.org/">iot.eclipse.org</a>. For example you can publish a message for a given topic:<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGQbCTPCf8H-lz5E2OQYSXa6OBMkSfXZjfO0oIHO075S2FuxlrcseN41ypdSQwJS7Js_G7u5IyUCPpmD8b7OZXk5yVljmWa8QgwU3pPw_EOKTIrC7_b0_61nR0msjwgvD12Cdksjru/s1600/mosquitto_pub.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="125" data-original-width="1343" height="57" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGQbCTPCf8H-lz5E2OQYSXa6OBMkSfXZjfO0oIHO075S2FuxlrcseN41ypdSQwJS7Js_G7u5IyUCPpmD8b7OZXk5yVljmWa8QgwU3pPw_EOKTIrC7_b0_61nR0msjwgvD12Cdksjru/s640/mosquitto_pub.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">$ mosquitto_pub -t foo/bar -h iot.eclipse.org -m "hello pytest-play!"</td></tr>
</tbody></table>
and see the response assuming that you previously subscribed to <i>foo/bar</i> (we see all messages sent with <i>mosquitto_pub</i> of our topics of interest here):<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHBAEHtQo1EJD3LzZ7W-k4jK5RuNJlb4fFgrWk-Cpq1toXBq-08y1Cm2s3buAMkWFuSVL2I76lKFhpXFq5rkv9aOUZNBO_qWTvEeE6OsARV7wRpvy8AHZX6vH5fIT1X5oX8yHxRMAo/s1600/mosquitto_sub.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="126" data-original-width="1080" height="74" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHBAEHtQo1EJD3LzZ7W-k4jK5RuNJlb4fFgrWk-Cpq1toXBq-08y1Cm2s3buAMkWFuSVL2I76lKFhpXFq5rkv9aOUZNBO_qWTvEeE6OsARV7wRpvy8AHZX6vH5fIT1X5oX8yHxRMAo/s640/mosquitto_sub.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">$ mosquitto_sub -t foo/bar/# -h iot.eclipse.org -v</td></tr>
</tbody></table>
<h2>
Prerequisites</h2>
<i>pytest-play</i> is multi platform because it is based on Python (installation might be different for different operative system).<br />
Using <i>Docker</i> instead no installation is required, you need to install <i>Docker</i> and you are ready to start playing with <i>pytest-play</i> without any installation:<br />
<ul>
<li>Windows, see <a href="https://docs.docker.com/windows/started">https://docs.docker.com/windows/started</a></li>
<li>OS X, see <a href="https://docs.docker.com/mac/started/">https://docs.docker.com/mac/started/</a></li>
<li>Linux, see <a href="https://docs.docker.com/linux/started/">https://docs.docker.com/linux/started/</a></li>
</ul>
<div>
As a user you should be confident with a shell and command line options.</div>
<h2>
Steps</h2>
And now let's start with our example.<br />
<h3>
Create a new folder project</h3>
Create a new folder (e.g., <i>fridge</i>) and enter inside.<br />
<h3>
Create a variables file</h3>
Create a <i>env.yml</i> file with the following contents:<br />
<blockquote class="tr_bq">
<i>pytest-play:<br /> mqtt_host: YOUR_MQTT_HOST<br /> mqtt_port: 20602<br /> mqtt_endpoint: foo/bar</i></blockquote>
You can have one or more configuration files defining variables for your convenience. Typically you have one configuration file or each target environment (e.g., <i>dev.yml</i>, <i>alpha.yml</i>, etc).<br />
<br />
We will use later this file for passing variables thanks to the <i>--variables env.yml</i> command line option, so you can switch environment passing different files.<br />
<h3>
Create the YML script file</h3>
Create a a <i>YAML</i> file called <i><b>test_</b>light_on<b>.yml</b></i> inside the <i>fridge</i> folder or any other subfolder if any. Note well: the <i>*.yml</i> extension and <i>test_</i> prefix matter otherwise the file won't be considered as executable at this time of writing.<br />
<br />
If you need to simulate a command or simulate a device activity you need just one command inside your YAML file:<br />
<blockquote class="tr_bq">
<i>- comment: send light turn ON command<br /> provider: mqtt<br /> type: publish<br /> host: "$mqtt_host"<br /> port: "$mqtt_port"<br /> endpoint: "$mqtt_endpoint/$device_serial_number/cmd"<br /> payload: '{"Payload":"244,1"}'</i></blockquote>
where <i>244</i> stands for the internal <i>ModBUS</i> registry reference for the fridge light and <i>1</i> stands for ON (and <i>0</i> for OFF).<br />
<br />
But... wait a moment. Until now we are only sending a payload to a MQTT broker resolving the <i>mqtt_host</i> variable for a given endpoint and nothing more... pretty the same business you can do with <i>mosquitto_pub</i>, right? You are right! That's why we are about to implement something of more:<br />
<ul>
<li>subscribe to our target topic where the expected response will come and store every single received message to a <i>messages</i> variable (it will contain an array of response payload strings)</li>
<li>add an asynchronous waiter waiting for the expected device response</li>
<li>once detected the expected response arrived, make some assertions</li>
<li>track testing metrics</li>
<li>enable support for parametrized scenarios with decoupled test data</li>
<li>Jenkins/CI capabilities (not covered in this article, see <a href="http://davidemoro.blogspot.com/2018/03/test-automation-python-pytest-jenkins.html">http://davidemoro.blogspot.com/2018/03/test-automation-python-pytest-jenkins.html</a>)</li>
</ul>
Put inside our file the following contents inside the <i>test_light_on.yml</i> file and save:<br />
<blockquote class="tr_bq">
<i>markers:<br /> - light_on<br />test_data:<br /> - device_serial_number: 8931087315095410996<br /> - device_serial_number: 8931087315095410997<br />---<br />- comment: subscribe to device data and store messages to messages variable once received (non blocking subscribe)<br /> provider: mqtt<br /> type: subscribe<br /> host: "$mqtt_host"<br /> port: "$mqtt_port"<br /> topic: "$mqtt_endpoint/$device_serial_number"<br /> name: "messages"<br />- comment: send light turn ON command<br /> provider: mqtt<br /> type: publish<br /> host: "$mqtt_host"<br /> port: "$mqtt_port"<br /> endpoint: "$mqtt_endpoint/$device_serial_number/cmd"<br /> payload: '{"Payload":"244,1"}'<br />- comment: start tracking response time (stored in response_time variable)<br /> provider: metrics<br /> type: record_elapsed_start<br /> name: response_time<br />- comment: wait for a device response<br /> provider: python<br /> type: while<br /> timeout: 12<br /> expression: 'len(variables["messages"]) == 0'<br /> poll: 0.1<br /> sub_commands: []<br />- command: store elapsed response time in response_time variable<br /> provider: metrics<br /> type: record_elapsed_stop<br /> name: response_time<br />- comment: assert that status light response was sent by the device<br /> provider: python<br /> type: assert<br /> expression: 'loads(variables["messages"][0])["measure_id"] == [488]'<br />- comment: assert that status light response was sent by the device with status ON<br /> provider: python<br /> type: assert<br /> expression: 'loads(variables["messages"][0])["bin_value"] == [1]'</i></blockquote>
Let's comment command by command and section by section the above YAML configuration.<br />
<h4>
Metadata, markers and decoupled test data</h4>
First of all the <i>---</i> delimiter splits an optional metadata document from the scenario itself. The metadata section in our example contains:<br />
<blockquote class="tr_bq">
<i>markers:<br /> - light_on</i></blockquote>
You can mark your scripts with one or more markers so that you can select which scenario will run from the command line using <b>marker expressions</b> like <i>-m light_off</i> or something like <i>-m "light_off and not slow"</i> assuming that you have some script marked with the pretend slow marker.<br />
<h4>
Decoupled test data and parametrization</h4>
Assume that you have 2 or more real devices providing different firmware versions always ready to be tested.<br />
<br />
In such case we want define our scenario once and it will be executed more thanks to <b>parametrization</b>. Our scenario will be executed for each any item defined in the <i>test_data</i> array in the metadata section. In our example it will be executed twice:<br />
<blockquote class="tr_bq">
<i>test_data:<br /> - device_serial_number: 8931087315095410996<br /> - device_serial_number: 8931087315095410997</i></blockquote>
If you want you can track different metrics for different serial numbers so that you are able to compare different firmware versions.<br />
<h4>
Subscribe to topics where we expect a device response</h4>
As stated in the official <i>play_mqtt</i> documentation <a href="https://github.com/davidemoro/play_mqtt">https://github.com/davidemoro/play_mqtt</a><br />
you can subscribe to one or more topics using the mqtt provider and type: subscribe. You have to provide the where the MQTT broker host lives (e.g., <i>iot.eclipse.org</i>), the port, obviously the topic you want to subscribe (e.g., <i>foo/bar/$device_serial_number/data/light</i> where <i>$device_serial_number</i> will be replaced with what you define in environment configuration files or for each <i>test_data</i> section.<br />
<blockquote class="tr_bq">
<i>- comment: subscribe to device data and store messages to messages variable once received (non blocking subscribe)<br /> provider: mqtt<br /> type: subscribe<br /> host: "$mqtt_host"<br /> port: "$mqtt_port"<br /> topic: "$mqtt_endpoint/$device_serial_number"<br /> name: "messages"</i></blockquote>
This is a non blocking call so that while the flow continues, it will collect underground every message published on the topics of our interest storing them to a <i>messages</i> variable.<br />
<br />
<i>messages</i> is an array containing all matching messaging coming from MQTT and you can access to the messages value in expressions with <i>variables["messages"]</i>.<br />
<h4>
Publish a command</h4>
<div>
This is self explaining (you can send any payload, even dynamic/parametrized payloads):<br />
<blockquote class="tr_bq">
<i>- comment: send light turn ON command<br /> provider: mqtt<br /> type: publish<br /> host: "$mqtt_host"<br /> port: "$mqtt_port"<br /> endpoint: "$mqtt_endpoint/$device_serial_number/cmd"<br /> payload: '{"Payload":"244,1"}'</i></blockquote>
</div>
<div>
where <i>244</i> is the internal reference and <i>1</i> stands for ON.</div>
<h4>
Track time metrics</h4>
<div>
This command let you start tracking time from now until a <i>record_elapsed_stop</i> will be executed:<br />
<blockquote class="tr_bq">
<i>- comment: start tracking response time (stored in response_time variable)<br /> provider: metrics<br /> type: record_elapsed_start<br /> name: response_time<br />... <one or more commands or asynchronous waiters here><br />- command: store elapsed response time in response_time variable<br /> provider: metrics<br /> type: record_elapsed_stop<br /> name: response_time</i></blockquote>
</div>
<div>
The time metric will be available under a variable name called in our example <i>response_time</i> (from <i>name: response_time</i>). For a full set of metrics related commands and options see <a href="https://github.com/pytest-dev/pytest-play">https://github.com/pytest-dev/pytest-play</a>.</div>
<div>
<br /></div>
<div>
You can record key metrics of any type for several reasons:</div>
<div>
<ul>
<li>make assertions about some expected timings</li>
<li>report key performance metrics or properties in custom <i>JUnit XML</i> reports (in conjunction with the command line option <i>--junit-xml results.xml</i> for example so that you have an historical trend of metrics for each past or present test execution)</li>
<li>report key performance metrics on <i>statsd</i> capable third party systems (in conjunction with the command line option <i>--stats-d [--stats-prefix play --stats-host http://myserver.com --stats-port 3000]</i>)</li>
</ul>
</div>
<h4>
While</h4>
Here we are waiting for a message response was collected and stored to the <i>messages</i> variable (do you remember the already discussed MQTT subscribe command in charge of collecting/storing messages of interest?):<br />
<blockquote class="tr_bq">
<i>- comment: wait for a device response<br /> provider: python<br /> type: while<br /> timeout: 12<br /> expression: 'len(variables["messages"]) == 0'<br /> poll: 0.1<br /> sub_commands: []</i></blockquote>
<div>
You can specify a timeout (e.g., <i>timeout: 12</i>), a poll time (how many wait seconds between a while iteration, in such case <i>poll: 0.1</i>) and an optional list of while's sub commands (not needed for this example).</div>
<div>
<br /></div>
<div>
When the expression returns a true-ish value, the while command exits.</div>
<br />
Does your device publish different kind of data on the same topic? Modify the while expression restricting to the messages of your interest, for example:<br />
<blockquote class="tr_bq">
<i>- comment: [4] wait for the expected device response<br /> provider: python<br /> type: while<br /> timeout: 12<br /> expression: 'len([item for item in variables["messages"] if loads(item)["measure_id"] == [124]]) == 0'<br /> poll: 0.1<br /> sub_commands: []</i></blockquote>
In the above example we are iterating over our array obtaining only the entries with a given <i>measure_id</i> where the loads is a builtin JSON parse (python's <i>json.loads</i>).<br />
<blockquote class="tr_bq">
<i><?xml version="1.0" encoding="utf-8"?><testsuite errors="0" failures="0" name="pytest" skipped="0" tests="1" time="10.664"><testcase classname="test_on.yml" file="test_on.yml" name="test_on.yml[test_data0]" time="10.477"><properties><property name="response_time" value="7.850502967834473"/></properties><system-out>...</i></blockquote>
<h4>
Assertions</h4>
<div>
And now it's assertions time:</div>
<div>
<blockquote class="tr_bq">
<i>- comment: assert that status light response was sent by the device<br /> provider: python<br /> type: assert<br /> expression: 'loads(variables["messages"][0])["measure_id"] == [488]'<br />- comment: assert that status light response was sent by the device with status ON<br /> provider: python<br /> type: assert<br /> expression: 'loads(variables["messages"][0])["bin_value"] == [1]'</i></blockquote>
Remember that the messages variables is an array of string messages? We are taking the first message (with <i>variables["messages"][0]</i> you get the first raw payload), parse the JSON payload so that assertions will be simpler (in our case <i>loads(variables["messages"][0])</i> for sake of completeness) obtaining a dictionary and then assert that we have the expected values under certain dictionary keys.<br />
<br />
As you can see pytest-play is not 100% codeless by design because it requires a very basic Python expressions knowledge, for example:<br />
<ul>
<li><i>variables["something"] == 0</i></li>
<li><i>variables["something"] != 5</i></li>
<li><i>not variables["something"]</i></li>
<li><i>variables["a_boolean"] is True</i></li>
<li><i>variables["a_boolean"] is False</i></li>
<li><i>variables["something"] == "yet another value"</i></li>
<li><i>variables["response"]["status"] == "OK" and not variables["response"]["error_message"]</i></li>
<li><i>"VALUE" in variables["another_value"]</i></li>
<li><i>len([item for item in variables["mylist"] if item > 0) == 0</i></li>
<li><i>variables["a_string"].startswith("foo")</i></li>
</ul>
One line protected Python-based expressions let you express any kind of waiters/assertions without having the extend the framework's commands syntax introducing an exotic YAML-based meta language that will never be able to express all the possible use cases. The basic idea behind Python expressions is that even for non programmers it is easier to learn the basics of Python assertions instead of trying to figure out how to express assertions in an obscure meta language.<br />
<br />
<i>pytest-play</i> is not related to MQTT only, it let you write actions and assertions against a real browser with Selenium, API/REST, <i>websockets</i> and more.<br />
<br />
So if you have to automate a task for a device simulator, a device driver, some simple API calls with assertions, asynchronous wait for a condition is met with timeouts or interact with browsers, cross technology actions (e.g., publish a MQTT message and poll a HTTP response until something happens) and decoupled test data parametrization... even if you are not a programmer because you don't have to deal with imports, function or class definitions and it is always available if you have Docker installed.<br />
<br />
<br />
<div>
And now you can show off with shining metrics!</div>
</div>
<h3>
Run your scenario</h3>
And finally, assuming that you are already inside your project folder, let's run our scenario using Docker (remember <i>--network="host"</i> if you want to send metrics to a server listening on <i>localhost</i>):<br />
<div>
<blockquote class="tr_bq">
<i>docker run --rm -it -v $(pwd):/src --network="host" davidemoro/pytest-play --variables env.yml --junit-xml results.xml --stats-d --stats-prefix play test_light_on.yml</i></blockquote>
The previous command will run our scenario printing the results and if there is a stats server listening on localhost metrics will be collected and you will be able to create live dashboards like the following one:</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSFIGNilWBOMEDjA1bsnKekt0jUL5jBsmpNVSynfVdukS6jEaiv3BQAFnz7FJX0dJci5mwYB0yRRIVs_k5mQQicstItUd5h-QBM5v7y48QgnoeCLZ_JmtVUMpkQAOr-XJIgGe1vrAN/s1600/output_5Ogb0S.gif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="308" data-original-width="586" height="210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSFIGNilWBOMEDjA1bsnKekt0jUL5jBsmpNVSynfVdukS6jEaiv3BQAFnz7FJX0dJci5mwYB0yRRIVs_k5mQQicstItUd5h-QBM5v7y48QgnoeCLZ_JmtVUMpkQAOr-XJIgGe1vrAN/s400/output_5Ogb0S.gif" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">statsd/Graphene response time dashboard</td></tr>
</tbody></table>
and metrics are stored in the results.xml file too:<br />
<blockquote class="tr_bq">
<i><?xml version="1.0" encoding="utf-8"?><testsuite errors="0" failures="0" name="pytest" skipped="0" tests="1" time="10.664"><testcase classname="test_on.yml" file="test_on.yml" name="test_on.yml[test_data0]" time="10.477"><properties><property name="response_time" value="7.850502967834473"/></properties><system-out>...</i></blockquote>
<h3>
Sum up</h3>
This was a very long article and we talked about a lot of technologies and tools. So if you are not yet familiar with some tools or technologies it's time to read some documentation and play with some hello world examples:<br />
<br />
<ul>
<li>MQTT stuff, you can play with <i>moquitto_sub</i>, <i>mosquitto_pub</i> and the free MQTT broker online <a href="http://iot.eclipse.org/">iot.eclipse.org</a> or with your things if you have a smart home </li>
<li><i>statsd</i>/<i>graphite</i>, you can install the server with Docker, build your dashboard on <a href="http://localhost/">http://localhost</a> and send fake metrics with the <i>echo</i> and <i>nc</i> command line</li>
<li>talking about <i>pytest-play</i>, pull the docker container and run some examples <a href="https://github.com/pytest-dev/pytest-play/tree/master/examples">https://github.com/pytest-dev/pytest-play/tree/master/examples</a></li>
<li>a <i>pytest</i> knowledge is useful too for the most useful command line options, see <a href="https://docs.pytest.org/en/latest">https://docs.pytest.org/en/latest</a></li>
</ul>
<div>
Any feedback is welcome!</div>
<h3>
Do you like pytest-play?</h3>
<div>
Let's get in touch for any suggestion, contribution or comments. Contributions will be very appreciated too!</div>
<div>
<!-- Place this tag where you want the button to render. -->
<a aria-label="Star pytest-dev/pytest-play on GitHub" class="github-button" data-icon="octicon-star" data-show-count="true" data-size="large" href="https://github.com/pytest-dev/pytest-play">Star</a>
<!-- Place this tag in your head or just before your close body tag. -->
<script async="" defer="" src="https://buttons.github.io/buttons.js"></script>
</div>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-41893645777520791732019-03-20T18:30:00.000+01:002019-03-20T21:07:32.134+01:00CSS selectors guidelines for smooth test automation with SeleniumMind exploding trying to locate a DOM element and after 20 minutes you get a selector like that?<br />
<blockquote class="tr_bq">
<i>.elem__container_search div:nth-child(1) .row1 tr:nth-child(5) input:nth-child(2)</i></blockquote>
or even worse you have to switch to <i>xpath</i>? And after a while you realize it is not correct spending more time? Or someone changed input elements order and your tests are broken?<br />
<br />
You should be able to address any important application element in a page with no pain in seconds with a CSS selector: <i>.last-update</i> is better than <i>.row1 tr:nth-child(5) input:nth-child(2)</i>, do you agree?<br />
<br />
If your answer is yes, this article if for you!<br />
<h3>
Do not blame developers, provide guidelines and examples</h3>
First of all do not blame developers for poor CSS selectors, provide guidelines and examples instead! Nowadays is even more important providing guidelines because with some Javascript frameworks you might see cryptic, sometimes random (!!!), codes as classes and ids not usable for our purposes.<br />
<h3>
Test robustness, test automation development and maintenance time are affected by poor selectors</h3>
In other terms a bad page design affects <b>productivity</b>:<br />
<ul>
<li>you should locate an element in seconds instead of minutes (<i>.last-update</i> vs <i>.row1 tr:nth-child(5) input:nth-child(2)</i>)</li>
<li>the <i>.row1 tr:nth-child(5) input:nth-child(2)</i> selector might not be valid for users with different roles while <i>.last-update</i> will be always valid </li>
<li>if the elements order change, no broken tests and <i>.last-update</i> will remain still valid (with <i>xpath</i> or selectors like <i>.row1 tr:nth-child(5) input:nth-child(2)</i> you will get broken tests)</li>
</ul>
... so less time wasted and more money with good conventions.<br />
<br />
<h3>
Define frontend development conventions together and follow them</h3>
<br />
If you are using a framework or a CMS probably there will be already a good selectors guideline so you can make it yours if it fits your needs, extend it if needed and contribute back with improvements.<br />
<br />
You can write examples for common use case so that everyone speaks the same language. E.g., define a naming convention for elements with status <i>class="wifi wifi--state-on"</i> / <i>class="wifi wifi--state-off"</i>.<br />
<br />
Alternatively you can adopt an existing and possibly widely used existing convention like BEM (Block Element Modifier - a methodology that provides a naming convention for frontend developers):<br />
<ul>
<li><a href="http://getbem.com/">http://getbem.com</a></li>
</ul>
or any other similar initiative.<br />
<h3>
Final thoughts</h3>
Sometimes you don't control the web application you are
testing so you need to explain what you are expecting and why it is
important communicating the improvements in terms of better test
automation robustness and development time.<br />
<br />
<br />
<br />
It is also useful measuring the time wasted due to bad practice so that
you might support better your proposal reporting to the management real
numbers in terms of saved/lost money. <br />
<br />
So my final suggestion is to find a solution together so that every future page will follow agreed guidelines according to existing best practice if possible.<br />
<br />
And as a developer always keep in mind the following questions as double check:<br />
<ul>
<li>am I able to locate this element with no pain?</li>
<li>am I able to understand the status of this element? </li>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-50286498218417626502019-02-13T00:10:00.000+01:002019-02-13T00:19:02.525+01:00Turn any program that uses STDIN/STDOUT into a WebSocket server using websocketdYesterday I tried for the first time <a href="http://websocketd.com/">websocketd</a> and it is amazing!<br />
This is a little gem created by <a href="https://twitter.com/joewalnes">@joewalnes</a> that let you implement language agnostic <i>WebSocket</i> applications based on any command line program reading text from <i>stdin</i> and writing a message output to <i>stdout</i>. A simple (and genial) approach.<br />
<br />
If think that it is perfect for testing too, so I updated an existing integration test for a WebSocket client of mine using a local <i>websocketd</i> server on TravisCI instead of relying on external public services requiring internet as done before (<i><a href="wss://echo.websocket.org/"><span class="blob-code-inner"><span class="pl-s"><span class="x x-first x-last">wss</span>://<span class="x x-first x-last">echo.websocket.org</span></span></span></a></i>). This activity was already on my roadmap so why not try out websocketd?!<br />
<br />
As you can see here (<a href="https://github.com/davidemoro/pytest-play-docker/pull/42/files">https://github.com/davidemoro/pytest-play-docker/pull/42/files</a>): in <i>.travis.yml</i> I added a <i>before_script</i> calling a <i>travis/setup_websocket.sh</i> script that installs and runs <i>websocketd</i> in background on port 8081 based on a simple <i>travis/echo_ws.sh</i> that reads a line from stdin echoing to stdout.<br />
<br />
The websocketd syntax is the following:<br />
<br />
<blockquote class="tr_bq">
<i>./websocketd --port=8081 ./echo_ws.sh</i></blockquote>
<br />
were <i>echo_ws.sh</i> can be <i>any executable</i> stdin/stdout based. More details in next section.<br />
<h2>
Wanna tryout websocketd? </h2>
Download a <i>websocketd</i> version compatible with your OS/architecture and unzip the folder from <a href="http://websocketd.com/#download">http://websocketd.com/#download</a>: it contains a <i>websocketd</i> executable ready to be used so no installation needed and follow the tutorials described in <a href="https://github.com/joewalnes/websocketd/wiki">https://github.com/joewalnes/websocketd/wiki</a>.<br />
<br />
Alternatively you can test <i>websocketd</i> using <a href="https://github.com/pytest-dev/pytest-play">pytest-play</a>'s <a href="https://github.com/davidemoro/play_websocket">play_websocket</a> plugin that is ready to be used assuming that you have Docker installed.<br />
<blockquote class="tr_bq">
<i>docker run --rm -it -v $(pwd):/src --network host davidemoro/pytest-play --variables variables.yml</i></blockquote>
Additional links you might find useful for this example:<br />
<ul>
<li><a href="https://github.com/joewalnes/websocketd/wiki">websocketd documentation</a></li>
<li><a href="https://github.com/pytest-dev/pytest-play">pytest-play</a> (almost codeless automation running YAML files)</li>
<li><a href="https://github.com/davidemoro/play_websocket">play_websocket</a> (pytest play plugin for websockets)</li>
</ul>
<script src="https://gist.github.com/davidemoro/2e376aac92901589ead815d46add7c10.js"></script>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-65822446025197398042019-02-12T08:44:00.000+01:002019-02-12T08:44:27.021+01:00Setting up Cassandra database service on TravisCI<br />
At this time of writing <a href="https://travis-ci.org/">TravisCI</a> says that if you want to run a <a href="http://cassandra.apache.org/">Cassandra</a> service you have to add a cassandra service according to <a href="https://docs.travis-ci.com/user/database-setup/#cassandra">https://docs.travis-ci.com/user/database-setup/#cassandra</a>:<br />
<blockquote class="tr_bq">
<i>services:<br /> - cassandra</i></blockquote>
but if you try to initialize cassandra you might find out that cassandra is not yet ready or running depending on timing.<br />
<br />
The solution is:<br />
<ul>
<li>a <i>.travis.yml</i> file (see <a href="https://github.com/davidemoro/pytest-play-docker/blob/master/.travis.yml">https://github.com/davidemoro/pytest-play-docker/blob/master/.travis.yml</a>) with the cassandra service and a "<span class="pl-s">bash travis/setup_cassandra.sh</span>" command in before_script section </li>
<li>a <i>travis/setup_cassandra.sh</i> (see <a href="https://github.com/davidemoro/pytest-play-docker/blob/master/travis/setup_cassandra.sh">https://github.com/davidemoro/pytest-play-docker/blob/master/travis/setup_cassandra.sh</a>) that waits for cassandra up and running (e.g., a <i>sleep 30</i>, a polling loop with max retries implemented in my <i>travis/setup_cassandra.sh</i> script or using <i>docker</i> service) before <i>cqlsh</i> initialization commands</li>
</ul>
And now cassandra is ready to be used in your tests (for example <a href="https://github.com/davidemoro/pytest-play-docker/blob/master/tests/test_cassandra.yml">https://github.com/davidemoro/pytest-play-docker/blob/master/tests/test_cassandra.yml</a> . In this case I'm using using plain yml files thanks to <a href="https://github.com/pytest-dev/pytest-play">pytest-play</a>)<br />
<script src="https://gist.github.com/davidemoro/86fe29a67963751cbec95ed7fc30c561.js"></script>Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-33630506843829370182019-02-09T23:59:00.001+01:002019-06-11T22:53:58.450+02:00High quality automated docker hub push using Github, TravisCI and pyup for Python tool distributions<b>UPDATE 20190611:</b> the article contents are still valid but I definitively switched from pyup to the <a href="http://requires.io/">requires.io</a> service. Why? There were an unexpected error on my pyup project and the support never sent any kind of feedback for weeks (is the project still alive?!). Anyway I happily switched to <i>requires.io</i>: I like the way they manage cumulative pull requests with errored packages and the support is responsive.<br />
<br />
Let's say you want to distribute a Python tool with docker using known good dependency versions ready to be used by end users... In this article you will see how to continuously keeping up to date a Docker Hub container with minimal managing effort (because I'm a lazy guy) using <a href="https://github.com/">github</a>, <a href="https://travis-ci.org/">TravisCI</a> and <a href="https://pyup.io/">pyup</a>.<br />
<br />
The goal was to reduce as much as possible any manual activity for
updates, check all works fine before pushing, minimize build times and
keep docker container always secure and updated with a final high quality confidence.<br />
<br />
As an example let's see what happens under the hood behind every <i>pytest-play</i> Docker Hub update on the official container <a href="https://cloud.docker.com/u/davidemoro/repository/docker/davidemoro/pytest-play">https://cloud.docker.com/u/davidemoro/repository/docker/davidemoro/pytest-play</a> (by the way if you are a <i>pytest-play</i> user: did you know that you can use Docker for running pytest-play and that
there is a docker container ready to be used on Docker Hub? See a complete and working example here <a href="https://davidemoro.blogspot.com/2019/02/api-rest-testing-pytest-play-yaml-chuck-norris.html">https://davidemoro.blogspot.com/2019/02/api-rest-testing-pytest-play-yaml-chuck-norris.html</a>)<br />
<h2>
Repositories</h2>
The docker build/publish stuff lives on another repository, so
<a href="https://github.com/davidemoro/pytest-play-docker">https://github.com/davidemoro/pytest-play-docker</a> is the repository that
implements the Docker releasing workflow for
<a href="https://github.com/pytest-dev/pytest-play">https://github.com/pytest-dev/pytest-play</a> on Docker Hub (<a href="https://hub.docker.com/r/davidemoro/pytest-play">https://hub.docker.com/r/davidemoro/pytest-play</a>).<br />
<h2>
Workflow</h2>
This is the highly automated workflow at this time of writing for the <i>pytest-play</i> publishing on Docker Hub:<br />
<ul>
<li>a new <i>pytest-play</i> version or <i>any other</i> pinned sub-dependency is published on <a href="https://pypi.org/">PyPI</a>.<br /><br />By the way <i>pytest-play</i> is automatically released on PyPI thanks to travis when a new tag comes on the master branch.<br />If you are curious see <a href="https://github.com/pytest-dev/pytest-play/blob/master/.travis.yml">https://github.com/pytest-dev/pytest-play/blob/master/.travis.yml</a>. Instead for tags, versions and changelog management I use the always evergreen <a href="https://pypi.org/project/zest.releaser">https://pypi.org/project/zest.releaser</a> described in <a href="https://opensourcehacker.com/2012/08/14/high-quality-automated-package-releases-for-python-with-zest-releaser/">High quality automated package releases for Python with zest.releaser</a></li>
<li><a href="https://pyup.io/">pyup</a> service opens a pull request on github for <i>pytest-play-docker</i> to be merged on master (if you want to reduce the pull requests rate configure pyup properly with a daily or weekly policy, see <a href="https://pyup.io/docs/bot/config/">pyup docs</a>)</li>
<li><a href="https://travis-ci.org/">travis</a> runs the build for the above pull request against the pull request branch and pull request status updated (pass or failing)</li>
<li>receive an email notification about a new pull requests</li>
<li>if the pull request status is pass merge pull request on master on <a href="https://github.com/davidemoro/pytest-play-docker/pulls">https://github.com/davidemoro/pytest-play-docker/pulls</a> otherwise investigate</li>
<li>travis runs tests again on master merge, if the current branch is master and tests passed:</li>
<ul>
<li>docker push to Docker Hub on <a href="https://hub.docker.com/r/davidemoro/pytest-play">https://hub.docker.com/r/davidemoro/pytest-play</a></li>
</ul>
</ul>
All tests executions run against the docker build so there is a warranty that what is pushed to Docker Hub works fine (it doesn't check only that the build was successful but it runs integration tests against the docker build), so no versions incompatibilities, no integration issues between all the integrated third party <i>pytest-play</i> plugins and no issues due to the operative system integration (e.g., I recently experienced an issue on alpine linux with a pip install <i>psycopg2-binary</i> that apparently worked fine but if you try to import <i>psycopg2</i> inside your code you get an unexpected import error due to a recent issue reported here <a href="https://github.com/psycopg/psycopg2/issues/684">https://github.com/psycopg/psycopg2/issues/684</a>).<br />
<br />
So now every time you run a command like the following one (see a complete and working example here <a href="https://davidemoro.blogspot.com/2019/02/api-rest-testing-pytest-play-yaml-chuck-norris.html">https://davidemoro.blogspot.com/2019/02/api-rest-testing-pytest-play-yaml-chuck-norris.html</a>):<br />
<blockquote class="tr_bq">
<i>docker run --rm -v $(pwd):/src davidemoro/pytest-play</i></blockquote>
you know what was the workflow for every automated docker push for <i>pytest-play</i>.<br />
<h2>
Acknowledgements</h2>
Many thanks to <a href="https://github.com/neg3ntropy">Andrea Ratto</a> for the 10 minutes travis build speedup due to Docker cache, from ~11 minutes to ~1 minute is a <b>huge improvement</b> indeed! It was possible thanks to the <i>docker pull davidemoro/pytest-play</i> command, the build with the <i>--cache-from davidemoro/pytest-play</i> option and running the longest steps in a separate and cacheable step (e.g., the very very long <i>cassandra-driver</i> compilation moved to <i>requirements_cassandra.txt</i> will be executed only if necessary).<br />
<br />
Relevant technical details about <a href="https://github.com/davidemoro/pytest-play-docker">pytest-play-docker</a> follows (some minor optimizations are still possible saving in terms of final size).<br />
<br />
<script src="https://gist.github.com/davidemoro/39e09baba101657dcf0a90578c68e0f9.js"></script>
<noscript>
pytest-play-docker/.travis.yml<br />
<blockquote class="tr_bq">
<i>sudo: required<br />services:<br />- docker<br />- ...</i><br />
<i>env:<br /> global:<br /> - IMAGE_NAME=davidemoro/pytest-play<br /> - secure: ...<br />before_script:<br />- ...</i><br />
<i>script:<br />- travis_wait docker pull python:3.7<br />- travis_wait docker pull "$IMAGE_NAME:latest"<br />- travis_wait 25 docker build --cache-from "$IMAGE_NAME:latest" -t "$IMAGE_NAME" .<br />- docker run -i --rm -v $(pwd)/tests:/src --network host -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock $IMAGE_NAME --splinter-webdriver=remote<br /> --splinter-remote-url=$REMOTE_URL<br />deploy:<br /> provider: script<br /> script: bash docker_push<br /> on:<br /> branch: master</i></blockquote>
<br />
pytest-play-docker/docker_push<br />
<blockquote class="tr_bq">
<i>#!/bin/bash<br />echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin<br />docker tag "$IMAGE_NAME" "$IMAGE_NAME:$TRAVIS_COMMIT"<br />docker tag "$IMAGE_NAME" "$IMAGE_NAME:latest"<br />docker push "$IMAGE_NAME":"$TRAVIS_COMMIT"<br />docker push "$IMAGE_NAME":latest</i></blockquote>
</noscript>
<br />
<h2>
Feedback</h2>
Any feedback will be always appreciated.<br />
<br />
Do you like the Docker hub push process for <a href="https://github.com/pytest-dev/pytest-play">pytest-play</a>? Let me know becoming a pytest-play stargazer!
<!-- Place this tag where you want the button to render. -->
<a aria-label="Star pytest-dev/pytest-play on GitHub" class="github-button" data-icon="octicon-star" data-show-count="true" data-size="large" href="https://github.com/pytest-dev/pytest-play">Star</a>
<!-- Place this tag in your head or just before your close body tag. -->
<script async="" defer="" src="https://buttons.github.io/buttons.js"></script>Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-36353807785054657192019-02-02T00:09:00.001+01:002019-02-02T08:34:06.385+01:00API/REST testing like Chuck Norris with pytest play using YAML<br />
In this article we will see how to write HTTP API tests with pytest using YAML files thanks to <a href="https://github.com/pytest-dev/pytest-play">pytest-play</a> >= 2.0.0 (pytest-play provides support for Selenium, MQTT, SQL and more. See <a href="https://github.com/pytest-dev/pytest-play#third-party-pytest-play-plugins">third party pytest-play plugins</a>).<br />
<br />
The guest star is <i>Chuck Norris</i> thanks to the public JSON endpoint available at <a href="https://api.chucknorris.io/">https://api.chucknorris.io/</a> so you will be able to run your test by your own following this example.<br />
<br />
Obviously this is a joke because Chuck Norris cannot fail so tests are not needed.<br />
<h2>
Prerequisites and installation</h2>
Installation is not needed, the only prerequisite is <a href="https://www.docker.com/">Docker</a> thanks to <a href="https://hub.docker.com/r/davidemoro/pytest-play">https://hub.docker.com/r/davidemoro/pytest-play</a>.<br />
<br />
Inside the above link you'll find the instructions needed for installing Docker for any platform.<br />
<br />
If you want to run this example without docker install <i>pytest-play</i> with the external plugin <a href="https://github.com/davidemoro/play_requests">play_requests</a> based on the fantastic <a href="http://docs.python-requests.org/">requests</a> library (<i>play_requests</i> is already included in docker container).<br />
<h2>
Project structure</h2>
You need:<br />
<ul>
<li>a folder (e.g., <i>chuck-norris-api-test</i>)</li>
<li>one or more <i>test_XXX.yml</i> files containing your steps (<i>test_</i> and<i> .yml</i> extension matter)</li>
</ul>
For example:<br />
<br />
<script src="https://gist.github.com/davidemoro/05fefed4084025eb256b8e2998a32bc2.js"></script>
As you can see each scenario will be repeated for any item you provide in <i>test_data</i> structure.<br />
<br />
The first example asserts that the categories list contains some values against this endpoint <a href="https://api.chucknorris.io/jokes/categories">https://api.chucknorris.io/jokes/categories</a>; the second example shows how to search for category (probably Chuck Norris will find you according to this Chuck Norris fact "<i>You don't find Chuck Norris, Chuck Norris finds you!</i>")<br />
<br />
Alternatively you can checkout this folder:<br />
<ul>
<li><a href="https://github.com/pytest-dev/pytest-play/tree/features/examples/chuck-norris-api-test">https://github.com/pytest-dev/pytest-play/tree/features/examples/chuck-norris-api-test</a></li>
</ul>
Documentation and options:<br />
<ul>
<li><a href="https://github.com/davidemoro/play_requests">https://github.com/davidemoro/play_requests</a></li>
</ul>
<ul>
</ul>
<h2>
Usage</h2>
Visit the project folder and run the following command line command:<br />
<br />
<blockquote class="tr_bq">
<i>docker run --rm -v $(pwd):/src davidemoro/pytest-play</i></blockquote>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOZDgVfHngrLQ8y6GNHuH4EWUvJjGS5uGxTW4C5lVVjksY2ZiAMw3PwifjXGgDB0wOMRVTqOa48qJ78Oon65WwDx2ITFk4yLPeMaI251XCtr9YeTQlOmobEx_Cs9ImlJwESo3O1jNE/s1600/pytest-play-api.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="741" data-original-width="1600" height="185" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOZDgVfHngrLQ8y6GNHuH4EWUvJjGS5uGxTW4C5lVVjksY2ZiAMw3PwifjXGgDB0wOMRVTqOa48qJ78Oon65WwDx2ITFk4yLPeMaI251XCtr9YeTQlOmobEx_Cs9ImlJwESo3O1jNE/s400/pytest-play-api.png" width="400" /></a></div>
<br />
<br />
You can append extra standard pytest variables like <i>-x</i>, <i>--pdb</i> and so on. See <a href="https://docs.pytest.org/en/latest/">https://docs.pytest.org/en/latest/</a><br />
<br />
<h2>
Homeworks</h2>
It's time to show off with a GET roundhouse kick! Ping me on twitter <a href="https://twitter.com/davidemoro">@davidemoro</a> sharing your pytest-play implementation against the random Chuck Norris fact generator by category!<br />
<br />
GET <a href="https://api.chucknorris.io/jokes/random?category=dev">https://api.chucknorris.io/jokes/random?category=dev</a><br />
<blockquote class="tr_bq">
{<br />
"category": ["dev"],<br />
"icon_url": "https:\/\/assets.chucknorris.host\/img\/avatar\/chuck-norris.png",<br />
"id": "yrvjrpx3t4qxqmowpyvxbq",<br />
"url": "https:\/\/api.chucknorris.io\/jokes\/yrvjrpx3t4qxqmowpyvxbq",<br />
"value": "Chuck Norris protocol design method has no status, requests or responses, only commands."<br />
} </blockquote>
<h2>
Do you like pytest-play?</h2>
Let's get in touch for any suggestion, contribution or comments. Contributions will be very appreciated too!
<br />
<!-- Place this tag where you want the button to render. -->
<a aria-label="Star pytest-dev/pytest-play on GitHub" class="github-button" data-icon="octicon-star" data-show-count="true" data-size="large" href="https://github.com/pytest-dev/pytest-play">Star</a>
<!-- Place this tag in your head or just before your close body tag. -->
<script async="" defer="" src="https://buttons.github.io/buttons.js"></script>Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-68786138975590644122018-04-18T00:11:00.000+02:002019-02-22T23:15:47.150+01:00Hello pytest-play!<a href="https://github.com/davidemoro/pytest-play">pytest-play</a> is a <strike>rec</strike>&play (rec not yet available) <i>pytest</i> plugin that let you execute a set of actions and assertions using commands serialized in JSON format. It tries to make test automation more affordable for non programmers or non Python programmers for browser, functional, API, integration or system testing thanks to its pluggable architecture and <a href="https://github.com/davidemoro/pytest-play#third-party-pytest-play-plugins">third party plugins</a> that let you interact with the most common databases and systems.<br />
<br />
In addition it provides also some facilitations for writing browser UI actions (e.g., implicit waits before interacting with an input element. The <a href="https://www.cypress.io/">Cypress</a> framework for me was a great source of inspiration) and asynchronous checks (e.g., wait until a certain condition is true).<br />
<br />
You can use <i>pytest-play</i> programmatically (e.g., use the <i>pytest-play</i> engine <b><i>as a library</i></b> for <i>pytest-play</i> standalone scenarios or using the pytest-play API implementing BDD steps).<br />
<br />
Starting from <i>pytest-play</i>>1.4.x it was introduced a new <b>experimental</b> feature that let you use <i>pytest-play</i> <b><i>as a framework</i></b> creating <b>Python-free</b> automated tests based on a JSON based serialization format for actions and assertions (in the next future the more user friendly YAML format will be supported).<br />
<br />
So now depending on your needs and skills you can choose to use <i>pytest-play</i> as a library or as a framework.<br />
<br />
In this article I'm going to show how to implement a <a href="https://plone.org/">Plone CMS</a> based login test using the python-free approach without having to write any line of Python code.<br />
<h3>
What is pytest-play and why it exists</h3>
In this section I'm going to add more information about the <i>pytest-play</i> approach and other considerations: if you want to see now how to implement our Python-free automated login test jump to the next section!<br />
<h4>
Hyper specialized tool problems</h4>
<br />
There are many commercial products or tools that offer solutions for API testing only, browser testing only. Sometimes hyper specialized tools might fit your needs (e.g., a content management system based web application) but sometimes they are not helpful for other distributed applications.<br />
<br />
For example an API-only platform is not effective for testing a CQRS based application. It is not useful testing only HTTP 200 OK response, you should test that all the expected commands are generated on the event store (e.g., Cassandra) or other side effects.<br />
<br />
Another example for an IoT applications and UI/browser only testing platforms. You cannot test reactive web apps only with a browser, you should control also simulated device activities (e.g., MQTT, queues, API) for messages/alarms/reports) or any other web based interactions performed by other users (e.g., HTTP calls); you might need to check asynchronously the expected results on web sockets instead of using a real browser implementing when some actions are performed.<br />
<h4>
What is pytest-play </h4>
In other words <i>pytest-play</i> is an open source testing solution based on the pytest framework that let you:<br />
<ul>
<li>write actions and cross assertions against different protocols and test levels in the same scenario (e.g., check HTTP response and database assertions)</li>
<li>minimize the development of Selenium-based asynchronous wait functions before interacting with input elements thanks to implicit wait functions that let you interact with web elements only when they are ready. You just focus on user actions, you are more productive and you reduce the chance of writing bad or not robust asynchronous wait functions</li>
<li>implement polling-based asynchronous waiter commands based on custom expressions when needed</li>
</ul>
using a serialization format (JSON at this time of writing, YAML in the next future) that should be more affordable for non technical testers, non programmers or programmers with no Python knowledge.<br />
<br />
Potentially you will be able to share and execute a new scenario not yet included in your test library copying and pasting a <i>pytest-play</i> JSON to a Jenkins build with parameters form like the following one (see the <b>PLAY</b> textarea):<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><img border="0" data-original-height="883" data-original-width="571" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIaUuziuPB99J_oi_5c7lIZ14H-Scx4lcVmBb35zc_1ir9hpA1DNPY2ucX60hg5u2td5krM7rBSxEPdaFFAzQAsweuoeNiXhzqSrCupXlGSCqWjMu4pqI4pFdeCd5E4b6hQF48XF7v/s640/jenkins_project_qa.PNG" style="margin-left: auto; margin-right: auto;" width="412" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;">From <a href="http://davidemoro.blogspot.it/2018/03/test-automation-python-pytest-jenkins.html">http://davidemoro.blogspot.it/2018/03/test-automation-python-pytest-jenkins.html</a></td></tr>
</tbody></table>
<br />
In addition if you are a technical user you can extend it
writing your own plugins, you can provide the integration with external
tools (e.g., test management tools, software metrics engines, etc), you
can decide the test abstraction depending on deadlines/skills/strategy
(e.g., use plain json files, a programmatic approach based on json
scenarios or BDD steps based on pytest-play).<br />
<h4>
What pytest-play is not</h4>
For example <i>pytest-play</i> doesn't provide a test scenario recorder but it enforces users to understand what they are doing.<br />
<br />
It requires a very very little programming knowledge for writing some assertions using simple code expressions but with a little training activity it is still affordable by non programmers (you don't have to learn a programming language, just some basic assertions).<br />
<br />
It is not feature complete but it is free software.<br />
<br />
If you want to know more in this previous article I've talked about:<br />
<ul>
<li><a href="http://davidemoro.blogspot.it/2018/03/test-automation-python-pytest-jenkins.html">test automation framework thoughts plus a comprehensive, detailed look at using pytest and Jenkins for automated testing</a>. </li>
</ul>
<h3>
A pytest-play example: parametrized login (featuring Plone CMS)</h3>
In this example we'll see how to write and execute pure json <i>pytest-play</i> scenarios with test data decoupled by the test implementation and test parametrization. I'm using the available online <a href="https://www.plone-demo.info/">Plone 5 demo site</a> kindly hosted by <b>Andreas Jung</b> (<a href="http://www.zopyx.com/">www.zopyx.com</a>).<br />
<br />
The project is available here:<br />
<ul>
<li><a href="https://github.com/davidemoro/pytest-play-plone-example">https://github.com/davidemoro/pytest-play-plone-example</a></li>
</ul>
The tests could be launched this way as a normal pytest project once you installed pytest and the dependencies (there is a <i>requirements.txt</i> file, see the above link):<br />
<blockquote class="tr_bq">
<div class="application-main " role="main">
<div class="" itemscope="" itemtype="http://schema.org/SoftwareSourceCode">
<div data-pjax-container="" id="js-repo-pjax-container">
<div class="container new-discussion-timeline experiment-repo-nav ">
<div class="repository-content ">
<div class="readme boxed-group clearfix announce instapaper_body rst" id="readme">
<br />
<br />
<br />
<br />
<article class="markdown-body entry-content" itemprop="text"><pre><pre>$ pytest --variables env-ALPHA.yml --splinter-webdriver firefox --splinter-screenshot-dir /tmp -x</pre>
</pre>
</article></div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<table class="highlight tab-size js-file-line-container" data-tab-size="8"><tbody></tbody></table>
Where the you can have multiple environment/variable files. E.g., <a href="https://github.com/davidemoro/pytest-play-plone-example/blob/master/env-ALPHA.yml">env-ALPHA.yml</a> containing the alpha base url and any other variables:<br />
<blockquote class="tr_bq">
<pre><pre>pytest-play:
base_url: https://plone-demo.info</pre>
</pre>
</blockquote>
Our login <a href="https://github.com/davidemoro/pytest-play-plone-example/blob/master/test_login.json">test_login.json</a> scenario contains (as you can see there are <b>NO</b> asynchronous waits because they are not needed for basic examples so you can focus on actions and assertions thanks to implicit waits):<br />
<blockquote class="tr_bq">
<pre><pre>{
"steps": [
{
"comment": "visit base url",
"type": "get",
"url": "$base_url"
},
{
"comment": "click on login link",
"locator": {
"type": "id",
"value": "personaltools-login"
},
"type": "clickElement"
},
{
"comment": "provide a username",
"locator": {
"type": "id",
"value": "__ac_name"
},
"text": "$username",
"type": "setElementText"
},
{
"comment": "provide a password",
"locator": {
"type": "id",
"value": "__ac_password"
},
"text": "$password",
"type": "setElementText"
},
{
"comment": "click on login submit button",
"locator": {
"type": "css",
"value": ".pattern-modal-buttons > input[name=submit]"
},
"type": "clickElement"
},
{
"comment": "wait for page loaded",
"locator": {
"type": "css",
"value": ".icon-user"
},
"type": "waitForElementVisible"
}
]
} </pre>
</pre>
</blockquote>
Plus an optional test scenario metadata file <a href="https://github.com/davidemoro/pytest-play-plone-example/blob/master/test_login.ini">test_login.ini</a> that contains pytest keyword and decoupled test data: <br />
<blockquote class="tr_bq">
<pre><pre>[pytest]
markers =
login
test_data =
{"username": "siteadmin", "password": "siteadmin"}
{"username": "editor", "password": "editor"}
{"username": "reader", "password": "reader"}</pre>
</pre>
</blockquote>
Thanks to the metadata file you have just one scenario and it will be executed 3 times (as many times as test data rows)!<br />
<br />
Et voilà, let's see it in action out scenario without having to write any line of Python code:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjklllP0I0P_NBgvC4BRX7SmmvtU6A005s32Bqn39JWUYikVZA0t6OqNGW7NoOFcOOpEB4-CwEeRZbWaH8CA0LIuq9wDYpK3JH-jI5dQhr5nsgHc3ZHyFTMNMd0FPFa4v4i_R5tQ50z/s1600/pytest-play-launch.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="672" data-original-width="1600" height="267" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjklllP0I0P_NBgvC4BRX7SmmvtU6A005s32Bqn39JWUYikVZA0t6OqNGW7NoOFcOOpEB4-CwEeRZbWaH8CA0LIuq9wDYpK3JH-jI5dQhr5nsgHc3ZHyFTMNMd0FPFa4v4i_R5tQ50z/s640/pytest-play-launch.PNG" width="640" /></a></div>
<br />
There is only a warning I have to remove but it worked and we got exactly 3 different test runs for our login scenario as expected! <br />
<h3>
pytest-play status</h3>
<i>pytest-play</i> should be still considered experimental software and many features needs to be implemented or refactored:<br />
<ul>
<li>yaml instead of json. YAML will become the primary configuration
format (it should be more user friendly as suggested by some users)</li>
<li>API should not be considered stable until future 2.x version </li>
<li>improve API testing when using pure json scenarios registering functions (e.g., invoke a function returning a valid authentication bearer for authenticated API testing)</li>
<li>implement some python <a href="http://docs.python-requests.org/en/master/">requests</a> library features not yet implemented in <a href="https://github.com/davidemoro/play_requests">play_requests</a> (e.g., cookies)</li>
<li>refactor parametrization and templating (Jinja?)</li>
<li>implement additional Selenium actions (e.g., right clicks, upload files, etc) </li>
<li>implement other cool Cypress ideas enabling non expert testers in writing more robust Selenium scenarios</li>
<li>add page object abstraction in <i>pytest-play</i> based Selenium scenarios with new commands that let you interact with page regions and interact with complex UI widgets</li>
<li>ownership change, waiting for <i>pytest-dev</i> core developers approval. Probably soon the ownership will change from <i>davidemoro/pytest-play</i> to <i>pytest-dev/pytest-play</i> once the approvation iter will finish </li>
</ul>
<h3>
PyCon Nove @ Florence</h3>
If you are going to attending next <a href="https://www.pycon.it/en/">PyCon Nove</a> in Florence don't miss the following <i>pytest-play</i> talk presented by <a href="https://www.pycon.it/conference/p/serena-martinetti">Serena Martinetti</a>:<br />
<ul>
<li><a href="https://www.pycon.it/conference/talks/integration-tests-ready-to-use-with-pytest-play">https://www.pycon.it/conference/talks/integration-tests-ready-to-use-with-pytest-play</a></li>
</ul>
<ul>
</ul>
<h3>
Do you like pytest-play?</h3>
Tweets about pytest-play happens on <a href="http://davidemoro/">@davidemoro</a>.<br />
Positive or negative feedback is always appreciated. If you find interesting the concepts behind <i>pytest-play</i> let me know with a tweet, add a new <i>pytest-play</i> adapter and/or add a GitHub star if you liked it:<br />
<br />
<!-- Place this tag where you want the button to render. -->
<a aria-label="Star pytest-dev/pytest-play on GitHub" class="github-button" data-icon="octicon-star" data-show-count="true" data-size="large" href="https://github.com/pytest-dev/pytest-play">Star</a>
<!-- Place this tag in your head or just before your close body tag. -->
<script async="" defer="" src="https://buttons.github.io/buttons.js"></script>
<br />
<h3>
Updates</h3>
<ul>
<li><i><b>[20190222]</b></i> pytest-play 2.0.0 released, no more backwards compatible with 1.x format. Using YAML syntax instead of JSON now and many other improvements and new features. See <a href="https://davidemoro.blogspot.com/2019/02/api-rest-testing-pytest-play-yaml-chuck-norris.html">https://davidemoro.blogspot.com/2019/02/api-rest-testing-pytest-play-yaml-chuck-norris.html</a> </li>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-28939929632199385102018-03-27T23:48:00.001+02:002019-02-22T23:21:38.363+01:00Test automation framework thoughts and examples with Python, pytest and JenkinsIn this article I'll share some personal thoughts about Test Automation Frameworks; you can take inspiration from them if you are going to evaluate different test automation platforms or assess your current test automation solution (or solutions).<br />
<br />
Despite it is a generic article about test automation, you'll find many examples explaining how to address some common needs using the Python based test framework named pytest and the Jenkins automation server: use the information contained here just as a comparison and feel free to comment sharing alternative methods or ideas coming from different worlds.<br />
<br />
It contains references to some well (or less) known pytest plugins or testing libraries too.<br />
<br />
Before talking about automation and test automation framework features and characteristics let me introduce the most important test automation goal you should always keep in mind.<br />
<h3>
Test automation goals: ROI</h3>
You invest in automation for a future return of investment.<br />
Simpler approaches let you start more quickly but in the long term
they don't perform well in terms of ROI and vice versa. In addition the initial complexity due to a higher level of abstraction may produce better results in the medium or long term: better ROI and some benefits for non technical testers too. Have a look at the test automation engineer ISTQB certification syllabus for more information:<br />
<ul>
<li><a href="https://www.istqb.org/downloads/category/48-advanced-level-test-automation-engineer-documents.html">https://www.istqb.org/downloads/category/48-advanced-level-test-automation-engineer-documents.html</a></li>
</ul>
<br />
So what I mean is that test automation is not easy: it doesn't mean just recording some actions or write some automated test procedures because <b>how</b> you decide to automate things affects the <b>ROI</b>. Your test automation strategy should consider your tester technical skills now and future evolutions, considerations about how to improve your system testability (is your software testable?), good test design and architecture/system/domain knowledge. In other words be aware of vendors selling "silver bullet" solutions promising smooth test automation for everyone, especially rec&play solutions: there are no silver bullets.<br />
<h3>
Test automation solution features and characteristics</h3>
A test automation solution should be enough generic and flexible, otherwise there is the risk of having to adopt different and maybe incompatible tools for different
kind of tests. Try to imagine the mess of having the following situation: one tool or commercial service for browser based tests only based on rec&play, one tool for API testing only, performance test frameworks that doesn't let you reuse existing scenarios, one tool for BDD only scenarios, different Jenkins jobs with different settings for each different tool, no test management tool integration, etc. A unique solution, if possible, would be better: something that let you choose the level of abstraction and that doesn't force you. Something that let you start simple and that follow your future needs and the skill evolution of your testers.<br />
That's one of the reasons why I prefer pytest over an hyper specialized solution like <a href="https://github.com/behave/behave">behave</a> for example: if you combine <a href="https://docs.pytest.org/en/latest/">pytest</a>+<a href="https://github.com/pytest-dev/pytest-bdd">pytest-bdd</a> you can write BDD scenarios too and you are not forced to use a BDD only capable test framework (without having the pytest flexibility and tons of additional plugins).<br />
<br />
And now, after this preamble, an unordered list of features or characteristics that you may consider for your test automation solution software selection:<br />
<ul>
<li>fine grained test selection mechanism that allows to be very selective when you have to choose which tests you are going to launch</li>
<li>parametrization</li>
<li>high reuse</li>
<li>test execution logs easy to read and analyze</li>
<li>easy target environment switch</li>
<li>block on first failure</li>
<li>repeat your tests for a given amount of times</li>
<li>repeat your tests until a failure occurs</li>
<li>support parallel executions</li>
<li>provide integration with third party software like test management tools</li>
<li>integration with cloud services or browser grids</li>
<li>execute tests in debug mode or with different log verbosity</li>
<li>support random tests execution order (the order should be reproducible if some problems occur thanks to a random seed if needed)</li>
<li>versioning support</li>
<li>integration with external metrics engine collectors</li>
<li>support different levels of abstraction (e.g., keyword driven testing, BDD, etc)</li>
<li>rerun last failed</li>
<li>integration with platforms that let you test against a large combination of OS and browsers if needed</li>
<li>are you able to extend your solution writing or installing third party plugins? </li>
</ul>
Typically a test automation engineer will be able to drive automated test runs using the framework command line interface (CLI) during test development but you'll find out very soon that you need an automation server for long running tests, scheduled builds, CI and here it comes Jenkins. Jenkins could be used by non technical testers for launching test runs or initialize an environment with some test data.<br />
<h3>
Jenkins</h3>
What is Jenkins? From the Jenkins website:<br />
<blockquote class="tr_bq">
<i><span class="st">Continuous Integration and Continuous Delivery. As an extensible automation server, <i>Jenkins</i> can be used as a simple CI server or turned into the continuous delivery hub for any project.</span></i></blockquote>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2WEZKLBmgpgfESDSWF0ZlKo2FU1-VCn05apbc8OU7J66fOGSy1AEKZkWIVdIPJuTs1PEupb7P2-v_e-wNvJESPX3Dve-vjsmOVu3b9QVfOzCLTrgzsMs3J5HtOu61Z_a47iBLqkpu/s1600/Jenkins.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="550" data-original-width="550" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2WEZKLBmgpgfESDSWF0ZlKo2FU1-VCn05apbc8OU7J66fOGSy1AEKZkWIVdIPJuTs1PEupb7P2-v_e-wNvJESPX3Dve-vjsmOVu3b9QVfOzCLTrgzsMs3J5HtOu61Z_a47iBLqkpu/s200/Jenkins.jpg" width="200" /></a>So thanks to Jenkins everyone can launch a parametrized automated test session just using a browser: no command line and nothing installed on your personal computer. So more power to non technical users thanks to Jenkins!<br />
<br />
With Jenkins you can easily schedule recurrent automatic test runs, start remotely via external software some parametrized test runs, implement a CI and many other things. In addition as we will see Jenkins is quite easy to configure and manage thanks to through the web configuration and/or Jenkins pipelines.<br />
<br />
Basically Jenkins is very good at starting builds and generally jobs. In this case Jenkins will be in charge of launching our parametrized automated test runs.<br />
<br />
And now let's talk a little bit of Python and the pytest test framework.<br />
<h3>
Python for testing </h3>
I don't know if there are some articles talking about statistics on the net about the correlation between Test Automation Engineer job offers and the Python programming language, with a comparison between other programming languages. If you find a similar resource share with me please!<br />
<br />
My personal feeling observing for a while many Test Automation Engineer job offers (or any similar QA job with some automation flavor) is that the Python word is very common. Most of times is one of the nice to have requirements and other times is mandatory.<br />
<br />
Let's see why the programming language of choice for many QA departments is Python, even for companies that are not using Python for building their product or solutions.<br />
<h4>
Why Python for testing</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUjkbZl6CbFNbb9Dn41rYBs5XTFZVbUzRawx28pinjxX4TWagO2YER2JB3qC4Bk1084xCH7kLQzldu_N6giINBDsDqzozUohBJ_QpjSC2I5wdlUNGunIdQBYtaVQFR77wfElK-EUez/s1600/python-logo.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="82" data-original-width="290" height="55" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUjkbZl6CbFNbb9Dn41rYBs5XTFZVbUzRawx28pinjxX4TWagO2YER2JB3qC4Bk1084xCH7kLQzldu_N6giINBDsDqzozUohBJ_QpjSC2I5wdlUNGunIdQBYtaVQFR77wfElK-EUez/s200/python-logo.png" width="200" /></a></div>
Why Python is becoming so popular for test automation? Probably because it is more affordable for people with no or little programming knowledge compared to other languages. In addition the Python community is very supportive and friendly especially with new comers, so if you are planning to attend any Python conference be prepared to fall in love with this fantastic community and make new friends (friends, not only connections!). For example at this time of writing you are still in time for attending <b>PyCon Nove</b> 2018 in the beautiful <b>Florence </b>(even better if you like history, good wine, good food and meet great people): <br />
<ul>
<li><a href="https://www.pycon.it/">https://www.pycon.it</a></li>
</ul>
You can just compare the most classical hello world, for example with Java:<br />
<blockquote class="tr_bq">
<i>public class HelloWorld { </i></blockquote>
<blockquote class="tr_bq">
<i> public static void main(String[] args) { </i></blockquote>
<blockquote class="tr_bq">
<i> System.out.println("Hello, World!");</i></blockquote>
<blockquote class="tr_bq">
<i> } </i></blockquote>
<blockquote class="tr_bq">
<i>}</i></blockquote>
and compare it with the Python version now:<br />
<blockquote class="tr_bq">
<i>print("Hello, World!")</i></blockquote>
Do you see any difference? If you are trying to explain to a non programmer how to print a line in the terminal window with Java you'll have to introduce public, static, void, class, System, installing a runtime environment choosing from different version, installing an IDE, running javac, etc and only at the end you will be able to see something printed on the screen. With Python, most of times it comes preinstalled in many distributions, you just focus on what to need to do. Requirements: a text editor and Python installed. If you are not experienced you start with a simple approach and later you can progressively learn more advanced testing approaches.<br />
<br />
And what about test assertions? Compare for example a Javascript based assertions:<br />
<blockquote class="tr_bq">
<i>expect(b).not.toEqual(c);</i></blockquote>
with the Python version:<br />
<blockquote class="tr_bq">
<i>assert b != c</i></blockquote>
So no <i>expect(a).not.toBeLessThan(b)</i>, <i>expect(c >= d).toBeTruthy()</i> or <i>expect(e).toBeLessThan(f)</i>: with Python you just say assert a >= 0 so nothing to remember for assertions!<br />
<br />
Python is a big fat and very powerful programming language but it follows a "<i>pay only for what you eat</i>" approach.<br />
<h4>
Why pytest</h4>
If Python is the language of your choice you should consider the pytest framework and
its high quality community
plugins and I think it is a good starting point for building your own test automation solution. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqfVdk-Gsi-G8zTfiQeAAQsuH4Fjjza-XWM1sb84Fevzk9nMLRnhY5M3GAqo9u_XiBKL03wNx1W6wfM5yC_RAsY7ojsOjKTR36nlY_dXHFNr46esguZ5r9NHbRG3HDtC-vU1w4juSb/s1600/pytest.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="143" data-original-width="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqfVdk-Gsi-G8zTfiQeAAQsuH4Fjjza-XWM1sb84Fevzk9nMLRnhY5M3GAqo9u_XiBKL03wNx1W6wfM5yC_RAsY7ojsOjKTR36nlY_dXHFNr46esguZ5r9NHbRG3HDtC-vU1w4juSb/s1600/pytest.png" /></a></div>
The pytest framework (<a href="https://docs.pytest.org/en/latest/">https://docs.pytest.org/en/latest/</a>) makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries.<br />
<br />
Most important pytest features:<br />
<ul>
<li>simple assertions instead of inventing assertion APIs (<i>.not.toEqual</i> or <i>self.assert*</i>)</li>
<li>auto discovery test modules and functions</li>
<li>effective CLI for controlling what is going to be executed or skipped using expressions</li>
<li>fixtures, easy to manage fixtures lifecycle for long-lived test resources and parametrized features make it easy and funny implementing what you found hard and boring with other frameworks</li>
<li>fixtures as function arguments, a dependency injection mechanism for test resources</li>
<li>overriding fixtures at various levels</li>
<li>framework customizations thanks to pluggable hooks</li>
<li>very large third party plugins ecosystem</li>
</ul>
I strongly suggest to have a look at the pytest documentation but I'd like to make some examples showing something about fixtures, code reuse, test parametrization and improved maintainability of your tests. If you are not a technical reader you can skip this section.<br />
<br />
I'm trying to explain fixtures with practical examples based on answers and questions:<br />
<ul>
<li><i><b>When should be created a new instance of our test resource?</b></i><br />You can do that with the fixture <i>scope</i> (<i>session</i>, <i>module</i>, <i>class</i>, <i>function</i> or more advanced options like <i>autouse</i>). Session means that your test resource will live for the entire session, module/class for all the tests contained in that module or class, with function you'll have an always fresh instance of your test resource for each test</li>
<li><i><b>How can I determine some teardown actions at the end of the test resource life?</b></i><br />You can add a sort of fixture finalizer after the <i>yield</i> line that will be invoked at the end of our test resource lifecycle. For example you can close a connection, wipe out some data, etc.</li>
<li><i><b>How can I execute all my existing tests using that fixture as many as your fixture configurations?</b></i><br />You can do that with <i>params</i>. For example you can reuse all your existing tests verifying the integration with different real databases, smtp servers. Or if you have the web application offering the same features deployed with a different look&feel for different brands you can reuse all your existing functional UI tests thanks to pytest's fixture parametrization and a page objects pattern where for different look&feel I don't mean only different CSS but different UI components (e.g. completely different datetime widgets or navigation menu), components disposition in page, etc.</li>
<li><i><b>How can I decouple test implementation and test data?</b></i> Thanks to <i>parametrize</i> you can decouple them and write just one time your test implementation. Your test will be executed as many times as your different test data</li>
</ul>
Here you can see an example of fixture parametrization (the <i>test_smtp</i> will be executed twice because you have 2 different fixture configurations): <br />
<blockquote class="tr_bq">
<i>import pytest</i><br />
<i>import smtplib</i><br />
<br />
<i>@pytest.fixture(<b>scope</b>="module",</i><br />
<i> <b>params</b>=["smtp1.com", "smtp2.org"])</i><br />
<i>def smtp(request):</i><br />
<i> smtp = smtplib.SMTP(request.param, 587, timeout=5)</i><br />
<i> <b>yield smtp</b></i><br />
<i> print("finalizing %s" % smtp)</i><br />
<i> smtp.close() </i><br />
<i><br /></i>
<i>def test_smtp(smtp):</i><br />
<i> # use smtp fixture (e.g., smtp.sendmail(...))</i><br />
<i> # and make some assertions. <br /> # The same test will be executed twice (2 different params)</i><br />
<i> ...</i></blockquote>
And now an example of test parametrization:<br />
<blockquote class="tr_bq">
<i>import pytest</i><br />
<i>@pytest.mark.<b>parametrize</b>("test_input,expected", [</i><br />
<i> ("3+5", 8),</i><br />
<i> ("2+4", 6),</i><br />
<i> ("6*9", 42), ])</i><br />
<i>def test_eval(test_input, expected):</i><br />
<i> assert eval(test_input) == expected</i></blockquote>
For more info see:<br />
<ul>
<li><a href="https://docs.pytest.org/">https://docs.pytest.org</a> </li>
<li><a href="https://docs.pytest.org/en/latest/fixture.html">https://docs.pytest.org/en/latest/fixture.html</a></li>
<li><a href="https://docs.pytest.org/en/latest/fixture.html#parametrizing-fixtures">https://docs.pytest.org/en/latest/fixture.html#parametrizing-fixtures</a></li>
<li><a href="https://docs.pytest.org/en/latest/parametrize.html">https://docs.pytest.org/en/latest/parametrize.html</a></li>
</ul>
This is only pytest, as we will see there are many pytest plugins that extend the pytest core features.<br />
<h4>
Pytest plugins</h4>
There are hundreds of pytest plugins, the ones I am using more frequently are: <br />
<ul>
<li><a href="https://github.com/pytest-dev/pytest-bdd">pytest-bdd</a>, <span class="col-11 text-gray-dark mr-2" itemprop="about">BDD library for the pytest runner
</span></li>
<li><a href="https://github.com/pytest-dev/pytest-variables">pytest-variables</a>, <span class="col-11 text-gray-dark mr-2" itemprop="about">plugin for pytest that provides variables to tests/fixtures as a dictionary via a file specified on the command line</span> </li>
<li><a href="https://github.com/pytest-dev/pytest-html">pytest-html</a>, plugin for generating HTML reports for pytest results </li>
<li><a href="https://github.com/pytest-dev/pytest-selenium">pytest-selenium</a>, plugin for running Selenium with pytest </li>
<li><a href="https://github.com/pytest-dev/pytest-splinter">pytest-splinter</a>, a pytest-selenium alternative based on Splinter. <span class="col-11 text-gray-dark mr-2" itemprop="about">pPytest splinter and selenium integration for anyone interested in browser interaction in tests
</span> </li>
<li><a href="https://github.com/pytest-dev/pytest-xdist">pytest-xdist</a>, a py.test plugin for test parallelization, distributed testing and loop-on-failures testing modes </li>
<li><a href="https://pypi.python.org/pypi/pytest-testrail">pytest-testrail</a>, pytest plugin for creating TestRail runs and adding results on the TestRail test management tool</li>
<li><a href="https://github.com/pytest-dev/pytest-randomly">pytest-randomly</a>, <span class="col-11 text-gray-dark mr-2" itemprop="about">a pytest plugin to randomly order tests and control random seed
</span> (but there are different random order plugins if you search for "pytest random")</li>
<li><a href="https://github.com/pytest-dev/pytest-repeat">pytest-repeat</a>, plugin for pytest that makes it easy to repeat a single test, or multiple tests, a specific number of times. You can repeat a test or group of tests until a failure occurs</li>
<li><a href="https://github.com/tierratelematics/pytest-play">pytest-play</a>, an experimental <strike>rec&</strike>play
pytest plugin that let you execute a set of actions and assertions
using commands serialized in JSON format. Makes test automation more
affordable for non programmers or non Python programmers for browser,
functional, API, integration or system testing thanks to its pluggable
architecture and many plugins that let you interact with the most common
databases and systems. It provides also some facilitations for writing browser UI actions (e.g., implicit waits before interacting with an input element) and asynchronous checks (e.g., wait until a certain condition is true)</li>
</ul>
Python libraries for testing:<br />
<ul>
<li><a href="https://github.com/mozilla/PyPOM">PyPOM</a>, python page object model for Selenium or Splinter </li>
<li><a href="https://github.com/tierratelematics/pypom_form">pypom_form</a>, a PyPOM abstraction that extends the page object model applied to forms thanks to declarative form schemas </li>
</ul>
Scaffolding tools:<br />
<ul>
<li><a href="https://github.com/tierratelematics/cookiecutter-qa">cookiecutter-qa</a>, generates a test automation project ready to be integrated with Jenkins and with the test management tool TestRail that provides working hello world examples. It is shipped with all the above plugins and it provides examples based on raw splinter/selenium calls, a BDD example and a pytest-play example </li>
<li><a href="https://github.com/tierratelematics/cookiecutter-performance">cookiecutter-performance</a><span class="col-11 text-gray-dark mr-2" itemprop="about">, generate a tox based environment based on Taurus bzt for performance test. <a href="http://blazemeter.com/">BlazeMeter</a> ready for distributed/cloud performance tests. Thanks to the bzt/taurus pytest executor you will be able to reuse all your pytest based automated tests for performance tests</span></li>
</ul>
<h3>
Pytest + Jenkins together</h3>
We've discussed about Python, pytest and Jenkins main ingredients for our cocktail recipe (shaken, not stirred). Optional ingredients: integration with external test management tools and selenium grid providers.<br />
<br />
Thanks to pytest and its plugins you have a rich command line interface (CLI); with Jenkins you can schedule automated builds, setup a CI, let not technical users or other stakeholders executing parametrized test runs or building test always fresh test data on the fly for manual testing, etc. You just need a browser, nothing installed on your computer.<br />
<br />
Here you can see how our recipe looks like:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMAKzasWI9Yfo-k70Z-f0BMpQQPWjUcMxUYmOm6Gvq0gVd-yTuHD-O3_3MAnZhnrzhedmgFfwlqQpIFKGnx8w-mmj3d0NyScRjsOSzJMobwo9XXWXuNDNRxRq3w9A0jLGhQLbRhyj0/s1600/jenkins_project_qa.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="883" data-original-width="571" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMAKzasWI9Yfo-k70Z-f0BMpQQPWjUcMxUYmOm6Gvq0gVd-yTuHD-O3_3MAnZhnrzhedmgFfwlqQpIFKGnx8w-mmj3d0NyScRjsOSzJMobwo9XXWXuNDNRxRq3w9A0jLGhQLbRhyj0/s640/jenkins_project_qa.PNG" width="412" /></a></div>
<br />
Now lets comment all our features provided by the Jenkins "build with parameters" graphical interface, explaining option by option when and why they are useful.<br />
<h3>
Target environment (ENVIRONMENT)</h3>
In this article we are not talking about regular unit tests, the basis for your testing pyramid. Instead we are talking about system, functional, API, integration, performance tests to be launched against a particular instance of an integrated system (e.g., dev, alpha or beta environments).<br />
<br />
You know, unit
tests are good they are not sufficient: it is
important to verify if the integrated system (sometimes different
complex systems developed by different teams under the same or third party organizations) works fine as it is supposed to do. It is
important because it might happen that 100% unit tested systems doesn't
play well after the integration for many different reasons. So with unit
tests you take care about your code quality, with higher test levels
you take care about your product quality. Thanks to these tests you can
confirm an expected product behavior or criticize your product.<br />
<br />
So thanks to the <i>ENVIRONMENT</i> option you will be able to choose one of the target environments. It is important to be able to reuse all your tests and launch them against different environments without having to change your testware code. Under the hood the pytest launcher will be able to switch between different environments thanks to the <i>pytest-variables</i> parametrization using the <i>--variables</i> command line option, where each available option in the <i>ENVIRONMENT</i> select element is bound to a variables files (e.g., DEV.yml, ALPHA.yml, etc) containing what the testware needs to know about the target environment.<br />
<br />
Generally speaking you should be able to reuse your tests without any modification thanks to a parametrization mechanism.If your test framework doesn't let you change target environment and it forces you to modify your code, change framework.<br />
<h3>
Browser settings (BROWSER)</h3>
This option makes sense only if you are going to launch browser based tests otherwise it will be ignored for other type of tests (e.g., API or integration tests).<br />
<br />
You should be able to select a particular version of browser (latest or a specific version) if any of your tests require a real browser (not needed for API tests just for making one example) and preferably you should be able to integrate with a cloud system that allows you to use any combination of real browsers and OS systems (not only a minimal subset of versions and only Firefox and Chrome like several test platforms online do). Thanks to the <i>BROWSER</i> option you can choose which browser and version use for your browser based tests. Under the hood the pytest launcher will use the <i>--variables</i> command line option provided by the <i>pytest-variables</i> plugin, where each option is bound to a file containing the browser type, version and capabilities (e.g., <i>FIREFOX.yml</i>, <i>FIREFOX-xy.yml</i>, etc). Thanks to pytest, or any other code based testing framework, you will be able to combine browser interactions with non browser actions or assertions.<br />
<br />
A lot of big fat warnings about rec&play online platforms for browser testing or if you want to implement your testing strategy using only or too many browser based tests. You shouldn't consider only if they provide a wide range of OS and versions, the most common browsers. They should let you perform also non browser based actions or assertions (interaction with queues, database interaction, http POST/PUT/etc calls, etc). What I mean is that sometimes only a browser is not sufficient for testing your system: it might be good for a CMS but if you are testing an IoT platform you don't have enough control and you will write completely useless tests or low value tests (e.g., pure UI checks instead of testing reactive side effects depending on eternal triggers, reports, device activity simulations causing some effects on the web platform under test, etc).<br />
<br />
In addition be aware that some browser based online testing platforms doesn't use Selenium for their browser automation engine under the hood. For example during a software selection I found an online platform using some Javascript injection for implementing user actions interaction inside the browser and this might be very dangerous. For example let's consider a login page that takes a while before the input elements become ready for accepting the user input when some conditions are met. If for some reasons a bug will never unlock the disabled login form behind a spinner icon, your users won't be able to login to that platform. Using Selenium you'll get a failing result in case of failure due to a timeout error (the test will wait for elements won't never be ready to interact with and after few seconds it will raise an exception) and it's absolutely correct. Using that platform the test was green because under the hood the input element interaction was implemented using DOM actions with the final result of having all your users stuck: how can you trust such platform?<br />
<h3>
OS settings (OS)</h3>
This option is useful for browser based tests too. Many Selenium grid vendors provide real browser on real OS systems and you can choose the desired combination of versions.<br />
<h3>
Resolution settings (RESOLUTION)</h3>
Same for the above options, many vendor solutions let you choose the desired screen resolution for automated browser based testing sessions.<br />
<h3>
Select tests by names expressions (KEYWORDS)</h3>
Pytest let you select the tests you are going to launch selecting a subset of tests that matches a pattern language based on test and module names.<br />
<br />
For example I find very useful to add the test management tool reference in test names, this way you will be able to launch exactly just that test:<br />
<blockquote class="tr_bq">
<i>c93466</i></blockquote>
Or for example all test names containing the login word but not c92411:<br />
<blockquote class="tr_bq">
<i>login and not c92411</i></blockquote>
Or if you organize your tests in different modules you can just specify the folder name and you'll select all the tests that live under that module:<br />
<blockquote class="tr_bq">
<i>api</i> </blockquote>
Under the hood the pytest command will be launched with <i>-k "EXPRESSION"</i>, for example<br />
<blockquote class="tr_bq">
<i>-k "c93466"</i></blockquote>
It is used in combination with markers, a sort of test tags. <br />
<h3>
Select tests to be executed by tag expressions (MARKERS)</h3>
Markers can be used alone or in conjunction with keyword expressions. They are a sort of tag expression that let you select just the minimum set of tests for your test run.<br />
<br />
Under the hood the pytest launcher uses the command line syntax <i>-m "EXPRESSION"</i>.<br />
<br />
For example you can see a marker expression that selects all tests marked with the <i>edit</i> tag excluding the ones marked with <i>CANBusProfileEdit</i>: <br />
<blockquote class="tr_bq">
<i>edit and not CANBusProfileEdit</i></blockquote>
Or execute only edit negative tests: <br />
<blockquote class="tr_bq">
<i>edit and negative</i></blockquote>
Or all integration tests <br />
<blockquote class="tr_bq">
<i>integration</i></blockquote>
It's up to you creating granular keywords for features and all you need for select your tests (e.g., functional, integration, fast, negative, ci, etc).<br />
<h3>
Test management tool integration (TESTRAIL_ENABLE)</h3>
All my tests are decorated with the test case identifier provided by the test management tool, in my company we are using <a href="http://www.gurock.com/testrail/">TestRail</a>.<br />
<br />
If this option is enabled the test results of executed tests will be reported in the test management tool.<br />
<br />
Implemented using the <i>pytest-testrail</i> plugin.<br />
<h3>
Enable debug mode (DEBUG)</h3>
The debug mode enables verbose logging.<br />
<br />
In addition for browser based tests open selenium grid sessions activating debug capabilities options (<a href="https://www.browserstack.com/automate/capabilities">https://www.browserstack.com/automate/capabilities</a>). For example verbose browser console logs, video recordings, screenshots for each step, etc. In my company we are using a local installation of <a href="https://github.com/zalando/zalenium">Zalenium</a> and <a href="https://www.browserstack.com/">BrowserStack</a> automate.<br />
<h3>
Block on first failure (BLOCK_FIRST_FAILURE)</h3>
This option is very useful for the following needs:<br />
<ul>
<li>a new build was deployed and you want to stop on the very first failure for a subset of sanity/smoke tests</li>
<li>you are launching repeated, long running, parallel tests and you want to block on first failure</li>
</ul>
The first usage let you gain confidence with a new build and you want to stop on the very first failure for analyzing what happened.<br />
<br />
The second usage is very helpful for:<br />
<ul>
<li>random problems (playing with number of repeated executions, random order and parallelism you can increase the probability of reproducing a random problem in less time)</li>
<li>memory leaks</li>
<li>testing system robustness, you can stimulate your system running some integration tests sequentially and then augment the parallelism level until your local computer is able to sustain the load. For example launching 24+ parallel integration tests on a simple laptop with pytest running on virtual machine is still fine. If you need something more heavy you can use distribuited <i>pytest-xdist</i> sessions or scale more with <a href="http://blazemeter.com/">BlazeMeter</a></li>
</ul>
As you can imagine you may combine this option with <i>COUNT</i>, <i>PARALLEL_SESSIONS</i>, <i>RANDOM_ENABLE</i> and <i>DEBUG</i> depending on your needs. You can test your tests robustness too.<br />
<br />
Under the hood implemented using the pytest's <i>-x</i> option.<br />
<h3>
Parallel test executions (PARALLEL_SESSIONS)</h3>
Under the hood implemented with pytest-xdist's command line option called <i>-n NUM</i> and let you execute your tests with the desired parallelism level.<br />
<br />
pytest-xdist is very powerful and provides more advanced options and network distributed executions. See <a href="https://github.com/pytest-dev/pytest-xdist">https://github.com/pytest-dev/pytest-xdist</a> for further options.<br />
<h3>
Switch from different selenium grid providers (SELENIUM_GRID_URL)</h3>
For browser based testing by default your tests will be launched on a remote grid URL. If you don't touch this option the default grid will be used (a local <a href="https://github.com/zalando/zalenium">Zalenium</a> or any other provider) but in case of need you can easily switch provider without having to change nothing in your testware.<br />
<br />
If you want you can save money maintaining and using a local Zalenium as default option; Zalenium can be configured as a selenium grid router that will dispatch capabilities that it is not able to satisfy. This way you will be able to save money and augment a little bit the parallelism level without having to change plan.<br />
<h3>
Repeat test execution for a given amount of times (COUNT)</h3>
Already discussed before, often used in conjunction with <i>BLOCK_FIRST_FAILURE</i> (pytest core <i>-x</i> option) <br />
<br />
If you are trying to diagnose an intermittent failure, it can be useful to run the same test or group of tests over and over again until you get a failure. You can use py.test's <i>-x</i> option in conjunction with pytest-repeat to force the test runner to stop at the first failure.<br />
<br />
Based on pytest-repeat's <i>--count=COUNT</i> command line option. <br />
<h3>
Enable random test ordering execution (RANDOM_ENABLE)</h3>
This option enables random test execution order.<br />
<br />
At the moment I'm using the <i>pytest-randomly</i> plugin but there are 3 or 4 similar alternatives I have to try out.<br />
<br />
By randomly ordering the tests, the risk of surprising inter-test dependencies is reduced.<br />
<h3>
Specify a random seed (RANDOM_SEED)</h3>
If you get a failure executing a random test, it should be possible to reproduce systematically rerunning the same tests order with the same test data.<br />
<br />
Always from the <i>pytest-randomly</i> readme:<br />
<blockquote class="tr_bq">
<i>By resetting the random seed to a repeatable number for each test, tests can
create data based on random numbers and yet remain repeatable, for example
factory boy’s fuzzy values. This is good for ensuring that tests specify the
data they need and that the tested system is not affected by any data that is
filled in randomly due to not being specified. </i></blockquote>
<h3>
Play option (PLAY)</h3>
This option will be discussed in a dedicated blog post I am going to write.<br />
<br />
Basically you are able to paste a JSON serialization of actions and assertions and the pytest runner will be able to execute your test procedure.<br />
<br />
You need just a computer with a browser for running any test (API, integration, system, UI, etc). You can paste how to reproduce a bug on a JIRA bug and everyone will be able to paste it on the Jenkins build with parameters form.<br />
<br />
See <a href="https://github.com/tierratelematics/pytest-play">pytest-play</a> for further information.<br />
<br />
If you are going to attending next Pycon in Florence don't miss the following <i>pytest-play</i> talk presented by <a href="https://www.pycon.it/conference/p/serena-martinetti">Serena Martinetti</a>:<br />
<ul>
<li><a href="https://www.pycon.it/conference/talks/integration-tests-ready-to-use-with-pytest-play">https://www.pycon.it/conference/talks/integration-tests-ready-to-use-with-pytest-play</a></li>
</ul>
<b><i>UPDATES</i></b>:<br />
<ul>
<li><b><i>[20180424]</i></b> pytest-play introduction <a href="http://davidemoro.blogspot.it/2018/04/hello-pytest-play.html">http://davidemoro.blogspot.it/2018/04/hello-pytest-play.html</a></li>
<li><b><i>[20180222]</i></b> pytest-play 2.0.0 released, no more compatible with 1.x format (using YAML instead of JSON and other improvements). See a detailed article showing how to use pytest-play for API REST testing using plain YAML files <a href="https://davidemoro.blogspot.com/2019/02/api-rest-testing-pytest-play-yaml-chuck-norris.html">https://davidemoro.blogspot.com/2019/02/api-rest-testing-pytest-play-yaml-chuck-norris.html</a> </li>
</ul>
<ul>
</ul>
<h2>
How to create a pytest project</h2>
If you are a little bit curious about how to install pytest or create
a pytest runner with Jenkins you can have a look at the following
scaffolding tool:<br />
<ul>
<li><a href="https://github.com/tierratelematics/cookiecutter-qa">cookiecutter-qa</a></li>
</ul>
It provides a hello world example that let you start with the test technique more suitable for you: plain selenium scripts, BDD or pytest-play JSON test procedures. If you want you can install page objects library. So you can create a QA project in minutes.<br />
<br />
Your QA project will be shipped with a <a href="https://github.com/tierratelematics/cookiecutter-qa/blob/master/%7B%7Bcookiecutter.project_slug%7D%7D/Jenkinsfile">Jenkinsfile</a> file that requires a <i>tox-py36</i> docker executor that provides a <i>python3.6</i> environment with <i>tox</i> already installed; unfortunately <i>tox-py36</i> is not yet public so you should implement it by your own at the moment.<br />
Once you provide a <i>tox-py36</i> docker executor the Jenkinsfile will create for you the build with parameters Jenkins form for you automatically on the very first Jenkins build for your project.<br />
<h2>
Conclusions</h2>
I hope you'll find some useful information in this article: nice to have features for test frameworks or platform, a little bit of curiosity for the Python world or new pytest plugin you never heard about.<br />
<br />
Feedback and contributions are always welcome.<br />
<br />
Tweets about test automation and new articles happens here:<br />
<ul>
<li><a href="https://twitter.com/davidemoro">@davidemoro</a></li>
</ul>
<ul>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com15tag:blogger.com,1999:blog-4821336113846598354.post-20633904714914778632018-03-06T00:17:00.001+01:002018-03-06T00:18:04.250+01:00Combinatorial testing: ACTSLast year I discovered the <b>ACTS</b> tool thanks to <a href="https://rbcs-us.com/about/">Rex Black</a>'s "Pairwise Testing with ACTS" webinar:<br />
<ul>
<li> <a href="https://rbcs-us.com/blog/pairwise-testing-with-acts-available-now/">https://rbcs-us.com/blog/pairwise-testing-with-acts-available-now/</a></li>
</ul>
and I find it very helpful when you have to deal with non trivial systems, you fear to miss something of important or there is a combinatorial explosion.<br />
<br />
ACTS let you create a model for your system defining parameters, boundary values, different types of inputs, valid and invalid data, relations between parameters and different generation algorithms, base choices (e.g., more important values you want to be used most often).<br />
<br />
More information from the ACTS used guide:<br />
<br />
<blockquote class="tr_bq">
<i>ACTS is a test generation tool for constructing t-way combinatorial test sets.<br />Currently, it supports t-way test set generation with t ranging from 1 to 6.<br />Combinatorial testing has been shown very effective in detecting faults that are caused by unexpected interactions between different contributing factors.</i></blockquote>
Always from the ACTS user guide:<br />
<blockquote class="tr_bq">
<i>A system is specified by a set of parameters and their values. A test set is a t-way test set if it satisfies the following property: Given any t parameters (out of all the parameters) of a system, every combination of values of these t parameters is covered in at least one test in the test set.<br />Currently, ACTS supports t-way test set generation for 1 = t = 6. Empirical studies show that t being up to 6 is sufficient for most practical applications. A special form of 1-way testing, called base-choice testing, is implemented in ACTS. Base-choice testing requires that every parameter value be covered at least once and in a test in which all the other values are base choices. Each parameter has one or more values designated as base choices. Informally, base choices are "more important" values, e.g., default values, or values that are used most often in operation.</i></blockquote>
So ACTS helps you to drastically reduce the testing combinations for moderate or very large systems. The most important ACTS output is the test set generation (e.g., a calc/Excel export file). The coverage function from the statistics tab is interesting too.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8xQz5teq0ajzaVr6vfUPXF43yNePVC_qxqyow12AbuZh0FZ_jQw6wCSmxgnKqfEP3bWkmRDFTNiD-mwC9Sy58JTG7Ze_2_4ElFMoCSO77YOZzDVhgUMpb7OIDdWNu_OB16R-GJpUY/s1600/acts_statistics.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="850" data-original-width="1600" height="339" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8xQz5teq0ajzaVr6vfUPXF43yNePVC_qxqyow12AbuZh0FZ_jQw6wCSmxgnKqfEP3bWkmRDFTNiD-mwC9Sy58JTG7Ze_2_4ElFMoCSO77YOZzDVhgUMpb7OIDdWNu_OB16R-GJpUY/s640/acts_statistics.PNG" width="640" /></a><br />
<br />
Since we have finite time and infinite test combinations, we have to be very<br />
selective and ACTS is a valid help.<br />
<br />
I find that the ACTS test cases export sometimes is not readable and you loose the base choice info, invalid data and you might be lost in combinations. I think it is a good idea adding some simple formulas and conditional formatting for improving the readability of the export, see base choices, distinguish between positive and negative scenarios (e.g., yellow), emphasize when you get the same output from different paths, visualize different shapes of test cases and convey more meaningful information about your data depending on your domain.<br />
<br />
This way you will be able to detect more easily what are the most important cases and prioritize them (e.g., feature confirmation and guessing the test cases with a higher probability of failure) for both positive and negative cases to be as effective as possible even if you have very little time for testing.<br />
<br />
Obviously before creating the ACTS project you should already have an idea of what are the most important scenarios and all the possible ways for obtaining a certain result but thanks to ACTS you can have a visual double check and see if you missed something of important or other edge cases.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFbhvBINnDMZV94WqCCe-5KUmxYR3ygKYFDFbPfBSCU7CeA-4H0vebVGvyroXBFmlKnY6Lbpa_4aGYiCZOCUtWwt9n9dCn4Mg3xeuwjdbRidM7PgPnnoTChlcnqJlSLbFjDGKSmPA7/s1600/test_set_visualization.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="822" data-original-width="1600" height="328" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFbhvBINnDMZV94WqCCe-5KUmxYR3ygKYFDFbPfBSCU7CeA-4H0vebVGvyroXBFmlKnY6Lbpa_4aGYiCZOCUtWwt9n9dCn4Mg3xeuwjdbRidM7PgPnnoTChlcnqJlSLbFjDGKSmPA7/s640/test_set_visualization.PNG" width="640" /></a></div>
<br />
<br />
When you identify the most interesting rows you can add a description note explaining why this particular scenario is important, prioritize all your test cases and add all the needed information and adding new scenarios or edge cases. The file is ready to be imported to a test management tool now and you have also an ordered execution plan based on priority and risk.<br />
<br />
And if you don't have enough time for testing the exported combinations you might evaluate if you have sufficient information to realize if the most relevant cases are covered, the level of confidence, coverage and you can evaluate if the residual risk is acceptable. <br />
<br />
This test set could be also used for automating your scenarios. If you use for example a keyword driven testing pattern or more abstract patterns you can implement just one time the needed logics and with little time you will cover all your hundreds of combinations because your test framework let you decouple the test data from the test implementation. In following articles we'll discuss about <a href="https://docs.pytest.org/en/latest/">pytest</a> fixtures/parametrization, <a href="http://pytest-bdd.readthedocs.io/en/latest/">pytest-bdd</a> and other test frameworks. Stay tuned!<br />
<br />
Links:<br />
<ul>
<li><a href="https://www.nist.gov/programs-projects/automated-combinatorial-testing-software-acts">https://www.nist.gov/programs-projects/automated-combinatorial-testing-software-acts</a> </li>
<li><a href="https://rbcs-us.com/blog/pairwise-testing-with-acts-available-now/">https://rbcs-us.com/blog/pairwise-testing-with-acts-available-now/</a></li>
<li><a href="https://www.istqb.org/downloads/category/48-advanced-level-test-automation-engineer-documents.html">https://www.istqb.org/downloads/category/48-advanced-level-test-automation-engineer-documents.html</a></li>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-56018737043674238472015-08-24T00:03:00.001+02:002015-08-24T00:03:10.981+02:00Introducing substancek. A Kotti projectLet me introduce <i>substancek</i>, a Kotti (<a href="http://kotti.pylonsproject.org/">http://kotti.pylonsproject.org</a>) project.<br />
<h2>
What it is substancek?</h2>
substancek is:<br />
<ul>
<li><b>Kotti</b> (web application framework) + <b>decoupled admin interface</b> </li>
</ul>
It is only an additional <b>layer</b> upon the following opinionated stack:<br />
<ul>
<li><b><i>kotti_backend</i></b>, a plugin that turns Kotti to a private admin interface (backend). See <a href="https://github.com/Kotti/kotti_backend">https://github.com/Kotti/kotti_backend</a></li>
<li><b><i>Kotti</i></b>, web development framework based on Pylons/Pyramid with workflows support, content types, security). See <a href="http://kotti.pylonsproject.org/">http://kotti.pylonsproject.org</a></li>
<li><b><i>Pylons/Pyramid</i></b>, one of the most flexible Python web frameworks. See <a href="http://www.pylonsproject.org/">http://www.pylonsproject.org</a></li>
<li><b><i>SQLAlchemy</i></b>, the Python SQL Toolkit and Object Relational Mapper. See <a href="http://www.sqlalchemy.org/">http://www.sqlalchemy.org</a></li>
<li><b><i>Python</i></b>. See <a href="https://www.python.org/">https://www.python.org</a></li>
</ul>
with the following <b>motto</b>:<br />
<blockquote class="tr_bq">
<i>"""(even) better development experience and complete frontend freedom"""</i></blockquote>
and introduces (or better promotes) the concept of <b>private admin area</b> (backend) decoupled from the public side (frontend) of your web applications built with Kotti.<br />
<br />
In other words it is a <b>set of technologies</b> addressed under the substancek brand that let you extend Kotti in order to use it just as a private backend administration area for your application data.<br />
<br />
So you are still using plain Kotti with an additional package (at least <i>kotti_backend</i> depending on what you need).<br />
<br />
If you want to know more I've discussed here benefits and why frontend decoupled from the backend pattern. See <b><a href="http://davidemoro.blogspot.it/2015/07/kotti-cms-successful-story-part-1.html">http://davidemoro.blogspot.it/2015/07/kotti-cms-successful-story-part-1.html</a></b> <br />
<ul>
</ul>
<ul>
</ul>
<h3>
substancek name</h3>
Tribute to:<br />
<ul>
<li><b>substance</b>d (<a href="http://www.substanced.net/">http://www.substanced.net</a>). Carefully designed, well documented Python web development framework based on Pyramid/ZODB. It shares the concept of management views decoupled from the retail views for each published resource</li>
<li><b>k</b>otti (<a href="http://kotti.pylonsproject.org/">http://kotti.pylonsproject.org</a>). The Kotti framework</li>
</ul>
<h3>
When substancek is for you</h3>
Any project of <b>any size</b> (from micro to XXL) involving <b>content management</b> that needs:<br />
<ul>
<li><b>RDBMS</b>. Kotti is an opinionated framework opposed to plain Pyramid. The persistence layer is managed by SQLAlchemy (<a href="http://www.sqlalchemy.org/">http://www.sqlalchemy.org</a>)</li>
<li><b>user friendly</b> and production ready <b>admin/editing interface</b> for content producers so they can immediately start adding <b>content</b> to your web application with copy/cut & paste, clean urls, etc</li>
<li><b>rapid development</b>. Pyramid and Kotti are easy to learn and let you become productive quickly and speed up your development. Even more with the decoupled admin interface.</li>
<li><b>stability</b>. Pyramid and Kotti exist since many years and they are solid rock solutions production ready</li>
<li><b>frontend freedom</b>. The admin interface comes for free while it is completely up to you add what your application needs (retail views). Since they are completely two different applications there are no css/js conflicts and you can integrate your preferred frontend tool chain without get influenced by how the admin interface is built.<a href="http://davidemoro.blogspot.it/2015/07/kotti-cms-successful-story-part-1.html"></a></li>
<li><b>flexibility</b>. Pyramid won't hold you back when your application is small and won't get in your way when your application becomes large.<br />Even if you start small, using Pyramid you anticipate what customers will need later avoiding expensive rewrites and this can <i>make the difference</i>. In other words this also means <i>successful projects</i> in the long term. Same for Kotti. See <a href="http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/introduction.html#what-makes-pyramid-unique">What makes Pyramid unique</a>.</li>
</ul>
So if you project needs (or in <b>future</b> iterations) one or more:<br />
<ul>
<li>complex <b>security</b> policies</li>
<li><b>workflows</b> </li>
<li><b>hierarchical data</b> support</li>
<li>or even <b>intranet/extranet</b> like collaboration areas</li>
</ul>
you might consider even more <b>substancek</b> (kotti_backend + Kotti + Pyramid + SQLAlchemy)<i></i>.<br />
<br />
For example: <br />
<ul>
<li>very small applications. For example a just one view public json
endpoint for published news-like resources consumed by a third party app
with an admin interface for editing contents</li>
<li>heavy Javascript based applications with modern frontend tools (eg: SPA + REST) with a decoupled admin interface</li>
<li>content management solutions</li>
<ul>
<li>blog </li>
</ul>
<ul>
<li>ecommerce</li>
</ul>
<ul>
<li>intranets</li>
</ul>
<ul>
<li>large CMS-ish applications</li>
</ul>
</ul>
<i>Note well</i>: if you don't need workflows don't be scared because there is no overkill. You
can use a one state workflow or no workflow at all for example. No
hierarchical data? Use not nestable resources and so on. If it comes
out later that you need them it will be quite easy converting your code.<br />
<h4>
Alternatives</h4>
You can use <b>plain Kotti</b>, without the substancek's <i>kotti_backend</i> setup. Or if you prefer <i>noSQL</i> try the excellent <b>substanced</b> (substanced + Pyramid + ZODB). Both solutions are lightweight, tested, well documented and easy to learn. Alternatively if you need really a minimal and unopinionated solution you might use <b>plain Pyramid</b>.<br />
<br />
Do you need something more? You might consider to use <b>Plone</b> (<a href="https://plone.org/">https://plone.org</a>) as a framework.<br />
<br />
Anyway the good news is that <b>Python</b> is plenty of good options.<br />
<h3>
substancek architecture details</h3>
As already told you the private admin area (backend) and the rest of the application (frontend) are two complete different applications with different settings, different views and shared authentication.<br />
<br />
Assuming you are going to use <i>PasteDeploy</i> to run your application, let's consider the following configuration files setup:<br />
<ul>
<li><i>backend-dev.ini</i>, configures the private admin interface based on Kotti thanks to the <i>kotti_backend</i> plugin</li>
<li><i>frontend-dev.ini</i>, configures your application you are developing (a full CMS frontend implementation or a microapp with just one retail view). </li>
<li><i>development.ini</i> (<b>optional</b>), mount the <i>backend-dev.ini</i> and <i>frontend-dev.ini</i> applications in the same process (<i>/admin</i> for the private admin interface and <i>/</i> for your application). Alternatively you can run the frontend and backend using two processes waiting for requests on different ports and play with rewrite rules.</li>
</ul>
<h4>
backend-dev.ini</h4>
<blockquote class="tr_bq">
<pre>[app:kotti]
use = egg:kotti
...
pyramid.includes =
pyramid_debugtoolbar
pyramid_tm
<b>kotti_backend.views.override_root_view</b>
kotti.configurators = kotti_tinymce.kotti_configure
<b>kotti_backend.kotti_configure</b>
<b>kotti.use_workflow = kotti_backend:workflows/simple_backend.zcml</b>
<b>kotti_backend.goto_frontend = 1</b> </pre>
</blockquote>
This is a normal Kotti setup with:<br />
<ul>
<li>enabled the (optional) root view override for our admin interface. In other words the default root view will be <i>@@contents</i> instead of the standard Kotti's hello page (see <i>pyramid.includes</i>)</li>
<li>added <i>kotti_backend</i> to <i>kotti.configurators</i></li>
<li>override the Kotti's default workflow with the one provided by kotti_backend (<i>kotti.use_workflow</i>). Playing with the additional <i>pview</i> permission you can decide your resources visibility on the frontend. See the workflow definition here <a href="https://github.com/Kotti/kotti_backend/blob/master/kotti_backend/workflows/simple_backend.zcml">https://github.com/Kotti/kotti_backend/blob/master/kotti_backend/workflows/simple_backend.zcml</a></li>
<li>enable a "Goto frontend" link (<i>kotti_backend.goto_frontend</i>) for easy switching from admin interface and frontend </li>
</ul>
See more options on the <i>kotti_backend</i>'s <i>README</i> file:<br />
<ul>
<li><a href="https://github.com/Kotti/kotti_backend">https://github.com/Kotti/kotti_backend</a></li>
</ul>
<h4>
frontend-dev.ini</h4>
<blockquote class="tr_bq">
<pre>[app:main]
use = egg:Kotti
...
<b>kotti.use_workflow = kotti_backend:workflows/simple_backend.zcml</b>
<b>kotti.configurators =
your_package.kotti_configure
kotti.base_includes =
kotti
kotti.views
</b></pre>
</blockquote>
On the frontend configuration file we share the same workflow in use on the admin interface (<i>kotti.use_workflow</i>).<br />
<br />
One of the most important configuration is the <i>kotti.base_includes</i> override: here we decide what will be loaded on our application. We omit all the Kotti views loaded by default in the standard setup and we load what we want to include where:<br />
<ul>
<li><i>kotti</i>, loads the kotti "core"</li>
<li><i>kotti.views</i> (optional), load some view discriminators and utils defined by Kotti if you need them</li>
<li><i>your_plugin.your_includes</i>, load your includes registering the views needed by your application</li>
</ul>
The <i>kotti.configurators</i> typically auto includes your package and tell what should be included in your application (<i>pyramid.includes</i>). See the Kotti documentation for more info. <br />
<br />
In other words:<br />
<blockquote class="tr_bq">
<i>"what is not loaded, it doesn't exist"</i></blockquote>
so the final result is that there is <b>nothing</b> exposed on the frontend except what you decide to load and extreme control. You can register just only one view application or configure a complex setup for a CMS-like application: it's up to you registering only the views your application needs and no more. This way you can use completely different frontend frameworks, different versions of Javascript libraries, you have no css/js conflicts and no need to hide unneeded things and you decide which resources will be published on the frontend.<br />
<br />
See also another advance usage pattern "<i>Using Kotti as a library</i>" <a href="http://kotti.readthedocs.org/en/latest/developing/advanced/as-a-library.html">http://kotti.readthedocs.org/en/latest/developing/advanced/as-a-library.html</a><br />
<h4>
development.ini</h4>
<blockquote class="tr_bq">
<pre># See http://pythonpaste.org/deploy/#paste-composite-factory
<b>[composite:main]
use = egg:Paste#urlmap
/ = config:frontend-dev.ini
/admin = config:backend-dev.ini</b>
[server:main]
use = egg:waitress#main
host = 127.0.0.1
port = 5000</pre>
</blockquote>
The (optional) <i>development.ini</i> shows how to configure a composite application with different mount points. You can change <i>/admin</i> with <i>/cms</i> or <i>/whateveryouwant</i> depending on your needs.<br />
<h4>
Examples </h4>
You can checkout the <b><a href="https://github.com/substancek/substancek_cms_theme">https://github.com/substancek/substancek_cms_theme</a></b> package if you want to see in action a (quite complex) example.<br />
<br />
I'm going to provide more and simpler examples (eg: a pretend micro application), see the roadmap.<br />
<h3>
What are the substancek related packages</h3>
Here you can see the whole substancek ecosystem: <br />
<ul>
<li><a href="https://github.com/Kotti/kotti_backend">kotti_backend</a>, generic package that turns Kotti to a private admin area. This is the <b>keystone</b> for every substancek like project </li>
<li>CMS-like applications</li>
<ul>
<li>substancek_cms (roadmap), a Kotti CMS distribution with an alternative frontend theme based on SASS, html/templates minification and assets optimization based on Yeoman tools. It will be based on the <i>kotti_project</i> experiment available here <a href="https://github.com/davidemoro/kotti_project">https://github.com/davidemoro/kotti_project</a></li>
<li><a href="https://github.com/substancek/substancek_cms_theme">substancek_cms_theme</a>, an example public side CMS implementation based on Kotti built with SASS and html/templates minification</li>
<li><a href="https://github.com/Kotti/kotti_actions">kotti_actions</a>, link actions backend implementation for header, footer and main navigation links</li>
<li><a href="https://github.com/Kotti/kotti_boxes">kotti_boxes</a>, portlets and box backend implementation</li>
<li><a href="https://github.com/Kotti/kotti_es">kotti_es</a>, elastic search support (to be refactored)</li>
</ul>
<li>common utilities</li>
<ul>
<li><a href="https://github.com/davidemoro/pyramid_html_minifier">pyramid_html_minifier</a>, generic Pyramid package that introduces Chameleon templates minification with no overhead. Required by <i>substancek_cms_theme</i>.</li>
<li><a href="https://github.com/davidemoro/build_commands">build_commands</a>, generic Python package that defines a set of initialization commands on install (eg: <i>python setup.py npm/bower/gulp</i>) for frontend stuff. Required by <i>substancek_cms_theme</i>.</li>
</ul>
</ul>
<h3>
Who's using substancek technologies</h3>
<h4>
MIP - graduate school of business</h4>
The <b>MIP</b> (Politecnico di Milano graduate school of business - <b><a href="http://www.mip.polimi.it/en">www.mip.polimi.it/en</a></b>) uses <i>substancek</i> technology inside for the private admin interface. This approach is so flexible that let you use Kotti as a private admin content management area and even implement your public views using other frameworks or non Python languages (for example PHP+Symfony2).<br />
<br />
See:<br />
<ul>
<li><a href="http://davidemoro.blogspot.com/2015/07/kotti-cms-successful-story-part-1.html">http://davidemoro.blogspot.com/2015/07/kotti-cms-successful-story-part-1.html</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/07/kotti-cms-frontend-decoupled-backend-part-2.html">http://davidemoro.blogspot.com/2015/07/kotti-cms-frontend-decoupled-backend-part-2.html</a></li>
</ul>
<h4>
substancek_cms_theme</h4>
This is a work in progress opinionated CMS frontend implementation that reuses existing Kotti templates and logics.<br />
<br />
The look and feel is the same you get with a standard Kotti installation but it shows how to integrate and distribute a Python package integrated with a Yeoman setup (<a href="http://yeoman.io/">http://yeoman.io</a>) that provides:<br />
<ul>
<li>production vs development setup. In production mode will be used assets and templates from the dist </li>
<li>Chameleon templates minification with no overhead thanks to <a href="https://github.com/davidemoro/pyramid_html_minifier">pyramid_html_minifier</a> when you are in "production" mode </li>
<li>SASS</li>
<li>image/assets optimization</li>
<li>build commands for npm/bower/gulp initialization thanks to the generic package <a href="https://github.com/davidemoro/build_commands">build_commands</a></li>
</ul>
Once installed you'll see the admin interface visiting <i>http://localhost:5000/cms</i>. <br />
See the code here: <a href="https://github.com/substancek/substancek_cms_theme">https://github.com/substancek/substancek_cms_theme</a><br />
<h3>
Next steps</h3>
If you want to contribute there is a lot to do:<br />
<ul>
<li>contributions to the Kotti core (for example <i>@@contents</i> pagination, REST, etc and new ideas)</li>
<li>creation of the <i>substancek_cms</i> package that puts together the <i>substancek_cms_theme</i> default theme plus the most useful third party Kotti plugins like news, events, etc with decoupled admin interface and Vagrant/Ansible automated setup based on <i>kotti_project</i> for easy installation/evaluation</li>
<li>creation of simple pretend packages, add more examples (microapp, blog)</li>
<li>a toolbar available on the frontend just of editor users for improved usability (the same shown in the MIP case history) implemented with a Pyramid tween</li>
<li>more work on <i>substancek_cms_theme</i>, implement advanced features shown in the MIP case history like link managers and portlets and refactor the Yeoman folder removing what is not strictly needed</li>
<li>scaffolding, help people create new projects with decoupled admin interface using Kotti as a framework</li>
<li><i>substancek</i> dedicated github page </li>
</ul>
Contributions, feedback or pings like "hey, I'm going to use Pyramid/Kotti for the my next project" will be very appreciated!<br />
<h2>
Documentation</h2>
<ul>
<li><a href="http://kotti.readthedocs.org/en/latest">http://kotti.readthedocs.org/en/latest</a></li>
<li><a href="http://pylons.readthedocs.org/en/latest">http://pylons.readthedocs.org/en/latest</a></li>
</ul>
<h2>
All Kotti posts published by <a href="https://twitter.com/davidemoro">@davidemoro</a><br />
</h2>
<ul>
<li><a href="http://davidemoro.blogspot.it/search/label/kotti">http://davidemoro.blogspot.it/search/label/kotti </a></li>
</ul>
<h2>
Twitter links</h2>
<ul>
<li><a href="https://twitter.com/pylons">https://twitter.com/pylons</a></li>
<li><a href="https://twitter.com/KottiCMS">https://twitter.com/KottiCMS</a></li>
<li><a href="https://twitter.com/davidemoro">https://twitter.com/davidemoro</a></li>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com3tag:blogger.com,1999:blog-4821336113846598354.post-43505088587358002015-07-18T23:50:00.001+02:002015-08-24T00:05:39.930+02:00Kotti CMS - frontend decoupled from backend. How we did it (part 2)<br />
In the previous article <b><a href="http://davidemoro.blogspot.it/2015/07/kotti-cms-successful-story-part-1.html">http://davidemoro.blogspot.it/2015/07/kotti-cms-successful-story-part-1.html</a></b> we have seen that:<br />
<ul>
<li>decoupled public website from the private content management is cool</li>
<li>Python and Pyramid is cool</li>
<li>don't use PHP please :)</li>
</ul>
No we will see:<br />
<ul>
<li>how to build a <b>pure Python</b> Kotti based setup with a private content management area decoupled from the public website (with tips, links, technical details and screenshots)</li>
</ul>
Here you can see some screenshots, implementation details and links. <br />
<h3>
Project setup</h3>
The installation folder is a package that contains all the
application-specific settings, database configuration, which packages
your project will need and where they lives.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4DFp-QdWA1bwWSDjIJyiSai9ey1jsDZ9SyGw6eY8g_5JsrthPYQR89ypxyWvtSR6UhizcyTW1f8HQwSYQvHGsccmbWn4ryeHtJvVLoazXd0lihOtdNz0jhsAxPYGTMJYDoNzp3tWY/s1600/xkcd-buildout.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="173" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4DFp-QdWA1bwWSDjIJyiSai9ey1jsDZ9SyGw6eY8g_5JsrthPYQR89ypxyWvtSR6UhizcyTW1f8HQwSYQvHGsccmbWn4ryeHtJvVLoazXd0lihOtdNz0jhsAxPYGTMJYDoNzp3tWY/s200/xkcd-buildout.png" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">From https://pypi.python.org/pypi/mr.developer.</td></tr>
</tbody></table>
The installation folder is a "<b>one command install</b>" meta package:<br />
<ul>
<li>replicable</li>
<li>under version control + git flow</li>
<li>documented + change list for each plugin and the project itself</li>
<li>with a tag based deploy</li>
<li>based on Python pip/requirements.txt</li>
<li>batteries included</li>
<li>exceptions logging on file (with logrotate) enabled by default</li>
<li>automated deploy </li>
</ul>
so let the computer works for us and have fun. <br />
See: <br />
<ul>
<li><a href="http://davidemoro.blogspot.com/2015/04/pip-for-buildout-folks.html">http://davidemoro.blogspot.com/2015/04/pip-for-buildout-folks.html</a></li>
<li><a href="https://github.com/davidemoro/kotti_project">https://github.com/davidemoro/kotti_project</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/03/pyramid-exceptions-logging.html">http://davidemoro.blogspot.com/2015/03/pyramid-exceptions-logging.html </a></li>
</ul>
<h3>
Populators</h3>
Populators are functions with no arguments that get called on system startup, they may then make automatic changes to the database like <b>content initialization</b>. <br />
<br />
Populators are very important because when you
install the project folder during development or on the very first
production instance you'll find all the most important contents and
sections by default. Things will be created automatically if the
database is empty, so you don't obtain a blank site on the very first
install.<br />
<br />
Populators are also good for improving the
first impact of the end users (I mean editors) with the platform because
they see all the main sections already there.<br />
<br />
See:<br />
<ul>
<li><a href="http://kotti.readthedocs.org/en/latest/developing/basic/configuration.html#kotti-populators">http://kotti.readthedocs.org/en/latest/developing/basic/configuration.html#kotti-populators</a></li>
</ul>
<h3>
Private backend area</h3>
Turning Kotti CMS into a private content administration is quite easy:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaeLdCBbPDJ778zhVbp9mWbdBu_aBGVBN-5GYtbbJpO_uD0Z3Bp2PUoKVHnabhpcQ2547K9lF25jqfA1L1BvO9iwO28C9EPbMvJvkTvDCfcVX4BpI3lPPVZvSlGtPpdWbVj6h_-IyN/s1600/mip-backend-login.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="182" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaeLdCBbPDJ778zhVbp9mWbdBu_aBGVBN-5GYtbbJpO_uD0Z3Bp2PUoKVHnabhpcQ2547K9lF25jqfA1L1BvO9iwO28C9EPbMvJvkTvDCfcVX4BpI3lPPVZvSlGtPpdWbVj6h_-IyN/s200/mip-backend-login.png" width="200" /></a></div>
<ul>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-intranet.html">http://davidemoro.blogspot.com/2015/02/kotti-cms-intranet.html</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-workflow-reference.html">http://davidemoro.blogspot.com/2015/02/kotti-cms-workflow-reference.html</a></li>
<li><a href="http://kotti.readthedocs.org/en/latest/developing/basic/security.html">http://kotti.readthedocs.org/en/latest/developing/basic/security.html</a></li>
</ul>
Later I've created a generic package that does all the things for you (<i>kotti_backend</i>):<br />
<ul>
<li><a href="https://github.com/Kotti/kotti_backend">https://github.com/Kotti/kotti_backend</a></li>
</ul>
so things are even easier now (install <i>kotti_backend</i>, done). <br />
<h3>
Multilingual</h3>
<i>kotti_multilingual</i> is your friend. <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5kUGdkkgcah1Xbfo1d4mtw1JsUc4PTs-9__M1k8mEgn5L3knjJajyLj1YuuAw0gh126jiJ_65rh_KYTmIdTvNbpFouEvw4pl0PZc8OpXK_jNt-TAJcIukzoUn8zWq_BHGlC1ivKe4/s1600/mip-backend.png" style="margin-left: auto; margin-right: auto;"><img border="0" height="168" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5kUGdkkgcah1Xbfo1d4mtw1JsUc4PTs-9__M1k8mEgn5L3knjJajyLj1YuuAw0gh126jiJ_65rh_KYTmIdTvNbpFouEvw4pl0PZc8OpXK_jNt-TAJcIukzoUn8zWq_BHGlC1ivKe4/s400/mip-backend.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Goto frontend link, translation management and link to the technical documentation online based on Sphinx</td></tr>
</tbody></table>
See:<br />
<ul>
<li><a href="https://github.com/Kotti/kotti_multilingual">https://github.com/Kotti/kotti_multilingual</a> </li>
<li><a href="http://davidemoro.blogspot.com/2015/04/kotti-multilingual.html">http://davidemoro.blogspot.com/2015/04/kotti-multilingual.html</a></li>
</ul>
<h3>
Elastic search</h3>
<i>kotti_es</i> provides <i>ElasticSearch</i>
integration for fulltext search. This plugin needs more love and and a complete refactor (it was built in a hurry and I'm not yet satisfied) but it proved there are no known issue after
months of intensive usage.<br />
Probably things will change, hope other guys with the same needs will contribute.<br />
<br />
See:<br />
<ul>
<li><a href="https://github.com/Kotti/kotti_es">https://github.com/Kotti/kotti_es</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/05/kotti-cms-elasticsearch-integration.html">http://davidemoro.blogspot.com/2015/05/kotti-cms-elasticsearch-integration.html</a></li>
</ul>
<h3>
Main navigation and header/footer links</h3>
You can use the <i>kotti_actions</i> plugin if you want to implement footer, header <b>links</b> or even nested main <b>navigation menus</b>. Obviously <i>kotti_actions</i> is ment to be used with a decoupled frontend.<br />
<br />
As you can see a custom colour can be assigned to courses, navigation links, sections and every kind of object thanks to the <b>json annotations</b> column provided by default by Kotti. So you can add arbitrary fields.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi68yJP_yid3V9sQlh_kzcR71l8T4McwwBpADfFaOYCtxow3AQ2nYlIT6K6pBH0XWV7UZENP7N9026PfpYB4m_rhrJkMNf9BKeh1PMWwZEVvrdYFKo3h44T8i-PMH3Vp6v1j31lFs2r/s1600/main-nav.png" style="margin-left: auto; margin-right: auto;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi68yJP_yid3V9sQlh_kzcR71l8T4McwwBpADfFaOYCtxow3AQ2nYlIT6K6pBH0XWV7UZENP7N9026PfpYB4m_rhrJkMNf9BKeh1PMWwZEVvrdYFKo3h44T8i-PMH3Vp6v1j31lFs2r/s400/main-nav.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">How the multilevel menu looks like on the public website</td></tr>
</tbody></table>
See:<br />
<ul>
<li><a href="https://github.com/Kotti/kotti_actions">https://github.com/Kotti/kotti_actions</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-annotations-store-arbitrary-data.html">http://davidemoro.blogspot.com/2015/02/kotti-cms-annotations-store-arbitrary-data.html </a></li>
</ul>
<h3>
Portlets </h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQ2xAzkzIx8IpD9ATR6CrDHZnP7auwdzW1to-fuISmvpOuSFpYNUlYNmDTcqDoBK2OS5i_NjLAgwaIn7xr2zloXD19AQ0ciYKIOAtZENKzzxBlboaiZy-yUQfZRrfG-qx0WNkZ7P-v/s1600/course-layout.png" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQ2xAzkzIx8IpD9ATR6CrDHZnP7auwdzW1to-fuISmvpOuSFpYNUlYNmDTcqDoBK2OS5i_NjLAgwaIn7xr2zloXD19AQ0ciYKIOAtZENKzzxBlboaiZy-yUQfZRrfG-qx0WNkZ7P-v/s320/course-layout.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The main layout based on box managers for portlets</td></tr>
</tbody></table>
The <i>kotti_boxes</i>
is your friend. This plugin is ment to be used with a decoupled
frontend. And it was quite quick implementing portlets because we didn't
need to customize the private backend area. <br />
<br />
You can define different page layouts for each resource type (home page, news, courses, etc) and show <i>boxes</i> in well defined areas (<i>box managers</i>), for example <i>LeftBoxManager</i>, <i>AboveFooterBoxManager</i> and so on. <br />
<br />
So
box and box managers are just non publishable contents and you can:<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgT1eV9oUGplh92bWvT3OmK1lKOO6SEz70Htm9OJXUHULdJikK72DmLClcXJ2cxsS6Y5WV6d_HqBRgesLY0inaMQXYV0zWfqhMeZXT9Te8qENxWP0MyPHxUJCxem0ezcfU_10UqRIeW/s1600/mip-portlets.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgT1eV9oUGplh92bWvT3OmK1lKOO6SEz70Htm9OJXUHULdJikK72DmLClcXJ2cxsS6Y5WV6d_HqBRgesLY0inaMQXYV0zWfqhMeZXT9Te8qENxWP0MyPHxUJCxem0ezcfU_10UqRIeW/s200/mip-portlets.png" width="135" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Banner portlets with links</td></tr>
</tbody></table>
<ul>
<li>copy/paste them</li>
<li>assign workflow with different security policies to box
and box managers</li>
<li>assign different views</li>
<li>share edit permission to certain box or box managers to particular users or groups </li>
<li>prevent certain type of boxes
to be included in some areas (for example: banner box image only
addable to the left box manager).</li>
</ul>
See:<br />
<ul>
<li><a href="https://github.com/Kotti/kotti_boxes">https://github.com/Kotti/kotti_boxes</a></li>
</ul>
<h3>
Editor toolbar</h3>
As you can see if you are logged in the frontend will show an editor toolbar with:<br />
<ul>
<li>link to the backend version of the page</li>
<li>object information (state, type of object, etc)</li>
</ul>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUIAX69212y7BzsU2NMCb2T3IYRN-vMwCKFKcAkw_SYTlXxx45Pkr7Gfk9eOX3jO2klkJl_nJ_ASkvwKUwz0GxHlW8ycf5DIPVV84SQdwfgjqh6lCgtMAwSYcr5yG1hAy8RPj0qQQB/s1600/mip-frontent-actions-1.png" style="margin-left: auto; margin-right: auto;"><img border="0" height="191" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUIAX69212y7BzsU2NMCb2T3IYRN-vMwCKFKcAkw_SYTlXxx45Pkr7Gfk9eOX3jO2klkJl_nJ_ASkvwKUwz0GxHlW8ycf5DIPVV84SQdwfgjqh6lCgtMAwSYcr5yG1hAy8RPj0qQQB/s400/mip-frontent-actions-1.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Info and links to the backend, edit and folder contents</td></tr>
</tbody></table>
<br />
or see exactly the website as an anonymous user (very common customer request):<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDCYDf2nXpdAoxmyeVLBv6U4ID1zWdy21U0w2-ZQSQX-m8tpOy_3oE438u-VkRdAcKLv98Yj4BilizRK0V-fLdZakURh8AzyQY_jsut8ajyRC8j2BRsL5Aokxli_5INwzduKcANtoj/s1600/mip-frontent-actions-2.png" style="margin-left: auto; margin-right: auto;"><img border="0" height="188" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDCYDf2nXpdAoxmyeVLBv6U4ID1zWdy21U0w2-ZQSQX-m8tpOy_3oE438u-VkRdAcKLv98Yj4BilizRK0V-fLdZakURh8AzyQY_jsut8ajyRC8j2BRsL5Aokxli_5INwzduKcANtoj/s400/mip-frontent-actions-2.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Anonymous view</td></tr>
</tbody></table>
You can also add more features, for example direct edit links for images or portlets or live edit features.<br />
<br />
Talking about a pure Python solution, you might implement this feature with a Pyramid Tween (I hope I'll have enough spare time to do that. Anyone would want to contribute? We are very welcome, contact me!):<br />
<ul>
<li><a href="http://docs.pylonsproject.org/projects/pyramid//en/latest/glossary.html#term-tween">http://docs.pylonsproject.org/projects/pyramid//en/latest/glossary.html#term-tween</a></li>
<li><a href="http://docs.pylonsproject.org/projects/pyramid//en/latest/narr/hooks.html#registering-tweens">http://docs.pylonsproject.org/projects/pyramid//en/latest/narr/hooks.html#registering-tweens </a></li>
</ul>
<h3>
Course types (custom content types)</h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWM72VZ_cVQ0aUE4AokeZI2-WIxKAi4xAqQKZatRqReEzJZg3sbvhp2cq_An0IWfvmNcu5tY5PndsxPmS6PU-Vmv5_aVngLYpt2ASEH_fuLgxoZPshVoBjNy4W_ogkFvZAWjose_pS/s1600/mip-course.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWM72VZ_cVQ0aUE4AokeZI2-WIxKAi4xAqQKZatRqReEzJZg3sbvhp2cq_An0IWfvmNcu5tY5PndsxPmS6PU-Vmv5_aVngLYpt2ASEH_fuLgxoZPshVoBjNy4W_ogkFvZAWjose_pS/s320/mip-course.png" width="295" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The course view with portlets and collapsable paragraphs</td></tr>
</tbody></table>
They are a sort of <i>rich documents</i> with an <i>image attachment</i> column and integrated with an external <i>ecommerce </i>site. When you add a course type there is an event that <b>initializes automatically subobjects</b> and the main <b>image attachement by default</b>, so less work for editors.<br />
<br />
In addition all <b>image</b> content types and image attachments are addable or editable just by allowed users thank to <b>custom roles</b> and global or local permission sharing.<br />
<br />
Collapsable paragraphs are implemented with custom content types not directly reachable on the frontend.<br />
<br />
There are a lot of fields on this content type, so they are grouped together using <b>fieldsets</b>. <br />
Editors can also create a base private course model and then <b>copy and paste</b> it when new courses should be added.<br />
<br />
Sometimes you want to <b>prevent the addability on the site root</b>
for particular object types, this way things will remain always tidy
(why you should add a course on the very root of the site?). <br />
<br />
See:<br />
<ul>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-how-to-create-new-content-image.html">http://davidemoro.blogspot.com/2015/02/kotti-cms-how-to-create-new-content-image.html</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/03/kotti-cms-events-insert-subobjects.html">http://davidemoro.blogspot.com/2015/03/kotti-cms-events-insert-subobjects.html</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/05/kotti-cms-how-to-add-new-roles.html">http://davidemoro.blogspot.com/2015/05/kotti-cms-how-to-add-new-roles.html</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/01/kotti-avoid-types-addable-in-content.html">http://davidemoro.blogspot.com/2015/01/kotti-avoid-types-addable-in-content.html</a> </li>
<li><a href="http://kotti.readthedocs.org/en/latest/developing/basic/security.html">http://kotti.readthedocs.org/en/latest/developing/basic/security.html </a></li>
</ul>
<h3>
Windows and MySQL issues and tips</h3>
Kotti can be installed on <i>Windows</i> but I <b>strongly</b> suggest to adopt a Unix-like server with <i>Postgresql</i> instead of <i>MySQL</i>as storage layer: <br />
<ul>
<li><a href="http://davidemoro.blogspot.com/2015/03/pyramid-mysql-windows-good-ugly-bad.html">http://davidemoro.blogspot.com/2015/03/pyramid-mysql-windows-good-ugly-bad.html</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/01/how-to-install-kotti-cms-on-windows.html">http://davidemoro.blogspot.com/2015/01/how-to-install-kotti-cms-on-windows.html</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/03/pyramid-exceptions-logging.html">http://davidemoro.blogspot.com/2015/03/pyramid-exceptions-logging.html </a> </li>
</ul>
<h3>
Tests</h3>
All the software is tested. Very happy with the <i>py.test</i> framework.<br />
<br />
See:<br />
<ul>
<li><a href="http://pytest.org/latest/">http://pytest.org/latest/</a></li>
<li><a href="http://davidemoro.blogspot.it/2015/02/python-mock-library-for-ninja-testing.html">http://davidemoro.blogspot.it/2015/02/python-mock-library-for-ninja-testing.html</a></li>
</ul>
<h3>
Other third party Kotti plugins</h3>
I've used the following third party plugins that can be used on a standard Kotti environment: <br />
<ul>
<li><a href="https://pypi.python.org/pypi/kotti_calendar"><i>kotti_calendar</i></a> for events (very good solution!)</li>
<li><a href="https://pypi.python.org/pypi/kotti_newsitem"><i>kotti_newsitem</i></a> for news</li>
<li><a href="https://pypi.python.org/pypi/kotti_navigation"><i>kotti_navigation</i></a>, add a configurable navigation on our backend</li>
<li><i><a href="https://pypi.python.org/pypi/kotti_link">kotti_link</a></i>, link types</li>
</ul>
See also the full list of available plugins: <br />
<ul>
<li><a href="https://pypi.python.org/pypi?%3Aaction=search&term=kotti&submit=search">https://pypi.python.org/pypi?%3Aaction=search&term=kotti&submit=search </a></li>
</ul>
<h2>
Photoes and credits</h2>
<a href="http://www.mip.polimi.it/en" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img alt="http://www.mip.polimi.it/en" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWKzE2gc777RGp3hLWUPIFT4VI7tzhMwIKeNWUjXMkpYoMJfu37eL2LqlODcVQ31Y8sagntRE0TCsoA-c4GCnjQtRwLCnvnonQCM2UOkfdgopmxOSyqK7Tl7SuPyIaN3vZM9o0Oprf/s1600/mip-logo.png" /></a>All the screenshots shown in this article are taken from the "<b>MIP Politecnico di Milano's graduate school of business</b>" website:<br />
<ul>
<li><a href="http://www.mip.polimi.it/">http://www.mip.polimi.it</a> </li>
</ul>
So the MIP's website backend is powered by Pylons/Pyramid and Kotti CMS, I'll write a non-technical <b>case study</b> soon. In the mean time many thanks to:<br />
<ul>
<li>MIP</li>
<ul>
<li><i>Simona Strepparola</i>, Head of Communication</li>
<li><i>Gabriele Bedani</i>, Microsoft Sysadmin</li>
<li>all the MIP staff </li>
</ul>
<li>Truelab </li>
<ul>
<li><i>Riccardo Bonini</i>, project manager </li>
<li><i>Ruben Barilani</i>, web developer</li>
<li><i>Caio Ceccon</i>, web developer (Faculty and Staff implementation). Python developer that joined us for only 5 days
but enough to become good friends! He never
touched a Pyramid/Pylons or Kotti application but he was able to be
productive in a couple of days, this proves the Pyramid/Pylons developer
friendliness</li>
<li><i>Davide Moro</i>, it's me</li>
<li>Andrea Sironi, creative director </li>
</ul>
</ul>
<h2>
Results </h2>
You can consider Kotti as a <b>very good</b>, <b>secure</b>, <b>flexible</b>, <b>battle tested</b> and <b>easy to approach</b> solution for important customers.<br />
<h2>
All Kotti posts published by <a href="https://twitter.com/davidemoro">@davidemoro</a><br />
</h2>
<ul>
<li><a href="http://davidemoro.blogspot.it/search/label/kotti">http://davidemoro.blogspot.it/search/label/kotti </a></li>
</ul>
<h2>
Next steps </h2>
Reading this article you should find all the bricks you need if you want to implement a public website decoupled from its backend with Kotti.<br />
<br />
Now I'm assembling the above bricks in order to provide an "easy to install" solution with the same pattern I've described here. This is my roadmap: <br />
<ul>
<li><b>kotti_backend</b> (<a href="https://github.com/Kotti/kotti_backend">https://github.com/Kotti/kotti_backend</a>). Already published on PyPI, ready to be used. Turns Kotti CMS into a private content administration area</li>
<li><b>kotti_frontend</b> (<a href="https://github.com/Kotti/kotti_frontend">https://github.com/Kotti/kotti_frontend</a>). Public website solution completely decoupled by the Kotti backend with modern frontend tools (I want to obtain something like <a href="http://davidemoro.blogspot.it/2014/09/plone-angularjs-yeoman-starter.html">http://davidemoro.blogspot.it/2014/09/plone-angularjs-yeoman-starter.html</a> or <a href="http://davidemoro.blogspot.it/2014/09/pyramid-starter-seed-yeomam-part-1.html">http://davidemoro.blogspot.it/2014/09/pyramid-starter-seed-yeomam-part-1.html</a>). Optional support for blank sheet themes if you need more flexibility. It is just an incomplete but quite promising prototype, it is still under development.<br /><b>UPDATE 20150824:</b> no more kotti_frontend. See <a href="http://davidemoro.blogspot.com/2015/08/introducing-substancek-kotti-project.html">http://davidemoro.blogspot.com/2015/08/introducing-substancek-kotti-project.html</a> and <a href="https://github.com/substancek/substancek_cms_theme">https://github.com/substancek/substancek_cms_theme</a></li>
<li>a new pyramid tween implementing the editor toolbar available on the public website (TODO)</li>
<li>a meta-package "one command" installer based (a modified version of <a href="https://github.com/davidemoro/kotti_project">https://github.com/davidemoro/kotti_project</a>). Still in TODO</li>
<li>installable themes and themes generation (scaffolding)</li>
<li>a Kotti distribution that provides all the most useful third party plugins, just if you need a CMS. Now you have to install all external components manually.</li>
<li>REST interface support (included in standard Kotti's roadmap), very useful if you want to built heavy Javascript based applications </li>
</ul>
It can be considered a good starting point for:<br />
<ul>
<li>CMS-ish application</li>
<li>a generic framework for building generic web apps or single page web applications (remember, Kotti is not only a CMS, it could be considered as a framework) </li>
</ul>
So stay tuned and if you like this work please consider to <b>contribute</b> with<br />
<ul>
<li>code</li>
<li>testing</li>
<li>issue reporting</li>
<li>Github stars</li>
<li>spreading the word </li>
</ul>
or why not sponsorships!<br />
<br />
And if you want to know more about Kotti and you are attending <a class="g-profile" href="https://plus.google.com/106616906931886736572" target="_blank">+EuroPython Conference</a> 2015 in Bilbao don't miss the <b>Andreas Kaiser</b>'s talk "<b><i>Standing on the Shoulders of Giants: The Kotti Web Application Framework</i></b>". I'll join the sprint (remotely) at the end of EuroPython so see you on IRC (<a href="http://webchat.freenode.net/?channels=kotti">http://webchat.freenode.net/?channels=kotti</a>). If you want to stay tuned follow <a href="https://twitter.com/KottiCMS">https://twitter.com/KottiCMS</a>.Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-60607685279462018402015-07-16T18:27:00.004+02:002015-07-19T22:59:40.076+02:00Kotti CMS - a successful story (part 1)Yet another <b>Kotti CMS</b> (<a href="http://kotti.pylonsproject.org/">http://kotti.pylonsproject.org</a>) article.<br />
<br />
We'll see in this blog post:<br />
<ul>
<li>advantages of the public website decoupled (frontend) from the private content administration area (backend) pattern </li>
</ul>
Furthermore in this article you might find traces of:<br />
<ul>
<li>PHP vs Python comparison</li>
<li>Symfony2 vs Pylons/Pyramid</li>
<li>Plone vs Kotti</li>
<li>Python evangelism</li>
</ul>
But before starting let's explain what is the background of this true story and how Kotti CMS helped a lot.<br />
<h2>
Update 20150718</h2>
Part 2 is out! See:<br />
<ul>
<li><b><a href="http://bit.ly/1CP3gVo">frontend decoupled from backend. How we did it (part 2)</a></b>
</li>
</ul>
<ul>
</ul>
<h2>
The challenge</h2>
That were the requirements for an important project I joined last January:<br />
<ul><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6cggQYaN4T4t8x-ZO6FzLyi31DEeLgDut6r9DYeSGvGxgurBD6l-wSwYQHX8ZQhlOs7q4HiorlD_l8e7qBwEOBIisvCt3w_oQSbwIOyeM2bou4upvKJjb4Izi3rWbWz9og9HgCNZt/s1600/we-have-a-problem.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="212" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6cggQYaN4T4t8x-ZO6FzLyi31DEeLgDut6r9DYeSGvGxgurBD6l-wSwYQHX8ZQhlOs7q4HiorlD_l8e7qBwEOBIisvCt3w_oQSbwIOyeM2bou4upvKJjb4Izi3rWbWz9og9HgCNZt/s320/we-have-a-problem.jpg" width="320" /></a>
<li>build a new CMS from scratch (yeah, I know, it doesn't make sense reinventing the wheel: see <a href="http://davidemoro.blogspot.it/2015/04/how-to-choose-your-cms.html">How to choose your CMS</a>)</li>
<li>based on PHP/Symfony </li>
<li>with only two (2) developers</li>
<li>end user friendly</li>
<li>through the web customization for visual sections by editors/administrators (home page text introduction, main navigation, header and footer links, boxes, etc). </li>
<li>with an external ecommerce integration</li>
<li>custom logics and behaviours</li>
<li>custom security based on roles, workflow, permission sharing</li>
<li>configurable portlets system (manageable columns)</li>
<li>custom navigation link managers</li>
<li>multilingual</li>
<li>elasticsearch integration</li>
<li>nested urls (folders hierarchy)</li>
<li>stable</li>
</ul>
... with a not postponable 3 months deadline!<br />
<br />
Obviously there was a previous huge problem at commercial/estimation level and <i>Peppa Pig</i> would have said the famous "<b>THIS IS IMPOSSIBLE!</b>" because build a new CMS is an incredibly enormous task... but let's see why we are still alive and how it was possible to meet these impossible requirements with hard work, a bit of (italian) inventiveness and the Python/Pyramid based Kotti CMS.<br />
<h3>
1 - Does anyone already found a solution to a similar problem?</h3>
Not listed here:<br />
<ul>
<li><a href="http://www.chucknorrisfacts.com/">http://www.chucknorrisfacts.com</a></li>
<li><a href="http://macgyver.wikia.com/wiki/List_of_problems_solved_by_MacGyver">http://macgyver.wikia.com/wiki/List_of_problems_solved_by_MacGyver</a></li>
</ul>
but I think the <i>chucknorrisfacts</i> guys should update their site. Obviously Chuck Norris can solve this problem, he can also prevent it with a roundhouse kick. <br />
<ul>
</ul>
<h3>
2 - Switch technology to Plone attempt</h3>
Now I'm serious :)<br />
<br />
The first thought was: try to change the technology stack switching to Plone (<a href="http://plone.org/">http://plone.org</a>) but the Windows/PHP/Symfony/MySQL stack was not an option, so no Plone for this project.<br />
<h3>
3 - Have a look at existing PHP-based CMS solutions </h3>
Second step, check if there is a decent CMS solution built with PHP with this requirements:<br />
<ul>
<li>secure </li>
<li>easy to extend and maintain</li>
<li>with a good and modern codebase</li>
<li>with an intuitive backend interface for editors</li>
</ul>
and we were not able to find any existing solution matching our requirements criteria.<br />
In addition introducing a new framework usually means more time and a lot of pain if you have to customize almost everything if the framework it is not built with flexibility in mind. <br />
<h3>
4 - try out Sonata + Symfony2 from scratch</h3>
We tried to setup a Symfony2 project powered by Sonata and we
had a look at to existing early stage "CMS"s Sonata based but it was
clear that it wasn't the right tool for building a real content hierarchy
aware CMS usable by end users. Trust me, CMS is another sort of thing:
you cannot call CMS the ugly version of the Django admin or something
that is not suitable for end users. Anyway, Sonata is a quite good solution if you are going to write a data administration area for your custom Symfony2 application. See <a href="http://symfony.com/doc/1.0/cmf/cookbook/creating_cms_using_cmf_and_sonata.html">http://symfony.com/doc/1.0/cmf/cookbook/creating_cms_using_cmf_and_sonata.html</a> and <a href="http://demo.sonata-project.org/">http://demo.sonata-project.org/</a>.<br />
<h3>
5 - The solution</h3>
It was clear that it was impossible for us building a heavy customized CMS with almost "enterprise" requirements based on PHP for our team composition, experience, tight deadlines, etc.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhghi4sMA4r9Q_kEDn6kkEY1AETlvdNGJGNFd9ONc0YE6kOOnqYDRo3jOCpyOwt19dGAXE2Z7xLkxUuNoBKiv0-l7MtqU6feIExtGXnhQkfVEY0m85yreYUVkmiQNrKL1KNQqkOvJUs/s1600/a-team-plan.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="235" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhghi4sMA4r9Q_kEDn6kkEY1AETlvdNGJGNFd9ONc0YE6kOOnqYDRo3jOCpyOwt19dGAXE2Z7xLkxUuNoBKiv0-l7MtqU6feIExtGXnhQkfVEY0m85yreYUVkmiQNrKL1KNQqkOvJUs/s320/a-team-plan.png" width="320" /></a>So the solution was: why not adopting a good CMS based on a relational database storage powered by Python (there are very good options) and use it as a content administration backend area with a <b>decoupled frontend</b> built with Symfony2/PHP?<br />
<br />
We had a look to Django and Pyramid CMS solutions and <b>Kotti</b> seemed the best option:<br />
<ul>
<li>very good code base (and its core it is damn small)</li>
<li>powered by Pylons/Pyramid (quite small, flexible, easy to extend and approachable Python web framework with url traversing capabilities for free)</li>
<li>clean user interface</li>
<li>powered by SQLAlchemy </li>
<li>very very flexible if you need to turn the whole system inside out</li>
<li>many good concepts available in Kotti or Pyramid/Pylons are largely inspired by the Zope/Plone world (two decades of successful web applications and content management experience there)</li>
<li>it is not only a CMS but a web development framework </li>
</ul>
This way we were able to be extremely efficient and take advantage of our expertise from the very first day:<br />
<ul>
<li>1 PHP developer on the PHP-Symfony frontend side</li>
<li>1 Python developer on the Python Kotti CMS backend</li>
</ul>
As we will see that was our "winning" solution but I don't consider it optimal: this pattern itself is clean, powerful and I think I'll adopt the same technique in the next future in <b>pure Python</b>. Effectively there were some (well known) problems with our setup:<br />
<ul>
<li>PHP is not Python</li>
<li>Symfony2 is not Pylons/Pyramid (security, routing, traversal, view declarations with discriminators)</li>
<li>Kotti is not Plone (very good solution but it is still quite
minimal), so I had to implement the missing parts we need. The good news is that
Kotti and Pyramid you are <b>very productive </b></li>
<li>but... Pylons/Pyramid is Pylons/Pyramid! Very happy and damn productive, I really enjoyed programming with this framework. Very impressed.</li>
</ul>
<h4>
PHP is not Python</h4>
Concepts like decorators not available as PHP builtin: cool things like decorators are only comments that are "compiled" later. And it is more hard implementing new "decorators" and test them compared to Python.<br />
<br />
It seems that you cannot set to a class an instance of another class (our Kotti's <i>type_info</i> for example).<br />
<br />
Doctrine or Propel works fine if you are adopting the Docrine or Propel pattern. If you need something of different, no way. We wrote a custom abstraction on the top of the pattern used by Kotti and SQLAlchemy for types inheritance with joined tables (!!). Last but not least, SQLAlchemy provides things like:<br />
<ul>
<li><i>association_proxy</i> (used in Kotti tags). See <a href="http://docs.sqlalchemy.org/en/latest/orm/extensions/associationproxy.html">http://docs.sqlalchemy.org/en/latest/orm/extensions/associationproxy.html</a></li>
<li>other techniques used by <i>kotti_multilingual</i> used for independent fields. See <a href="https://github.com/Kotti/kotti_multilingual/blob/master/kotti_multilingual/sqla.py">https://github.com/Kotti/kotti_multilingual/blob/master/kotti_multilingual/sqla.py</a></li>
</ul>
and it feels quite magic because with PHP solutions the same patterns are not replicable so easily.<br />
<br />
Bit operations math for cookies shared auth among Symfony2 frontend and Kotti CMS backend.<br />
<br />
Generally it was a bit frustrating having to write a lot of code what you get for free with Python with just 2 lines of code. <br />
<h4>
Symfony2 is not Pylons/Pyramid </h4>
What is Symfony2: it is one of the better frameworks available
in the PHP world, probably the most promising. It is not a monolithic
framework, you can also use standalone symfony components as well in
your non-Symfony PHP projects (many famous existing PHP projects are
switching to Symfony). Symfony2 plays well with other existing
components like ORMs and it tries to bring innovation to PHP.<br />
It
promotes best development practise, testing, modularity, extensibility.
It has its own dependency injection system and it let you write
templates with a quite good template system named Twig.<br />
<br />
Anyway,
quite good ideas with concepts stolen from other non-PHP framework, in
particular dependency injection plus a configuration hooks if you want
to replace existing components without having to modify an existing
application (but no component adapter patterns). So with Symfony you can
write good code, <b>despite PHP</b>.<br />
<br />
Yeah, Symfony2 is quite
good and it tries to bring innovation to the PHP world: very good. But
if you feel innovative and you like so much innovation why not switching
to other technologies that were already more innovative 20 years ago?!
And now things are even better. Programming languages like Python and
web frameworks like Pylons/Pyramid are on another planet: <b>they really seem built by aliens</b> compared to other PHP frameworks.<br />
<br />
For example with Python you have:<br />
<ul>
<li>package managers like <i>pip</i> that exists since many years</li>
<li>ORM with extreme flexibility (<i>SQLAlchemy</i> is on another planet
too). Doctrine or Propel works fine just if you follow the
Doctrine/Propel way, if you need alternative patterns they might not
follow your needs with a lot of pain</li>
<li>less security vulnerabilities (see <a href="https://plone.org/products/plone/security/overview">https://plone.org/products/plone/security/overview</a>)</li>
<li>easy security setup for your applications</li>
<li>component adapter patterns</li>
<li>easy debugging (PDB)</li>
<li>routes and above all concepts like <b>traversal</b> (see <a href="http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/muchadoabouttraversal.html">http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/muchadoabouttraversal.html</a>)<b><br /></b></li>
<li>clean syntax and builtin concepts like decorators and context managers</li>
<li>no random byte code cache corruption</li>
<li>more productivity </li>
</ul>
Installation, setup, deploys and installation management seems more simpler too with Python in my opinion. If you setup your projects with <i>pip</i> you can deploy your applications with a couple of commands:<br />
<blockquote class="tr_bq">
<pre>$ git fetch
$ git checkout tags/0.1.5
$ pip install -r requirements.txt
$ restart your application</pre>
</blockquote>
or if you have complex requirements you can have a look at buildout.<br />
See:<br />
<ul>
<li><a href="http://davidemoro.blogspot.com/2015/04/pip-for-buildout-folks.html">http://davidemoro.blogspot.com/2015/04/pip-for-buildout-folks.html</a></li>
</ul>
So it is no more time "<b>with PHP you just copy and paste the
code</b>" if you are doing things seriously as you can see:
<a href="http://symfony.com/doc/current/cookbook/deployment/tools.html">http://symfony.com/doc/current/cookbook/deployment/tools.html</a><br />
Probably complex is the wrong word, it would be better verbose. For verbose I mean: composer install with a lot of options and d<i>ump-autoload</i>,
clear the Symfony2 prod cache avoiding cache corruption issues and
install assets, cache warmup, be sure there are right permissions for
writeable directories, etc.<br />
So the "just copy and paste the code" <b>myth (#1)</b> with PHP is not true anymore and I don't like it.<br />
<br />
And the very first PHP initial setup on your computer is not so
easy. I think for a newbie it is much more easy a different one or two commands approach:
install requirements and launch one command. <br />
If
you have to start with a PHP/Symfony2 setup just for developing is
quite a long and error prone task. That's why my very first experiment with PHP-Symfony2 was: adopt <b>Vagrant/Ansible</b> for environment setup and provisioning. If you need a Vagrant/Ansible example you can have a look at here: <a href="https://github.com/davidemoro/symfony-vagrant-ansible">https://github.com/davidemoro/symfony-vagrant-ansible</a>.<br />
<br />
Another false <b>myth (#2)</b>: the most used technology it is better than others: no. <br />
<br />
Yet another false <b>myth (#3)</b>: the most used technology is <b>more secure</b>: no, absolutely not. See again <a href="https://plone.org/products/plone/security/overview">https://plone.org/products/plone/security/overview</a>.<br />
<br />
Let me show other things you can do better with Python:<br />
<ul>
<li>"byte code cache" corruption issues</li>
<li>missing traversal concept</li>
<li>views registration and override</li>
<li>debugging</li>
<li>security configuration </li>
</ul>
The idea of a byte code cache (<a href="http://symfony.com/doc/current/book/performance.html">http://symfony.com/doc/current/book/performance.html</a>) is to remove the need to constantly recompile the PHP source code for improved performance. There shouldn't be any downside but we experienced random problems with <b>cache corruption</b> (not funny when it happens in production). <br />
<br />
Symfony2: <b>traversal... what?!</b> You need to implement it, while in Pylons/Pyramid comes for free. See <a href="http://docs.pylonsproject.org/projects/pyramid//en/latest/narr/muchadoabouttraversal.html">http://docs.pylonsproject.org/projects/pyramid//en/latest/narr/muchadoabouttraversal.html</a><br />
<br />
Add or customize views. Just one example: register or override the default view of a <i>Document</i> only if its parent is a <i>Course</i> if <b>damn easy with Pylons/Pyramid</b> (just register a new view with a discriminator, one line). With other frameworks you'll need to write a lot of not generic code. See <a href="http://docs.pylonsproject.org/projects/pyramid//en/latest/narr/viewconfig.html">http://docs.pylonsproject.org/projects/pyramid//en/latest/narr/viewconfig.html</a><br />
<br />
Debugging is a pain with PHP. Debugging things with PHP and/or Symfony is a bit frustrating if you had previous experience with the Python debugger (pdb) or with the pyramid debug toolbar and its <b><i>interactive through the web exception shell</i></b>. If something goes wrong <i>you can see what's the problem interactively</i> or why a variable is "None" <i>without leaving the browser</i>. See:<br />
<ul>
<li><a href="http://docs.pylonsproject.org/projects/pyramid-debugtoolbar/en/latest/index.html#the-toolbar">http://docs.pylonsproject.org/projects/pyramid-debugtoolbar/en/latest/index.html#the-toolbar </a></li>
</ul>
I didn't liked at all the <b>security configuration</b> of the Symfony2 framework with its "firewall" concept: the official doc itself admits it might be <b>confusing</b> and <b>tough</b> to set up just "because security is complex!" (see <a href="https://symfony.com/doc/current/book/security.html">https://symfony.com/doc/current/book/security.html</a>). Things can be complex but if your API lead people to be confused, probably you have a <b>bad design problem</b>.<br />
Anyway it is more easy working with Pylons/Pyramid if you are going to
write applications with complex security requirements (even
workflow-based security thanks to the third party <i>repoze.workflow</i>). See <a href="http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/security.html">http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/security.html</a><br />
<br />
I
hope you don't consider the above personal thoughts as not constructive
rants: Symfony2 is a good and promising PHP framework and if you are
interested in building web applications with PHP it is definitively a
very good choice. I have been working with Symfony for 6 months and this is just my opinion compared to my previous programming experience with Zope, Plone, Grok, BFG, Pylons/Pyramid, Django and a bit of Nodejs/Express (KeystoneJS, SailsJS, etc).<br />
<br />
So the previous Symfony2/PHP vs
Python/Pyramid/Python comparison boils down to: if you are a curious web
developer I think it really worth leaving your "comfort zone" and have a
look at something of new. It doesn't matter if you are going to use
Python (Django, Pyramid/Pylons, Flask, morepath, etc), Ruby, NodeJS or
whatever else: you'll learn a lot of useful new things for sure and
innovate your work!<br />
<h4>
Kotti CMS is not Plone</h4>
Kotti is very lightweight framework and its core is damn small because:<br />
<ul>
<li>a lot of things are demanded to the SQLAlchemy layer for storage, queries, indexing, etc.</li>
<li>simpler stack compared to Plone/Zope/CMF/Archetypes/Dexterity/etc (just Kotti, Pylons/Pyramid, SQLAlchemy + a list of well known and widely adopted third party libraries. So if you spend time learning Pylons/Pyramid, you can easily switch to other frameworks). For example if you have to write event handlers with Kotti you'll notice that things are more
simpler (no event handlers hell or having to manage duplicated events)</li>
<li>tries to keeps things simple</li>
<li>new programmers with no Pylons/Pyramid background are productive after a couple of days</li>
</ul>
and probably in next releases its core will shrink again.<br />
<br />
It is very easy to start with: just a couple of days and you will be ok because it is very lightweight.<br />
<br />
So programmers with no Pylons/Pyramid nor SQLAlchemy knowledge (or even not Python developers) will be productive very soon with Kotti. Obviously if you have a previous SQLAlchemy, Plone, Pylons/Pyramid or any other Python frameworks background it is better but it is not required at all. <br />
<br />
I think Kotti is very good if you have to write a custom relational database based application with CMS-ish features without having to remove all the things provided by Plone that you don't need at all: with Kotti or Pylons/Pyramid you only pay for what you eat. But Kotti is not only good for CMS-ish applications: it a <b>framework</b> that does very well as a backend for heavy Javascript single page web applications.<br />
<br />
Obviously Kotti is not Plone:<br />
<ul>
<li>community (the Plone's community is much more large)</li>
<li>Plone has a lot of third party plugins</li>
<li>no auto generated add/edit forms (not a problem because it is very simple to create or customize forms thanks to Colander/Deform. So no z3c.form there!)</li>
<li>no portlets (topic covered in next blog post)</li>
<li>no link actions (eg: <i>portal_actions</i>, topic covered in next blog post)</li>
<li>missing permalinks and not breakable links</li>
<li>no collections</li>
<li>no PloneFormGen-like plugins (but we used a very good form builder solution available online as a service with integrated CRM) </li>
<li>etc </li>
</ul>
but if you need a feature not provided by Kotti or other third party plugins don't worry because you will be productive very soon, even more if you adopt a frontend decoupled from the backend pattern.<br />
<h2>
Why frontend decoupled from the backend pattern</h2>
A couple of definitions about my frontend and backend concepts:<br />
<ul>
<li><i>frontend</i>. It is our public website, where anonymous users browse the contents of the website. In our case it is built with PHP/Symfony2 but obviously you can also do all things with Python alone (pure Python is the best option)</li>
<li><i>backend</i>. It is our private content management area, built with Kotti CMS, Pylons/Pyramid and Python</li>
</ul>
This pattern is so flexible that let you implement the public area with a completely different technology. This way you can choose the best solution for the content management private area (for example: Python) and keep control of the frontend with your in-house developers (for example: PHP).<br />
If you adopt a separation among frontend and backend your work will be even more agile:<br />
<ul>
<li>you can start with a "blank sheet" theme (topic covered in next articles)</li>
<li>you don't waste time removing or hiding features or unneeded views, just implement what you need. Probably guys with experience with big fat frameworks understand what I mean.</li>
<li>no CSS or Javascript conflicts with the backend. You won't be influenced at all how to integrate your usual frontend toolchain with a framework with its own tools or opinions because they are two completely different applications. So there is absolute frontend freedom, just choose the development stack you prefer</li>
<li>you can develop new complex features like portlets (box shown in views) without having to touch at all the backend interface because <b>the backend is just a (raw) backend area</b>! They are just another type of content (not publishable on the frontend but rendered in views). So you can adopt workflows on portlets, you can copy/cut and paste them, use different views on portlets, assign custom views, create custom addability rules based on portlet types, etc. This way you can develop complex features like portlets in a fraction of time!</li>
<li>since the frontend and the backend are served on the same domain, there is shared
authentication so you can implement toolbar, live edit or view site as anonymous for editors</li>
<li>less SEO headaches for non-publishable objects (portlets, tabs, collapsable sections, etc), you can store in the
administration area objects that won't be published by the frontend at
all. It is very handy implement these things like regular contents because you inherit workflows, same navigation/editing interface, etc. For example you can manage collapsable paragraphs as regular content type objects on the
backend (<i>yoursite.com/cms/document/paragraph1</i>) but they will be rendered on the document view <i>yoursite.com/document</i> (if you try to
access <i>yoursite.com/document/paragraph1 </i>from the frontend you'll get a <i>
404 NotFound</i>). Since they are not published on the public website you don't have to
protect urls, no need to redirect to the main parent content, etc: all these things requires a lot of extra
work if you want to take care of this kind of details</li>
<li>if you use the same technology for the public website and the private area, you can <b>reuse code</b> already defined on the backend (for example you can reuse the breadcrumbs callable view with a completely template)</li>
</ul>
And more: <br />
<ul>
<li>faster. You can build the frontend with less security checks (just expose public data with less complexity due to assertion about local roles,
groups, etc)</li>
<li>more secure. You'll keep the administration area (backend)
completely private and the frontend in our case built with PHP with a low privilege database user
(readonly) </li>
<li>you can rebuild the frontend without having to touch the administration area (backend)</li>
<li>frontend easy replaceable</li>
<li>different fulltext search policy for frontend vs frontend. On the backend there is the standard search results (you search for a text contained on a collapsable document and you'll land on that collapsable document on the backend, just what you expect from the editor perspective). If you search the same word on the public website you'll land on the parent document of the collapsable document (good for visitors) </li>
</ul>
<h2>
Photoes (credits)</h2>
A-team photoes from <a href="http://iqtell.com/2014/05/10-reasons-why-the-a-team-is-better-than-your-team/">http://iqtell.com/2014/05/10-reasons-why-the-a-team-is-better-than-your-team/</a> <br />
<h2>
All Kotti posts published by <a href="https://twitter.com/davidemoro">@davidemoro</a><br />
</h2>
<ul>
<li><a href="http://davidemoro.blogspot.it/search/label/kotti">http://davidemoro.blogspot.it/search/label/kotti </a></li>
</ul>
<ul>
</ul>
<h2>
End of part 1</h2>
In this article we have seen that:<br />
<ul>
<li>decoupled is cool!</li>
<li>Python is cool! </li>
</ul>
Instead in next blog posts I'll talk about:<br />
<ul>
<li>how to build a <b>pure Python</b> Kotti based setup <b>case study</b> with a private content management area decoupled from the public website (with tips, links and screenshots, etc).</li>
</ul>
<h2>
Update 20150718</h2>
Published part 2! See:<br />
<ul>
<li><b><a href="http://bit.ly/1CP3gVo">frontend decoupled from backend. How we did it (part 2)</a></b>
</li>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-51009546578152255942015-05-29T23:23:00.002+02:002015-05-29T23:23:17.216+02:00Kotti CMS - how to add new roles<b>Kotti CMS</b> has an intuitive user and permissions management that scales to fit the requirements of larger organizations.<br />
<br />
This blog post is a small recipe: how to add a new role in user management views and sharing views for local roles management. <br />
<br />
It is quite easy, it requires few lines of code you can add to your <i>__init__.py</i> module:<br />
<blockquote class="tr_bq">
<pre>from kotti.security import (
Principal,
ROLES,
SHARING_ROLES,
set_roles,
set_sharing_roles,
set_user_management_roles,
)
from kotti_yourpackage import _
def add_role(role_id, role_title):
""" Add role in share view and user management views """
UPDATED_ROLES = ROLES.copy()
UPDATED_ROLES[role_id] = Principal(role_id,
title=role_title)
UPDATED_SHARING_ROLES = list(SHARING_ROLES)
UPDATED_SHARING_ROLES.append(role_id)
set_roles(UPDATED_ROLES)
set_sharing_roles(UPDATED_SHARING_ROLES)
set_user_management_roles(UPDATED_SHARING_ROLES + ['role:admin'])
add_role(u'role:customer', _(u'Customer'))</pre>
</blockquote>
<br />
And now, we have a new <i>Customer</i> role ready to be used:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNPJeyeVGem_fbfeQ8XYVu7pn7fgxSwXPc5QTsZNsinuUhfyp4brnHMT0LqljTZqvtVjM1KWcuuCIkdb9-UT1P69mZzAYBiXrBi6Fs4oyhy3ChaC7t3RA3NrlVJ_TqFgg-4FLwykTk/s1600/kotti-sharing.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="170" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNPJeyeVGem_fbfeQ8XYVu7pn7fgxSwXPc5QTsZNsinuUhfyp4brnHMT0LqljTZqvtVjM1KWcuuCIkdb9-UT1P69mZzAYBiXrBi6Fs4oyhy3ChaC7t3RA3NrlVJ_TqFgg-4FLwykTk/s400/kotti-sharing.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">As you can see there is a new (pretend) "Customer" role</td></tr>
</tbody></table>
You can assign the <i>Customer</i> role as global or local (just on single nodes) to end users or even better group of users. <br />
<br />
After that you can work with ACLs or, better, configure what the customer role can do writing a custom <b>workflow</b> (see <a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-workflow-reference.html">http://davidemoro.blogspot.com/2015/02/kotti-cms-workflow-reference.html</a>).<br />
<h3>
All Kotti posts published by <a href="https://twitter.com/davidemoro">@davidemoro</a>:<br />
</h3>
<ul>
<li><a href="http://davidemoro.blogspot.it/search/label/kotti">http://davidemoro.blogspot.it/search/label/kotti </a></li>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com1tag:blogger.com,1999:blog-4821336113846598354.post-59715781793800756272015-05-23T00:51:00.001+02:002015-05-23T00:51:51.923+02:00Kotti CMS - ElasticSearch integrationAnnouncing a new <a href="http://kotti.pylonsproject.org/">Kotti CMS</a> (<i>Python</i> web framework based on <i>Pylons</i>/<i>Pyramid</i> and <i>SQLAlchemy</i>) plugin that provides <a href="https://www.elastic.co/products/elasticsearch">ElasticSearch</a> integration for fulltext search and indexing:<br />
<ul>
<li><b>kotti_es</b> (<a href="https://github.com/Kotti/kotti_es">https://github.com/Kotti/kotti_es</a>)</li>
</ul>
Development status? It should be considered <b>experimental</b> because this is the very first implementation. So any kind of help will be very appreciated! Beer, testing, pull releases, feedback, improving test coverage and so on.<br />
<h3>
Acknowledgements</h3>
<i>kotti_es</i> is based on a <b>pyramid_es</b> fork (<a href="https://github.com/truelab/pyramid_es/tree/feature-wrapper">https://github.com/truelab/pyramid_es/tree/feature-wrapper</a>, there is a PR in progress). The pyramid_es author is Scott Torborg (<a href="https://github.com/storborg">https://github.com/storborg</a>).<br />
<h3>
Configuration</h3>
The configuration is very simple.<br />
<br />
Just enable the kotti_es plugin just add the kotti_es plugin, choose the index name and elastic search server addresses.<br />
<br />
From the <i>kotti_es</i> README file: <br />
<blockquote class="tr_bq">
<pre>kotti.configurators =
kotti_es.kotti_configure
elastic.index = your_project
elastic.servers = localhost:9200
elastic.ensure_index_on_start = 1
kotti_es.blacklist =
Image
...
kotti.search_content = kotti_es.util.es_search_content</pre>
</blockquote>
<h3>
Index already existing contents</h3>
With <i>kotti_es</i> you can reindex all your already existing contents without any change to the original Kotti code base with just one command:<br />
<blockquote class="tr_bq">
<pre>$ reindex_es -c app.ini</pre>
</blockquote>
So <i>kotti_es</i> plays well with models defined by third party plugins that are not <i>ElasticSearch</i> aware. You can install kotti_es on an already existing Kotti instance.<br />
<h3>
Custom behaviours </h3>
If you want you can override/extend the default indexing policy just registering your custom adapter. See the <i>kotti_es </i>tests for more info.<br />
<br />
So no need to change existing models, no need to inherit from mixin classes and so on. <br />
<h3>
Video</h3>
<i>kotti_es</i> in action:<br />
<br />
<h3>
Wanna know more about Kotti CMS?</h3>
If you want to know more about Kotti CMS have a look at: <br />
<ul>
<li><a href="http://kotti.pylonsproject.org/">http://kotti.pylonsproject.org/</a></li>
<li><a href="http://www.pylonsproject.org/">http://www.pylonsproject.org/</a></li>
<li><a href="http://www.sqlalchemy.org/">http://www.sqlalchemy.org/</a></li>
</ul>
All Kotti posts published by <a href="https://twitter.com/davidemoro">@davidemoro</a>:<br />
<ul>
<li><a href="http://davidemoro.blogspot.it/search/label/kotti">http://davidemoro.blogspot.it/search/label/kotti </a></li>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-41337222613294343192015-04-11T00:40:00.000+02:002015-05-29T23:56:50.172+02:00kotti_multilingual<b>kotti_multilingual</b> is a package still in an early stage of development that adds to the <b>Kotti CMS</b> (<a href="http://kotti.pylonsproject.org/">http://kotti.pylonsproject.org/</a>) multilingual capabilities. It is neither feature complete nor can be considered API stable: so things will change!<br />
<br />
You can find various fixes on this fork (a PR is still in the review phase):<br />
<ul>
<li><a href="https://github.com/davidemoro/kotti_multilingual">https://github.com/davidemoro/kotti_multilingual</a></li>
</ul>
<h3>
How it works</h3>
First of all you should add a <i>LanguageRoot</i> folder in your root site. It is like the standard folderish <i>Document</i>, but with an editable <i>Language</i> attribute where you set the language code (eg: en, it, etc).<br />
<br />
Once you have created two or more language folders (with >2 languages there is a problem with the translation link actions at this time of writing) you can add your contents and translate them. <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWTtRJOObuez2cL_rrxenQWsivfl303QcF4zasSjRqGCjwrfeMy9Lp5BGI8NC7YQAXoRl2f3n-2CNxdCKvpm907jvwUtbZtmAVvG5g7bvv0VsC_kCCL6SfBWsx-GwV8DYVijBopytj/s1600/translate_link.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="142" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWTtRJOObuez2cL_rrxenQWsivfl303QcF4zasSjRqGCjwrfeMy9Lp5BGI8NC7YQAXoRl2f3n-2CNxdCKvpm907jvwUtbZtmAVvG5g7bvv0VsC_kCCL6SfBWsx-GwV8DYVijBopytj/s1600/translate_link.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The translate menu prompts a translate into italian action from /en/welcome</td><td class="tr-caption" style="text-align: center;"><br /></td></tr>
</tbody></table>
If you click on the translate into action, it will create a translated instance in <i>/it/welcome</i> (you can rename it later in <i>/it/benvenuto</i> or whatever you like) and you'll be redirected to a regular edit form prefilled with the english fields values.<br />
<br />
Once saved, you can switch to the existing translation and navigate among languages as shown in the following picture:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5yzZVJwc2FitDI0aQn2k3AAOCQwwT52PGfvRfDMhcTT1eal0uAja2WG7sQI8p2rjniqRPEpEitpwjM_VzPXdKcBpyEDepPG8z-9NEeZ6OvQVA-Nga3B69x6dWNWDNdW4EXaQXRpBD/s1600/translation_italian.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="146" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5yzZVJwc2FitDI0aQn2k3AAOCQwwT52PGfvRfDMhcTT1eal0uAja2WG7sQI8p2rjniqRPEpEitpwjM_VzPXdKcBpyEDepPG8z-9NEeZ6OvQVA-Nga3B69x6dWNWDNdW4EXaQXRpBD/s1600/translation_italian.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">You can switch to the existing English translation</td></tr>
</tbody></table>
<i>kotti_multilingual</i> supports the quite advanced concept of language independent fields: a field whose values should be inherited by translations, only editable on the root translation. <br />
<br />
You can see for example a select widget in edit mode on the root translation:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnZOol6opXvWmDBr4svluC9DwQgGW3UXb7qGDdFE9eESrR-WEnSaV18Rtlcen-MDprXNuGwa8KhlGm5XZAKJlA0OJO9uWkkEL_D1PBcKeamPWSdyTd2m14DkiBqYvOqpndSfXqQ00r/s1600/write_field1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="40" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnZOol6opXvWmDBr4svluC9DwQgGW3UXb7qGDdFE9eESrR-WEnSaV18Rtlcen-MDprXNuGwa8KhlGm5XZAKJlA0OJO9uWkkEL_D1PBcKeamPWSdyTd2m14DkiBqYvOqpndSfXqQ00r/s1600/write_field1.png" width="400" /></a></div>
And the same field in readonly mode on the translated object:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVgxWUpdv4_04o4S8KP7lOm7Yc9dowyY6TFDEgSBXCuPn9Go7nT0cjAprZypXsGi5XCUp0MEGJze7eID7v1Y5yaHp2yTDlwfrmOR5Xzs6BUyOeHgpHjE2GExYQJHDHbpfD2cHm96Fy/s1600/readonly_field.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="40" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVgxWUpdv4_04o4S8KP7lOm7Yc9dowyY6TFDEgSBXCuPn9Go7nT0cjAprZypXsGi5XCUp0MEGJze7eID7v1Y5yaHp2yTDlwfrmOR5Xzs6BUyOeHgpHjE2GExYQJHDHbpfD2cHm96Fy/s1600/readonly_field.png" width="400" /></a></div>
See the <i>kotti_multilingual.widget.i10n_widget_factory</i> code for more info.<br />
<h3>
Code examples </h3>
And now code examples.<br />
<h4>
resources.py</h4>
You can define language independent fields in your type_info attribute on your resource.<br />
<blockquote class="tr_bq">
<pre>class YourResource(...):
...
type_info = Document.type_info.copy(
...
language_independent_fields= ['course_sku',],
)</pre>
</blockquote>
<h4>
views/edit.py</h4>
The edit form does not require changes, you just need to apply the <i>i10n_widget_factory</i> on your language independent fields (in some particular cases you need a bit more complex setup when you have to deal with not null column, required fields, etc). In these particular cases you'll have to play with <i>get_source</i> (<i>kotti_multilingual.api.source</i>) and put the widget in readonly mode. If you experience problems cloning select widgets you might have to migrate to deferred widgets (that creates a new widget instance each time) and set the widget mode in readonly when needed. <br />
<br />
<blockquote class="tr_bq">
<pre>from kotti_multilingual.widget import i10n_widget_factory
...
from kotti_multilingual.api import get_source </pre>
</blockquote>
<blockquote class="tr_bq">
<pre>@colander.deferred
def deferred_widget(node, kw):
request = kw['request']
context = request.context
...
widget = SelectWidget(values=available_tags, multiple=True)
if get_source(context) is not None:
widget.readonly = True
return widget
class YourResourceSchema(colander.Schema):
course_sku = colander.SchemaNode(
colander.String(),
title=_(u"Course SKU"),
missing=u"",
widget=i10n_widget_factory(TextInputWidget),
)
class YourResourceAddForm(ImageAddForm):
schema_factory = YourResourceSchema
...
def get_bind_data(self):
bind_data = super(YourResourceAddForm, self).get_bind_data()
# we tell to the i10n_widget_factory that this is an addform,
# so our widgets will be shown as usual in edit mode
bind_data['addform'] = True
return bind_data</pre>
</blockquote>
<h3>
Final thoughts</h3>
Yes, it is a very very young package but very promising!<br />
It is not complete and probably it never will be complete because <i>SQLAlchemy</i> is huge and I think it is not possible to cover all the possible <i>SQLAlchemy</i> combinations.<br />
<br />
For example this fork includes support for the <i>SQLAlchemy</i>'s <i>association_proxy</i> feature and language independent fields (in this case the <i>copy_properties_blacklist</i> attribute on your resource is your friend).<br />
<br />
This is open source, dude: if you need something that is not yet covered, just fork <i>kotti_multilingual</i>, implement the missing parts and share with others!<br />
<h4>
Update 20150427</h4>
Merged and released new version of kotti_multilingual on PyPI <a href="https://pypi.python.org/pypi/kotti_multilingual/0.2a3">https://pypi.python.org/pypi/kotti_multilingual/0.2a3</a>.<br />
Development happens on <a href="https://github.com/Kotti/kotti_multilingual">https://github.com/Kotti/kotti_multilingual</a> now.<br />
<h3>
All Kotti posts published by <a href="https://twitter.com/davidemoro">@davidemoro</a>:<br />
</h3>
<ul>
<li><a href="http://davidemoro.blogspot.it/search/label/kotti">http://davidemoro.blogspot.it/search/label/kotti </a></li>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-34467399218246973192015-04-08T23:53:00.001+02:002015-06-29T23:26:16.479+02:00Pip for buildout folks... or buildout for pip folks.<br />
<br />
In this article I'm going to talk about how to manage software (Python) projects with <b>buildout</b> or <b>pip</b>.<br />
<br />
What do you mean for <b>project</b>?<br />
<blockquote class="tr_bq">
A package that contains all the
application-specific settings, database configuration, which packages
your project will need and where they lives.</blockquote>
Projects should be managed like a software if you want to assure the needed <b>quality</b>:<br />
<ul>
<li>under version control</li>
<li>following a good branching model, see one of my preferred blog posts <a href="http://nvie.com/posts/a-successful-git-branching-model/">http://nvie.com/posts/a-successful-git-branching-model/</a></li>
<li>with a <i>CHANGES.rst</i>. It is very very important. If you deploy frequently small releases you can easily understand when you introduced a weird bug and fix it, you can see what version is installed on production, etc. You can keep track about what changed in a particular release. For example all software updates
(example: kotti_boxes: 0.1.1 -> 0.1.2) with full info about why
changed, when, who, etc. Based on this rst file you can also generate as
well the project documentation with Sphinx (<a href="http://sphinx-doc.org/">http://sphinx-doc.org/</a>), etc. </li>
<li>tagged (<i>zest.releaser</i> can help you. See <a href="https://opensourcehacker.com/2012/08/14/high-quality-automated-package-releases-for-python-with-zest-releaser/">https://opensourcehacker.com/2012/08/14/high-quality-automated-package-releases-for-python-with-zest-releaser/</a>)</li>
<li>documentation </li>
<li>etc</li>
</ul>
This blog post is not:<br />
<ul>
<li>intended to be a complete guide to pip or buildout. If you want to know more about pip or buildout</li>
<li>talking about how to deploy remotely your projects</li>
</ul>
<h3>
Buildout</h3>
I've been using buildout for many years and we are still good friends.<br />
Buildout definition (from <a href="http://www.buildout.org/">http://www.buildout.org</a>):<br />
<blockquote class="tr_bq">
<i>"""</i><br />
<i>Buildout
is a Python-based build system for creating, assembling and deploying
applications from multiple parts, some of which may be non-Python-based.
It lets you create a buildout configuration and reproduce the same
software later. </i><br />
<i>"""</i></blockquote>
With buildout you can build and share reproducible environments, <b>not only for Python based components</b>.<br />
<br />
Before
buildout (if I remember well the first time I get started to use
buildout was in 2007, probably during the very first Plone Sorrento sprint) it
was a real pain sharing a complete and working developing environment
pointing to the right version of several repositories, etc. With
buildout it was questions of minutes.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsjKnBReBf-WFkVEDvEniVuC4UvsgMQ1jMl1TvFqfkOKJ9M95qlYBzc3cV3juFQhyfBMCTpFKnL_xvsVg2nQByDANFbjosy3zmcgPyDYXqEkosMEIRPoQOnK2kuo7Y16uVy3ljE_0Y/s1600/xkcd-buildout.png" style="margin-left: auto; margin-right: auto;"><img border="0" height="278" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsjKnBReBf-WFkVEDvEniVuC4UvsgMQ1jMl1TvFqfkOKJ9M95qlYBzc3cV3juFQhyfBMCTpFKnL_xvsVg2nQByDANFbjosy3zmcgPyDYXqEkosMEIRPoQOnK2kuo7Y16uVy3ljE_0Y/s1600/xkcd-buildout.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">From https://pypi.python.org/pypi/mr.developer.<br />
Probably with pip there is less fun because there isn't a funny picture that celebrates it?!</td></tr>
</tbody></table>
Buildout configuration files are modular and
extensible (not only on per-section basis). There are a lot of buildout
recipes, probably the one I prefer is <b>mr.developer</b>
(<a href="https://pypi.python.org/pypi/mr.developer">https://pypi.python.org/pypi/mr.developer</a>). It allowed me to fetch
different versions of the repositories depending on the buildout profile
in use, for example:<br />
<ul>
<li>production -> each developed private egg point to a tag version</li>
<li>devel -> the same eggs point to the develop/master</li>
</ul>
You can accomplish this thing creating different configurations for different profiles, like that:<br />
<pre><blockquote class="tr_bq">
[buildout]<br />
...<br />
[sources]<br />
your_plugin = git git@github.com:username/your_plugin.git<br />
...
</blockquote>
</pre>
I don't like calling <i>./bin/buildout -c [production|devel].cfg</i>
with the <i>-c</i> syntax because it is too much error prone. I prefer to create a symbolic link
to the right buildout profile (called buildout.cfg) and you'll perform the same command both
in production or during development always typing:<br />
<pre><blockquote class="tr_bq">
$ ./bin/buildout</blockquote>
</pre>
This way you'll avoid nasty errors
like launching a wrong profile in producion. So use just the plain <i>./bin/buildout</i> command and live happy.<br />
<br />
With buildout you can show and freeze all the installed versions of your packages providing a <i>versions.cfg</i> file.<br />
<br />
Here you can see my preferred buildout recipes:<br />
<ul>
<li><a href="https://pypi.python.org/pypi/mr.developer">https://pypi.python.org/pypi/mr.developer</a>, already discussed before</li>
<li><a href="https://pypi.python.org/pypi/collective.recipe.cmd">https://pypi.python.org/pypi/collective.recipe.cmd</a>, useful for launching commands</li>
<li><a href="https://pypi.python.org/pypi/z3c.recipe.template">https://pypi.python.org/pypi/z3c.recipe.template</a>, useful for dynamic template generation (like logrotate files, nginx/apache configurations, etc)</li>
<li><a href="https://pypi.python.org/pypi/zc.recipe.cmmi">https://pypi.python.org/pypi/zc.recipe.cmmi</a>, compilation on the fly for dependencies hard to install. Sometimes you can provide a very basic optional buildout configuration in your packages if they depends on external software hard to configure (for example you can provide a fully configured solr installation if you assume a specific setup) </li>
</ul>
Buildout or not buildout, one of the of the most common needs it is the ability to switch from <b>develop</b> to <b>tags</b> depending on you are in development or production mode and <b>reproduce</b> the same software later. I can't figure out to manage software installations without this quality assurance.<br />
<br />
More info: <a href="http://www.buildout.org/">http://www.buildout.org</a> <br />
<h3>
Pip</h3>
Let's see how to create reproducible environments with develop or tags dependencies for production environments with pip (<a href="https://pip.pypa.io/en/latest/">https://pip.pypa.io/en/latest/</a>).<br />
<br />
Basically you specify your devel requirements on a <i>devel-requirements.txt</i> file (the name doesn't matter) pointing to the develop/master/trunk on your repository.<br />
<br />
There is another file that I call <i>production-requirements</i> (the file name doesn't matter) that it is equivalent to the previous one but:<br />
<ul>
<li>without devel dependencies you don't want to install in production mode</li>
<li>tagging your private applications (instead of master -> 0.1.1)</li>
</ul>
This way it is quite simple seeing which releases are installed in production mode, with no cryptic hash codes.<br />
<br />
You can use now the <i>production-requirements.txt</i> as a template for generating an easy to read <i>requirements.txt</i>. You'll use this file when installing in production.<br />
<br />
You can create a regular <i>Makefile</i> if you don't want to repeat yourself or make scripts if you prefer:<br />
<ul>
<li>compile Sphinx documentation</li>
<li>provide virtualenv initialization</li>
<li>launch tests against all developed eggs </li>
<li>update the final <i>requirements.txt</i> file</li>
</ul>
For example if you are particular lazy you can create a script that will create your <i>requirements.txt</i> file using the <i>production-requirements.txt</i> like a template.<br />
This is a simple script, it is just an example, that shows how to build your requirements.txt omitting lines with grep, sed, etc:<br />
<blockquote class="tr_bq">
<pre><span class="c">#!/bin/bash</span>
<a href="https://www.blogger.com/null" name="cl-2"></a>
<a href="https://www.blogger.com/null" name="cl-3"></a>pip install -r production-requirements.txt
<a href="https://www.blogger.com/null" name="cl-4"></a>pip freeze -r production-requirements.txt <span class="p">|</span> grep -v mip_project <span class="p">|</span> sed <span class="s1">'1,2d'</span> > requirements.txt</pre>
</blockquote>
When running this script, you should activate another Python environment in order to not pollute the production requirements list with development stuff. <br />
<br />
If you want to make your software reusable and as flexible as possible, you can add a regular <i>setup.py</i> module with optional dependencies, that you can activate depending on what you need. For example in devel-mode you might want to activate an entry point called <i>docs</i> (see <i>-e .[docs]</i> in <i>devel-requirements.txt</i>) with optional Sphinx dependencies. Or in production you can install MySQL specific dependencies (<i>-e .[mysql]</i>). <br />
<br />
In the examples below I'll also show how to refer to external requirements file (url or a file). <br />
<h4>
setup.py</h4>
You can define optional extra requirements in your <i>setup.py</i> module.<br />
<blockquote class="tr_bq">
<pre><span class="n">mysql_requires</span> <span class="o">=</span> <span class="p">[</span>
<a href="https://www.blogger.com/null" name="cl-27"></a> <span class="s">'MySQL-python'</span><span class="p">,</span>
<a href="https://www.blogger.com/null" name="cl-28"></a> <span class="p">]</span>
<a href="https://www.blogger.com/null" name="cl-31"></a>
<a href="https://www.blogger.com/null" name="cl-32"></a><span class="n">docs_requires</span> <span class="o">=</span> <span class="p">[</span>
<a href="https://www.blogger.com/null" name="cl-33"></a> <span class="s">'Sphinx'</span><span class="p">,</span>
<a href="https://www.blogger.com/null" name="cl-34"></a> <span class="s">'docutils'</span><span class="p">,</span>
<a href="https://www.blogger.com/null" name="cl-35"></a> <span class="s">'repoze.sphinx.autointerface'</span><span class="p">,</span>
<a href="https://www.blogger.com/null" name="cl-37"></a><span class="p">]</span>
<a href="https://www.blogger.com/null" name="cl-38"></a>...
<a href="https://www.blogger.com/null" name="cl-39"></a>
<a href="https://www.blogger.com/null" name="cl-40"></a><span class="n">setup</span><span class="p">(</span>
<a href="https://www.blogger.com/null" name="cl-41"></a> <span class="n">name</span><span class="o">=</span><span class="s">'mip_project'</span><span class="p">,</span>
<a href="https://www.blogger.com/null" name="cl-42"></a> <span class="n">version</span><span class="o">=</span><span class="n">version</span><span class="p">,</span>
<a href="https://www.blogger.com/null" name="cl-43"></a> ...
<a href="https://www.blogger.com/null" name="cl-70"></a> <span class="n">extras_require</span><span class="o">=</span><span class="p">{</span>
<a href="https://www.blogger.com/null" name="cl-71"></a> <span class="s">'mysql'</span><span class="p">:</span> <span class="n">mysql_requires</span><span class="p">,</span>
<a href="https://www.blogger.com/null" name="cl-73"></a> <span class="s">'docs'</span><span class="p">:</span> <span class="n">docs_requires</span><span class="p">,</span></pre>
<pre><span class="p"> ... </span>
<a href="https://www.blogger.com/null" name="cl-75"></a> <span class="p">},</span></pre>
</blockquote>
<h4>
devel-requirements.txt</h4>
Optional extra requirement can be activated using the [] syntax (see <i>-e .[docs]</i>).<br />
You can also include external requirement files or urls (see <i>-r</i>) and tell pip how to fetch some concrete dependencies (see <i>-e git+...#egg=your_egg</i>).<br />
<blockquote class="tr_bq">
<pre>-r https://github.com/.../.../blob/VERSION/requirements.txt</pre>
<pre> </pre>
<pre><a href="https://www.blogger.com/null" name="cl-3"></a># Kotti
<a href="https://www.blogger.com/null" name="cl-4"></a>Kotti[development,testing]==VERSION
<a href="https://www.blogger.com/null" name="cl-5"></a>
<a href="https://www.blogger.com/null" name="cl-6"></a># devel (to no be added in production)
<a href="https://www.blogger.com/null" name="cl-7"></a>zest.releaser
<a href="https://www.blogger.com/null" name="cl-8"></a>
<a href="https://www.blogger.com/null" name="cl-9"></a># Third party's eggs
<a href="https://www.blogger.com/null" name="cl-10"></a>kotti_newsitem==0.2
<a href="https://www.blogger.com/null" name="cl-11"></a>kotti_calendar==0.8.2
<a href="https://www.blogger.com/null" name="cl-12"></a>kotti_link==0.1
<a href="https://www.blogger.com/null" name="cl-13"></a>kotti_navigation==0.3.1
<a href="https://www.blogger.com/null" name="cl-14"></a>
<a href="https://www.blogger.com/null" name="cl-15"></a># Develop eggs
<a href="https://www.blogger.com/null" name="cl-18"></a>-e git+https://github.com/truelab/kotti_actions.git#egg=kotti_actions
<a href="https://www.blogger.com/null" name="cl-19"></a>-e git+https://github.com/truelab/kotti_boxes.git#egg=kotti_boxes
<a href="https://www.blogger.com/null" name="cl-20"></a>...
<a href="https://www.blogger.com/null" name="cl-22"></a>
<a href="https://www.blogger.com/null" name="cl-23"></a>-e .[docs]</pre>
</blockquote>
<h4>
production_requirements.txt</h4>
The production requirements should point to tags (see <i>@VERSION</i>). <br />
<blockquote class="tr_bq">
<pre>-r https://github.com/Kotti/Kotti/blob/VERSION/requirements.txt
<a href="https://www.blogger.com/null" name="cl-4"></a>Kotti[development,testing]==VERSION <a href="https://www.blogger.com/null" name="cl-2"></a>
<a href="https://www.blogger.com/null" name="cl-3"></a><a href="https://www.blogger.com/null" name="cl-4"></a>
<a href="https://www.blogger.com/null" name="cl-5"></a># Third party's eggs
<a href="https://www.blogger.com/null" name="cl-6"></a>kotti_newsitem==0.2
<a href="https://www.blogger.com/null" name="cl-7"></a>kotti_calendar==0.8.2
<a href="https://www.blogger.com/null" name="cl-8"></a>kotti_link==0.1
<a href="https://www.blogger.com/null" name="cl-9"></a>kotti_navigation==0.3.1
<a href="https://www.blogger.com/null" name="cl-10"></a>
<a href="https://www.blogger.com/null" name="cl-11"></a># Develop eggs
<a href="https://www.blogger.com/null" name="cl-12"></a>-e git+https://github.com/truelab/kotti_actions.git@0.1.1#egg=kotti_actions
<a href="https://www.blogger.com/null" name="cl-15"></a>-e git+https://github.com/truelab/kotti_boxes.git@0.1.3#egg=kotti_boxes
<a href="https://www.blogger.com/null" name="cl-16"></a>...
<a href="https://www.blogger.com/null" name="cl-18"></a>
<a href="https://www.blogger.com/null" name="cl-19"></a>-e .[mysql] </pre>
</blockquote>
<b>requirements.txt</b><br />
The <i>requirements.txt</i> is autogenerated based on the <i>production-requirements.txt</i> model file. All the installed versions are appended in alphabetical at the end of the file, it can be a very long list.<br />
All the tag versions provided in the production-requirements.txt are automatically converted to hash values (<i>@VERSION</i> -> <i>@3c1a191...</i>).<br />
<blockquote class="tr_bq">
<pre>Kotti==1.0.0a4
<a href="https://www.blogger.com/null" name="cl-2"></a>
<a href="https://www.blogger.com/null" name="cl-3"></a># Third party's eggs
<a href="https://www.blogger.com/null" name="cl-4"></a>kotti-newsitem==0.2
<a href="https://www.blogger.com/null" name="cl-5"></a>kotti-calendar==0.8.2
<a href="https://www.blogger.com/null" name="cl-6"></a>kotti-link==0.1
<a href="https://www.blogger.com/null" name="cl-7"></a>kotti-navigation==0.3.1
<a href="https://www.blogger.com/null" name="cl-8"></a>
<a href="https://www.blogger.com/null" name="cl-9"></a># Develop eggs
<a href="https://www.blogger.com/null" name="cl-10"></a>-e git+https://github.com/truelab/kotti_actions.git@3c1a1914901cb33fcedc9801764f2749b4e1df5b#egg=kotti_actions-dev
<a href="https://www.blogger.com/null" name="cl-13"></a>-e git+https://github.com/truelab/kotti_boxes.git@3730705703ef4e523c566c063171478902645658#egg=kotti_boxes-dev
...<a href="https://www.blogger.com/null" name="cl-14"></a><a href="https://www.blogger.com/null" name="cl-16"></a>
<a href="https://www.blogger.com/null" name="cl-17"></a>
<a href="https://www.blogger.com/null" name="cl-18"></a>## The following requirements were added by pip freeze:
<a href="https://www.blogger.com/null" name="cl-19"></a>alembic==0.6.7
<a href="https://www.blogger.com/null" name="cl-20"></a>appdirs==1.4.0
<a href="https://www.blogger.com/null" name="cl-21"></a>Babel==1.3
<a href="https://www.blogger.com/null" name="cl-22"></a>Beaker==1.6.4</pre>
<pre>... </pre>
</blockquote>
<h3>
Final consideration</h3>
Use pip to install Python packages from <a href="https://pypi.python.org/pypi">Pypi</a>. <br />
<br />
If you’re looking for management of fully integrated cross-platform software
stacks, buildout is for you.<br />
<br />
With buildout no Python code needed unless you are going to write new
recipes (the plugin mechanism provided by buildout to add new
functionalities to your software building, see <a href="http://buildout.readthedocs.org/en/latest/docs/recipe.html">http://buildout.readthedocs.org/en/latest/docs/recipe.html</a>).<br />
<br />
Instead with pip you can manage also cross-platform stacks but you loose the flexibility of buildout recipes and inheritable configuration files.<br />
<br />
Anyway if you consider buildout too magic or you just need a way to switch from production vs development mode you can use pip as well.<br />
<h3>
Links</h3>
If you need more info have a look at the following urls:<br />
<ul>
<li><a href="https://python-packaging-user-guide.readthedocs.org/en/latest/current.html">https://python-packaging-user-guide.readthedocs.org/en/latest/current.html</a></li>
<li><a href="http://www.buildout.org/">http://www.buildout.org</a> </li>
<li><a href="https://pip.pypa.io/en/latest/">https://pip.pypa.io/en/latest/</a></li>
</ul>
Other useful links:<br />
<ul class="simple">
<li><a class="reference external" href="http://www.ianbicking.org/blog/2008/12/using-pip-requirements.html">http://www.ianbicking.org/blog/2008/12/using-pip-requirements.html</a></li>
<li><a class="reference external" href="https://caremad.io/2013/07/setup-vs-requirement/">https://caremad.io/2013/07/setup-vs-requirement/</a></li>
<li><a class="reference external" href="https://devcenter.heroku.com/articles/python-pip">https://devcenter.heroku.com/articles/python-pip</a></li>
<li><a class="reference external" href="https://pip.pypa.io/en/latest/user_guide.html">https://pip.pypa.io/en/latest/user_guide.html</a></li>
</ul>
<h3>
Update 20150629</h3>
If you want an example I've created a pip-based project for <b>Kotti CMS</b> (<a href="http://kotti.pylonsproject.org/">http://kotti.pylonsproject.org</a>):<br />
<ul>
<li><a href="https://github.com/davidemoro/kotti_project">https://github.com/davidemoro/kotti_project</a> </li>
</ul>
<ul class="simple">
</ul>
<h4>
</h4>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com1tag:blogger.com,1999:blog-4821336113846598354.post-19298671493081791592015-04-01T23:26:00.004+02:002015-07-20T10:35:30.116+02:00How to choose your CMSThis article is dedicated to <b>non Python folks</b> searching for a good CMS because I've seen too many times people taking bad decisions.<br />
<br />
If you are searching for a <b>CMS</b> solution or are you writing a custom web application with content management features inside, you should consider also one of the existing solutions built with <b>Python</b> before choosing your platform. Even if your core business is based on different technologies or other programming languages like Java or PHP.<br />
<br />
I'm not telling that you have to choose Python solutions, I'm just
suggesting you to start with a 360 degrees serious software selection before choosing. <br />
<br />
Obviously the final choice depends on what you have to do: a lot of times a Wordpress site with a few dollars theme and a couple of third party plugins is good enough.<br />
<br />
But if you are plan to build something more complex that involves:<br />
<ul>
<li>top level security solutions </li>
<li>complex security requirements </li>
<li>extensibility without having to fork the existent code</li>
<li>heavy, but still maintainable, customizations in core logics</li>
<li>you are not going to build a website, but something of more complex like an intranet</li>
<li>you are going to build something that can evolve for covering future changes without requiring a complete rewrite </li>
</ul>
you should evaluate carefully your future platform before deciding.<br />
<br />
If you have the above strong requirements, you can have a look at the following projects:<br />
<ul>
<li><a href="https://plone.org/">Plone</a> (nosql, enterprise level, a lot of plugins)</li>
<li><a href="http://kotti.pylonsproject.org/">Kotti</a> (traditional database storage, more lightweight, easy to approach)</li>
</ul>
They are both Python based, quite different solutions but they have common roots and a long CMS tradition. They are open source, secure, with a friendly an skilled community
and a <b>clean editing experience</b> compared with other "CMS" solutions (you
can't call CMS something that looks like a messy version of an admin
interface! Just good for technicians or for small projects, not for
content editors).<br />
<br />
I'm just suggesting: if you are evaluating CMS frameworks, don't choose a solution because it is built with your preferred programming language or framework but consider also other existing solutions. Don't choose a project because you were able to change the logo and few colours quickly. You should evaluate the main available solutions and after that choose the best fit for you... and be prepared to <b>change</b> technology if needed.<br />
<br />
Change attitude also means continuously searching for something of better, find new opportunities, learning something of new, learn new development pratices, get involved in other open source communities: it is definitively a good thing. You are not disrupting your existing knowledge, you are <b>improving </b>your experience and enlarging your <b>knowledge</b>. So don't be scared, choose the best for you and be open minded!<br />
<br />
Another hint: don't be tempted to write from scratch a new CMS unless you are know well what you are doing and/or you have a good budget with dozens of experienced developers. Still having the idea of writing your new CMS from scratch?! Have a look at how many contributors and how many years are needed before an open source CMS it is considered stable with a good feature set and then decide. Don't reinvent the wheel!<br />
<br />
Or if you don't want to learn another framework or programming language but you found the best fit for you: <b>hire a consultant</b> but always <b>choose the best</b>. <br />
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi58r6ZgX2vsRVUW41rTsbUgI_eEBr2IrLXhobYiGtM7Zh8EZwyE4U1qRlg8ZwL3quUnsLgDCC56pLwz-kkZh30wx6Ez7tQNEOo-WHZzoc2BI86hRWrok_N1mmLte1h3jE2FQrgis6Q/s1600/indiana-jones.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi58r6ZgX2vsRVUW41rTsbUgI_eEBr2IrLXhobYiGtM7Zh8EZwyE4U1qRlg8ZwL3quUnsLgDCC56pLwz-kkZh30wx6Ez7tQNEOo-WHZzoc2BI86hRWrok_N1mmLte1h3jE2FQrgis6Q/s1600/indiana-jones.jpg" width="130" /></a><i>PS: if you like adventure and you chose a RDBMS based solution like Kotti you have another option: you can follow a <b>hybrid approach</b>.</i><br />
<br />
<i>This way you can use Kotti as a private content management area and implement your public website with the technology or frontend tools you like exposing data marked as published on the database. There are a lot of benefits due to backend decoupled from the frontend.</i><br />
<br />
<i>Maybe I'll talk about a similar successful case history built with this technique in an another blog post. Stay tuned with <a href="https://twitter.com/davidemoro">@davidemoro</a>.</i><br />
<h2>
Update <i>20150720 - hybrid approach</i></h2>
<ul>
<li>advantages of the public website decoupled (frontend) from the private content administration area (backend) pattern - a true story <b><a href="http://davidemoro.blogspot.com/2015/07/kotti-cms-successful-story-part-1.html">http://davidemoro.blogspot.com/2015/07/kotti-cms-successful-story-part-1.html</a></b><i> </i></li>
<li>decoupled public website reference (technical reference, links, screenshots, case study) <a href="http://davidemoro.blogspot.com/2015/07/kotti-cms-frontend-decoupled-backend-part-2.html"><b>http://davidemoro.blogspot.com/2015/07/kotti-cms-frontend-decoupled-backend-part-2.html</b></a><i><br /></i></li>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-5154283430947697022015-03-28T14:34:00.001+01:002015-03-28T14:34:54.888+01:00Pyramid exceptions loggingIf you want to log exceptions with <b>Pyramid</b> (<a href="http://www.pylonsproject.org/">http://www.pylonsproject.org/</a>) you should start reading carefully the following resources: <br />
<ul>
<li><a href="http://docs.pylonsproject.org/docs/pyramid/en/latest/narr/logging.html">http://docs.pylonsproject.org/docs/pyramid/en/latest/narr/logging.html</a></li>
<li><a href="http://pyramid-exclog.readthedocs.org/en/latest/">http://pyramid-exclog.readthedocs.org/en/latest/</a></li>
</ul>
and once enabled <b>pyramid_exclog</b> check your tween chain order as explained here <a href="http://pyramid-exclog.readthedocs.org/en/latest/#explicit-tween-configuration">http://pyramid-exclog.readthedocs.org/en/latest/#explicit-tween-configuration</a>.<br />
<br />
If you get in trouble some some reason, maybe you'll find this article helpful in some way. Depending on how you serve your application, it might happen that your <i>.ini</i> configuration logging settings are not considered at all.<br />
<br />
Covered arguments in this post:<br />
<ul>
<li>how to log exceptions based on PasteDeploy configuration <b>.ini</b> files</li>
<li>running your app with <b>uwsgi</b></li>
<li>or with <b>CherryPy</b> (see the how to deploy pyramid applications on windows and the <b>Windows service</b> section on <a href="http://pyramid-cookbook.readthedocs.org/en/latest/deployment/windows.html">http://pyramid-cookbook.readthedocs.org/en/latest/deployment/windows.html</a>)</li>
</ul>
<ul>
</ul>
<h3>
How to configure your PasteDeploy .ini file</h3>
In the following example I'm using the <b>handlers.RotatingFileHandler</b> (a python logrotate implementation), but feel free to use <b>FileHandler</b> (without <i>handlers.</i>).<br />
You can configure your <b>production.ini</b> like the following one:<span class="k">
</span><br />
<blockquote class="tr_bq">
<pre><span class="k">...</span>
<span class="k">###</span>
<span class="k"># logging configuration</span>
<span class="k"># <a href="http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/logging.html">http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/logging.html</a></span>
<span class="k">###</span>
<span class="k">[loggers]</span>
<span class="k">keys = root, mip_project<b>, exc_logge</b></span>
<span class="k">[handlers]</span>
<span class="k">keys = console<b>, exc_handle</b></span>
<span class="k">[formatters]</span>
<span class="k">keys = generic<b>, exc_formatter</b></span>
<span class="k">[logger_root]</span>
<span class="k">level = WARN</span>
<span class="k">handlers = console</span>
<span class="k">[logger_mip_project]</span>
<span class="k">level = WARN</span>
<span class="k">handlers =</span>
<span class="k">qualname = mip_project</span>
<span class="k"><b>[logger_exc_logger]</b></span>
<span class="k"><b>level = ERROR</b></span>
<span class="k"><b>handlers = exc_handler</b></span>
<span class="k"><b>qualname = exc_logger</b></span>
<span class="k">[handler_console]</span>
<span class="k">class = StreamHandler</span>
<span class="k">args = (sys.stderr,)</span>
<span class="k">level = NOTSET</span>
<span class="k">formatter = generic</span>
<span class="k"><b>[handler_exc_handler]</b></span>
<span class="k"><b>class = handlers.RotatingFileHandler</b></span>
<span class="k"><b>args = ('exception.log', 'a', 10000000, 10)</b></span>
<span class="k"><b>level = ERROR</b></span>
<span class="k"><b>formatter = exc_formatter</b></span>
<span class="k">[formatter_generic]</span>
<span class="k">format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s</span>
<span class="k"><b>[formatter_exc_formatter]</b></span>
<span class="k"><b>format = %(asctime)s %(message)s</b></span>
<span class="k">[uwsgi]</span>
<span class="k">plugin = python</span>
...
<span class="k">master = true</span>
<span class="k">processes = 1</span>
<span class="k">threads = 4</span>
<span class="k"><b>virtualenv = %d/your_python</b></span>
<span class="k"><b>chdir = %d</b></span>
<span class="k"><b>home = %d/your_python</b></span></pre>
</blockquote>
In the above example it is important omitting <b>%(here)s/</b> in the <i>RotatingFileHandler</i> args, otherwise it won't be resolved running your app with <i>uwsgi</i>.<br />
<h3>
If uwsgi complains about no paste.script.util.logging_config is available</h3>
If your logger configuration is not considered by uwsgi and you see a warning more or less like a missing "<i>paste.script.util.logging_config</i>" module, just install PasteScript in your virtualenv:<br />
<pre><blockquote class="tr_bq">
(your_python) $ pip install PasteScript
</blockquote>
</pre>
And make sure your <i>[uwsgi]</i> section is pointing to your virtualenv (see previous section).<br />
<br />
Anyway, once installed PasteScript (I've tested the 1.7.5 version), all went fine:<br />
<pre><blockquote class="tr_bq">
$ uwsgi --ini-paste-logged production.ini</blockquote>
</pre>
Unfortunately I wasn't able to get exception logs enabled if I call the
above command in a supervisord configuration file. Any suggestions?<br />
<br />
See also:<br />
<ul>
<li><a href="http://lists.unbit.it/pipermail/uwsgi/2012-February/003619.html">http://lists.unbit.it/pipermail/uwsgi/2012-February/003619.html</a></li>
<li><a href="http://lists.unbit.it/pipermail/uwsgi/2012-February/003620.html">http://lists.unbit.it/pipermail/uwsgi/2012-February/003620.html</a> </li>
</ul>
<h3>
Windows service and CherryPy </h3>
If you want to serve a Pyramid application with CherryPy (or creating a CherryPy based Windows service) with exception logging enabled you'll have to setup logging by hand because CherryPy does not consider your logging configuration.<br />
<br />
See also:<br />
<ul>
<li><a href="http://pyramid-cookbook.readthedocs.org/en/latest/deployment/windows.html">http://pyramid-cookbook.readthedocs.org/en/latest/deployment/windows.html</a> </li>
<li><a href="http://davidemoro.blogspot.com/2015/03/pyramid-mysql-windows-good-ugly-bad.html">http://davidemoro.blogspot.com/2015/03/pyramid-mysql-windows-good-ugly-bad.html</a></li>
</ul>
You can follow the above guides and add the bold lines in your <b>pyramidsvc.py</b> in order to have exceptions logged:<br />
<blockquote class="tr_bq">
<pre><span class="kn">from</span> <span class="nn">cherrypy</span> <span class="kn">import</span> <span class="n">wsgiserver</span>
<a href="https://www.blogger.com/null" name="cl-31"></a><b><span class="kn">from</span> <span class="nn">pyramid.paster</span> <span class="kn">import</span> <span class="n">get_app</span>
<a href="https://www.blogger.com/null" name="cl-32"></a><span class="kn">from</span> <span class="nn">pyramid.paster</span> <span class="kn">import</span> <span class="n">setup_logging</span></b>
<a href="https://www.blogger.com/null" name="cl-33"></a><span class="kn">import</span> <span class="nn">os</span>
</pre>
<pre><a href="https://www.blogger.com/null" name="cl-34"></a><span class="k">... </span></pre>
<pre><span class="k"> def</span> <span class="nf">SvcDoRun</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span></pre>
<pre><span class="p"> </span>
<a href="https://www.blogger.com/null" name="cl-35"></a> <span class="n">path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">__file__</span><span class="p">))</span>
<a href="https://www.blogger.com/null" name="cl-36"></a>
<a href="https://www.blogger.com/null" name="cl-37"></a> <span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<a href="https://www.blogger.com/null" name="cl-38"></a>
<a href="https://www.blogger.com/null" name="cl-39"></a> <b><span class="n">app</span> <span class="o">=</span> <span class="n">get_app</span><span class="p">(</span><span class="n">CONFIG_FILE</span><span class="p">)</span>
<a href="https://www.blogger.com/null" name="cl-40"></a> <span class="n">setup_logging</span><span class="p">(</span><span class="n">CONFIG_FILE</span><span class="p">)</span></b></pre>
<pre><span class="p"> .... </span></pre>
</blockquote>
<h3>
Links</h3>
A very interesting overview on Python logging and <b>syslog</b>:<br />
<ul>
<li><a href="https://hynek.me/articles/taking-some-pain-out-of-python-logging/">https://hynek.me/articles/taking-some-pain-out-of-python-logging/</a></li>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-46909871333115864522015-03-24T23:25:00.002+01:002015-04-01T16:04:32.319+02:00Pyramid, MySQL and Windows: the good, the ugly and the bad<a href="http://www.pylonsproject.org/">Pyramid</a>, MySQL and Windows: the <b>good</b> (Pyramid), the <b>ugly</b> and the <b>bad</b>. This title does not fit perfectly the main characters of this blog post because some of theme are both ugly and bad, but it doesn't matter.<br />
<br />
Just in case you are going to set up a Pyramid project based on MySQL and Windows (sometimes you have to)... there are a couple of things useful to know. But let's start with a chronologically order.<br />
<h3>
Day 0 - morning</h3>
You feel like the brave Steeve McQueen:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPHCMW9YnA9NJoboXjsMBfGKsqsEngh8oDoFrxGkyCQIp7tsSEoTk9QyHPmFS51xZIsCERzaBsJ3KVGNk63f_sVocAAGRNYR645NoNUIjs4dS_Vbt7iD6f0wXNCzJmzTI_L5hLAzVi/s1600/papillon1.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPHCMW9YnA9NJoboXjsMBfGKsqsEngh8oDoFrxGkyCQIp7tsSEoTk9QyHPmFS51xZIsCERzaBsJ3KVGNk63f_sVocAAGRNYR645NoNUIjs4dS_Vbt7iD6f0wXNCzJmzTI_L5hLAzVi/s1600/papillon1.jpeg" /></a></div>
<h3>
Evening - day 0</h3>
At the end of the day you'll feel also like Steeve McQueen, but a little more proved: <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGgyoIE12y10tKyw0dpr7nhaKzpyZnLxy-LOF0mCstCn23Pa0LL4wiatXuduYC0YJ7hLC3S7B2L9UNbvGW7sD1KB4riGKataTXvCDutmq3R04REfxECrO5iUIKNs7lbre2Qx8zqvuv/s1600/papillon2.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGgyoIE12y10tKyw0dpr7nhaKzpyZnLxy-LOF0mCstCn23Pa0LL4wiatXuduYC0YJ7hLC3S7B2L9UNbvGW7sD1KB4riGKataTXvCDutmq3R04REfxECrO5iUIKNs7lbre2Qx8zqvuv/s1600/papillon2.jpeg" height="135" width="320" /></a></div>
<br />
<h3>
What happened?</h3>
Random problems with normal transactions considered too large, thread disconnected, error log entries like:<br />
<ul class="simple">
<li>InterfaceError</li>
<li>OperationalError</li>
<li>MySQL Server has gone away</li>
<li>database connection failure</li>
<li>TimeoutError: QueuePool limit of size ... overflow ... reached,
connection timed out, timeout ...</li>
<li>pyramid process hangs </li>
<li>etc</li>
</ul>
<h3>
The solution</h3>
<br />
1 - adjust your <b>my.ini</b> options like that:<br />
<blockquote class="tr_bq">
<pre class="literal-block">[mysqld]
<b>max_allowed_packet = 64MB</b> # adjust this parameter according to your situation
<b>wait_timeout = 28800
interactive_timeout = 2880</b> </pre>
</blockquote>
2 - be sure your <b>production.ini</b> file looks like the following one (with <b>Python 2</b>):<br />
<blockquote class="tr_bq">
sqlalchemy.url = <b>mysql+mysqldb</b>://USER:PASSWORD@127.0.0.1:3306/YOURDB<b>?charset=utf8&use_unicode=0</b><br />
# For Mysql "MySQL Connection Timeout and SQLAlchemy Connection Pool Recycling" issues see:<br />
# <a href="http://docs.sqlalchemy.org/en/latest/core/pooling.html#setting-pool-recycle">http://docs.sqlalchemy.org/en/latest/core/pooling.html#setting-pool-recycle</a><br />
# <a href="http://douglatornell.ca/blog/2012/01/08/staying-alive/">http://douglatornell.ca/blog/2012/01/08/staying-alive/</a><br />
<b>sqlalchemy.pool_recycle = 3600</b>
</blockquote>
3 - you can schedule a restart of your application once a day.<br />
<br />
4 - <i>[OPTIONAL, not only Windows related]</i> adjust your SqlAlchemy configuration parameters according to how many threads your server runs. For example (<b>production.ini</b>):<br />
<blockquote class="tr_bq">
<pre><span class="na">sqlalchemy.pool_size</span> <span class="o">=</span> <span class="s">20</span>
<a href="https://www.blogger.com/null" name="cl-23"></a><span class="na">sqlalchemy.max_overflow</span> <span class="o">=</span> <span class="s">10</span></pre>
</blockquote>
5 - if you are using CherryPy as a Windows service, be sure your <span class="s"> '<i>engine.autoreload.on</i>'</span> option is set to <b>False</b>. <br />
<br />
<br />
6 - <b>[UPDATE20150401]</b> check your sql-mode if you want to prevent not handled ProxyError exceptions due to input longer than the max size provided on the field model. See <a href="https://github.com/Kotti/Kotti/issues/404">https://github.com/Kotti/Kotti/issues/404</a><br />
<br />
<br />
<h3>
Results </h3>
<br />
No more exceptions or odd behaviours!<br />
<h3>
Links</h3>
<ul>
<li><a href="http://docs.sqlalchemy.org/en/rel_0_9/dialects/mysql.html#unicode-encoding-decoding">http://docs.sqlalchemy.org/en/rel_0_9/dialects/mysql.html#unicode-encoding-decoding</a></li>
<li><a href="http://docs.sqlalchemy.org/en/rel_0_9/dialects/mysql.html#module-sqlalchemy.dialects.mysql.mysqldb">http://docs.sqlalchemy.org/en/rel_0_9/dialects/mysql.html#module-sqlalchemy.dialects.mysql.mysqldb</a></li>
<li><a href="http://docs.sqlalchemy.org/en/latest/core/pooling.html#setting-pool-recycle">http://docs.sqlalchemy.org/en/latest/core/pooling.html#setting-pool-recycle</a></li>
<li><a href="http://douglatornell.ca/blog/2012/01/08/staying-alive/">http://douglatornell.ca/blog/2012/01/08/staying-alive/</a> </li>
</ul>
<ul>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-28166265387435211712015-03-17T23:11:00.003+01:002015-05-29T23:58:12.428+02:00Kotti CMS events - insert subobjects automaticallyYet another small recipe for <b>Kotti CMS</b>: how to initialize automatically a new object once inserted with <b>events</b>, for example adding a subobject.<br />
<br />
Use case? When someone creates a <i>UserProfile</i> object <b>/users/name-surname</b>, an event should create automatically a profile image in <b>/users/name-surname/photo</b> (a Kotti image instance).<br />
<br />
It is quite simple. Let's see our <b>your_package/events.py</b> module, where <i>IMAGE_ID</i> is equals to <i>'photo'</i>:<br />
<blockquote class="tr_bq">
<pre>import os
from kotti.events import (
ObjectInsert,
subscribe,
notify,
)
from kotti.resources import Image
from your_package.resources import UserProfile
from your_package.config import IMAGE_ID
<b>@subscribe(ObjectInsert, UserProfile)</b>
def user_profile_added(event):
obj = event.object
if <b>IMAGE_ID not in obj.keys()</b>:
image_path = <b>os.path.join</b>(
os.path.dirname(__file__),
'data', 'fallback.png'
)
with open(image_path, <b>'rb'</b>) as image_file:
obj[IMAGE_ID] = image_obj = Image(
title='Image',
in_navigation=False,
data=image_file.read(),
filename=u'fallback.png',
mimetype=u'image/png',
)
<b>notify(ObjectInsert(image_obj, event.request))</b> </pre>
</blockquote>
Notes:<br />
<ol>
<li>the subscribe decorator will register our handler when a <i>UserProfile</i> resource will be inserted</li>
<li>we should check if <i>IMAGE_ID</i> is already instanciated (prevent errors on paste) </li>
<li>if you want your code will work both for Unix-like or under Windows, use <b>os.path.join</b> instead of a plain <i>data/fallback.png</i> path (path separator issues)</li>
<li>the final <b>b</b> in the <i>open</i> is important if you want to write code that works under Windows, see <a href="https://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files">https://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files</a>. On Unix, it doesn’t hurt to append a 'b' to the mode, so you can use it platform-independently for all binary files</li>
<li>notify the image insertion</li>
</ol>
And <b>your_package/__init__.py</b> (in this example I've used the <b>scan</b> method but you could also register your event handlers imperatively): <br />
<blockquote class="tr_bq">
<pre>...
def includeme(config):
""" Don't add this to your ``pyramid_includes``, but add the
``kotti_configure`` above to your ``kotti.configurators`` instead.
:param config: Pyramid configurator object.
:type config: :class:`pyramid.config.Configurator`
"""
...
<b>config.scan(__name__)</b></pre>
</blockquote>
And... tests of course (<b>your_package/tests/test_events.py</b>):<br />
<blockquote class="tr_bq">
<pre>from pytest import fixture
from kotti.testing import DummyRequest
@fixture
def user_profile(db_session, config, root):
""" returns dummy UserProfile.
"""
<b>config.include('your_package')</b>
from your_package.resources import UserProfile
root['userprofile'] = userprofile = UserProfile(title='UserProfile')
from kotti.events import notify
from kotti.events import ObjectInsert
<b>notify(ObjectInsert(course, DummyRequest()))</b> return userprofile
class TestUserProfileWithEvents:
def test_assert_userprofile_image(self, user_profile):
from your_package.config import IMAGE_ID
assert IMAGE_ID in user_profile.keys()
... </pre>
</blockquote>
You can test also if everything goes right after a copy/paste action (see the Kotti's tests).<br />
<br />
Done!<br />
<h2>
All posts about Kotti</h2>
<ul>
<li><a href="http://davidemoro.blogspot.com/2015/03/pyramid-mysql-windows-good-ugly-bad.html">Pyramid, MySQL and Windows: the good, the ugly and the bad</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/03/kotti-cms-events-insert-subobjects.html">Kotti CMS events - insert subobject automatically</a> </li>
<li><a href="http://davidemoro.blogspot.it/2015/02/kotti-cms-intranet.html">Kotti CMS - how to turn your Kotti CMS into an intranet</a> </li>
<li><a href="http://davidemoro.blogspot.it/2015/02/kotti-cms-annotations-store-arbitrary-data.html">Kotti CMS - how to store arbitrary data with annotations </a></li>
<li><a href="http://davidemoro.blogspot.com/2015/01/how-to-install-kotti-cms-on-windows.html">How to install Kotti CMS on Windows</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/01/kotti-avoid-types-addable-in-content.html">Kotti CMS - avoid types addable in content root</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-how-to-create-new-content-image.html">Kotti CMS - how to create a new content type with an image</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-workflow-reference.html">Kotti CMS - workflow reference</a></li>
</ul>
<h3>
All Kotti posts published by <a href="https://twitter.com/davidemoro">@davidemoro</a>:<br />
</h3>
<ul>
<li><a href="http://davidemoro.blogspot.it/search/label/kotti">http://davidemoro.blogspot.it/search/label/kotti </a></li>
</ul>
<br />
<ul>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-11352348706821139772015-02-28T00:15:00.000+01:002015-06-23T17:44:57.830+02:00Kotti CMS - how to turn your Kotti CMS into an intranetIn the previous posts we have seen that Kotti is a minimal but robust high-level Pythonic web application framework based on Pyramid that includes an <b>extensible</b> CMS solution, both user and developer friendly. For developer friendly I mean that you can be productive in one or two days without any knowledge of Kotti or Pyramid if you already know the Python language programming.<br />
<br />
If you have to work relational databases, hierarchical data, workflows or complex security requirements Kotti is your friend. It uses well know Python libraries.<br />
<br />
In this post we'll try to turn our Kotti CMS public site into a <b>private intranet/extranet</b> service.<br />
<br />
I know, there are other solutions keen on building intranet or collaboration portals like <b>Plone</b> (I've been working 8 years on large and complex intranets, big public
administration customers with thousands of active users and several
editor teams, multiple migrations, etc) or the <b>KARL</b> project. But let's pretend that in our use case we have simpler requirements and we don't need too complex solutions, features like communities, email subscriptions or similar things.<br />
<br />
Thanks to the Pyramid and Kotti's architectural design, you can turn your public website into an intranet without having to fork the Kotti code: <b>no forks</b>!<br />
<h3>
How to turn your site into an intranet</h3>
This could be an hard task if you use other CMS solutions, but with Kotti (or the heavier Plone) it will requires you just 4 steps:<br />
<ol>
<li>define a custom intranet workflow </li>
<li>apply your custom worklows to images and files (by default they are not
associated to any workflow, so once added they are immediatly public) </li>
<li>set a default fallback permission for all views</li>
<li>override the default root ACL (populators)</li>
</ol>
<h4>
1 - define a custom intranet workflow</h4>
Intranet workflows maybe different depending on your organization requirements. It might be very simple or with multiple review steps.<br />
<br />
The important thing is: no more granting the view permission for anonymous users, unless you are willing to define an externally published state <br />
<br />
With Kotti you can design your workflow just editing an xml file. For further information you can follow the <a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-workflow-reference.html">Kotti CMS - workflow reference</a> article. <br />
<h4>
2 - apply your custom workflow to images and files</h4>
By default they are not
associated to any workflow, so once added they are immediately public. <br />
<br />
This step will requires you just two additional lines of code in your includeme or kotti_configure function.<br />
<br />
Already described here: <a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-workflow-reference.html">Kotti CMS - workflow reference</a>, see the "<i>How to enable the custom workflow for images and files</i>" section.<br />
<h4>
3 - set a default fallback permission</h4>
In your <i>includeme</i> function you just need to tell the configurator to set a default permission even for public views already registered.<br />
<br />
I mean that if somewhere into the Kotti code there is any callable view not associated to a permission, it won't be accessible by anonymous after this step.<br />
<br />
In your includeme function you'll need to :<br />
<blockquote class="tr_bq">
def includeme(config):<br />
...<br />
# set a default permission even for public views already registered<br />
# without permission<br />
<b>config.set_default_permission('view')</b> </blockquote>
If you want to bypass the default permission for certain views, you can decorate them with a special permission (<b>NO_PERMISSION_REQUIRED</b> from pyramid.security) which indicates that the view should always be executable by entirely anonymous users, regardless of the default permission. See:<br />
<ul>
<li><a href="http://docs.pylonsproject.org/docs/pyramid/en/latest/api/security.html#pyramid.security.NO_PERMISSION_REQUIRED">http://docs.pylonsproject.org/docs/pyramid/en/latest/api/security.html#pyramid.security.NO_PERMISSION_REQUIRED</a> </li>
</ul>
<h3>
4 - override the default root ACL (populators)</h3>
The default Kotti's ACL associated with the root of the site<br />
<blockquote class="tr_bq">
from kotti.security import SITE_ACL</blockquote>
gives <i>view</i> privileges to every user, including anonymous.<br />
You can override this configuration to require users to log in before they can view any of your site's pages. To achieve this, you'll have to set your site's ACL as shown on the following url:<br />
<ul>
<li><a href="http://kotti.readthedocs.org/en/latest/developing/advanced/close-for-anonymous.html">http://kotti.readthedocs.org/en/latest/developing/advanced/close-for-anonymous.html</a></li>
</ul>
You'll need you add or override the default populator. See the kotti.populators options here:<br />
<ul>
<li><a href="http://kotti.readthedocs.org/en/latest/developing/basic/configuration.html#kotti-populators">http://kotti.readthedocs.org/en/latest/developing/basic/configuration.html#kotti-populators</a></li>
</ul>
<h3>
Results</h3>
After reading this article you should be able to close your Kotti site for anonymous users and obtaining a simple, private intranet-like area.<br />
<br />
Off-topic: you can also use Kotti as a content backend-only administration area for public websites, with a complete decoupled frontend solution.<br />
<br />
<b>UPDATE 20150623</b>: now you can achieve the same goals described in this article installing <b>kotti_backend</b>. See <a href="https://github.com/Kotti/kotti_backend">https://github.com/Kotti/kotti_backend</a><br />
<h3>
Useful links</h3>
<ul>
<li><a href="http://plone.org/">http://plone.org</a></li>
<li><a href="https://github.com/ploneintranet/ploneintranet.suite">https://github.com/ploneintranet/ploneintranet.suite</a></li>
<li><a href="http://en.wikipedia.org/wiki/KARL_project">http://en.wikipedia.org/wiki/KARL_project</a> </li>
<li><a href="http://karlproject.org/">karlproject.org</a> (20150228, offline)</li>
<li><a href="http://www.sixfeetup.com/app-development/karl-intranet">http://www.sixfeetup.com/app-development/karl-intranet</a></li>
<li><a href="http://www.sixfeetup.com/app-development/karl-intranet/karl-enhancements">http://www.sixfeetup.com/app-development/karl-intranet/karl-enhancements</a></li>
</ul>
<h2>
All posts about Kotti</h2>
<ul>
<li><a href="http://davidemoro.blogspot.com/2015/03/pyramid-mysql-windows-good-ugly-bad.html">Pyramid, MySQL and Windows: the good, the ugly and the bad</a> </li>
<li><a href="http://davidemoro.blogspot.com/2015/03/kotti-cms-events-insert-subobjects.html">Kotti CMS events - insert subobject automatically</a> </li>
<li><a href="http://davidemoro.blogspot.it/2015/02/kotti-cms-intranet.html">Kotti CMS - how to turn your Kotti CMS into an intranet</a> </li>
<li><a href="http://davidemoro.blogspot.it/2015/02/kotti-cms-annotations-store-arbitrary-data.html">Kotti CMS - how to store arbitrary data with annotations </a></li>
<li><a href="http://davidemoro.blogspot.com/2015/01/how-to-install-kotti-cms-on-windows.html">How to install Kotti CMS on Windows</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/01/kotti-avoid-types-addable-in-content.html">Kotti CMS - avoid types addable in content root</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-how-to-create-new-content-image.html">Kotti CMS - how to create a new content type with an image</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-workflow-reference.html">Kotti CMS - workflow reference</a></li>
</ul>
<h3>
All Kotti posts published by <a href="https://twitter.com/davidemoro">@davidemoro</a>:<br />
</h3>
<ul>
<li><a href="http://davidemoro.blogspot.it/search/label/kotti">http://davidemoro.blogspot.it/search/label/kotti </a></li>
</ul>
<br />
<ul>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-74598987063183383332015-02-24T23:15:00.001+01:002015-05-29T23:59:01.702+02:00Kotti CMS - how to store arbitrary data with annotationsWith <b>Kotti CMS</b> you can extend existing types inheriting from a base class (eg: Document) and obtain another type of object (eg: MyDocument) with new fields, new workflows, custom views, custom addability conditions, etc.<br />
<br />
But sometimes you may want to add a custom field to one or more resources, without having to create a new type. For example you might want to add a colour attribute to all existing Document objects, let's imagine a simple select widget with few colours that will be used for adding a class depending on the choosen colour.<br />
<br />
By default Kotti is shipped with an <b>annotations</b> column that can be used to store arbitrary data in a nested dictionary.<br />
<br />
You can store arbitrary data in the nested dictionary with a syntax similar to the following one:<br />
<blockquote class="tr_bq">
context.annotations['SOMEKEY'] = VALUE</blockquote>
and read annotations with:<br />
<blockquote class="tr_bq">
context.annotations</blockquote>
All you need to do is <b>overriding</b> the add and edit form of your target class. With Pyramid is quite easy to extending an existing application and override views, assets, routes, etc. See <a href="http://docs.pylonsproject.org/docs/pyramid/en/latest/narr/extending.html">http://docs.pylonsproject.org/docs/pyramid/en/latest/narr/extending.html</a> for further info.<br />
<br />
Here you can see one possible implementation:<br />
<blockquote class="tr_bq">
from pyramid.view import view_config<br />
import colander<br />
from deform.widget import SelectWidget <br />
from kotti_actions.views.edit.actions.link import (<br />
LinkActionAddForm as OriginalLinkActionAddForm,<br />
LinkActionEditForm as OriginalLinkActionEditForm,<br />
)<br />
from kotti_actions.resources import (<br />
LinkAction<br />
)<br />
...</blockquote>
colours = [<br />
('', 'Select'),<br />
('red', 'Red'),<br />
('brown', 'Brown'),<br />
('beige', 'Beige'),<br />
('blue', 'Blue'),<br />
]<br />
<blockquote class="tr_bq">
<br />
def add_colour(schema):<br />
schema['colour'] = colander.SchemaNode(<br />
colander.String(),<br />
title=_('Colour'),<br />
widget=SelectWidget(values=colours),<br />
missing=u"",<br />
)<br />
<br />
@view_config(name=LinkAction.type_info.add_view, permission='add',<br />
renderer='kotti:templates/edit/node.pt')<br />
class LinkActionAddForm(OriginalLinkActionAddForm):<br />
""" Form to add a new instance of CustomContent. """<br />
<br />
def schema_factory(self):<br />
schema = super(LinkActionAddForm, self).schema_factory()<br />
add_colour(schema)<br />
return schema<br />
<br />
def add(self, **appstruct):<br />
colour = u''<br />
try:<br />
colour = appstruct.pop('colour')<br />
except KeyError:<br />
pass<br />
obj = super(LinkActionAddForm, self).add(**appstruct)<br />
<br />
obj.annotations['colour'] = colour<br />
return obj<br />
<br />
<br />
@view_config(name='edit', context=LinkAction, permission='edit',<br />
renderer='kotti:templates/edit/node.pt')<br />
class LinkActionEditForm(OriginalLinkActionEditForm):<br />
""" Form to edit existing calendars. """<br />
<br />
def schema_factory(self):<br />
schema = super(LinkActionEditForm, self).schema_factory()<br />
add_colour(schema)<br />
return schema<br />
<br />
def before(self, form):<br />
super(LinkActionEditForm, self).before(form)<br />
colour = self.context.annotations.get('colour')<br />
if colour:<br />
form.appstruct.update({'colour': colour})<br />
<br />
def edit(self, **appstruct):<br />
super(LinkActionEditForm, self).edit(**appstruct)<br />
self.context.annotations['colour'] = appstruct['colour']</blockquote>
<br />
Now our LinkAction add and edit form will have an additional select with our colours.<br />
<h2>
All posts about Kotti</h2>
<ul>
<li><a href="http://davidemoro.blogspot.com/2015/03/pyramid-mysql-windows-good-ugly-bad.html">Pyramid, MySQL and Windows: the good, the ugly and the bad</a> </li>
<li><a href="http://davidemoro.blogspot.com/2015/03/kotti-cms-events-insert-subobjects.html">Kotti CMS events - insert subobject automatically</a> </li>
<li><a href="http://davidemoro.blogspot.it/2015/02/kotti-cms-intranet.html">Kotti CMS - how to turn your Kotti CMS into an intranet</a> </li>
<li><a href="http://davidemoro.blogspot.it/2015/02/kotti-cms-annotations-store-arbitrary-data.html">Kotti CMS - how to store arbitrary data with annotations </a></li>
<li><a href="http://davidemoro.blogspot.com/2015/01/how-to-install-kotti-cms-on-windows.html">How to install Kotti CMS on Windows</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/01/kotti-avoid-types-addable-in-content.html">Kotti CMS - avoid types addable in content root</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-how-to-create-new-content-image.html">Kotti CMS - how to create a new content type with an image</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-workflow-reference.html">Kotti CMS - workflow reference</a></li>
</ul>
<h3>
All Kotti posts published by <a href="https://twitter.com/davidemoro">@davidemoro</a>:<br />
</h3>
<ul>
<li><a href="http://davidemoro.blogspot.it/search/label/kotti">http://davidemoro.blogspot.it/search/label/kotti </a></li>
</ul>
<br />
<ul>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-490978748810565282015-02-11T23:47:00.002+01:002015-07-01T00:05:15.433+02:00Python mock library for ninja testingIf you are going to write <b>unit tests</b> with Python you should consider this library: <b>Python mock</b> (<a href="https://pypi.python.org/pypi/mock">https://pypi.python.org/pypi/mock</a>).<br />
<br />
Powerful, elegant, easy, documented (<a href="http://www.voidspace.org.uk/python/mock/">http://www.voidspace.org.uk/python/mock/</a>)...<br />
and <b>standard</b>: <i>mock</i> is now part of the Python standard library, available as <i>unittest.mock</i> in Python 3.3 onwards.<br />
<h3>
Simple example</h3>
Let's suppose you have an existing validator function based on a dbsession import used for querying a relational database. If you are going to write unit tests, you should focus on units without involving real database connections.<br />
<br />
<b>validators.py</b><br />
from yourpackage import DBSession<br />
<br />
def validate_sku(value):<br />
...<br />
courses = DBSession.query(Course).\<br />
filter(Course.course_sku == value).\<br />
filter(Course.language == context.language).\<br />
all()<br />
# validate data returning a True or False value<br />
...<br />
<br />
<b>tests/test_validators.py</b><br />
def test_validator():<br />
import mock<br />
with mock.patch('yourpackage.validators.DBSession') as dbsession:<br />
instance = dbsession<br />
instance.query().filter().filter().all.return_value = [mock.Mock(id=1)]<br />
from yourpackage.validators import sku_validator<br />
assert sku_validator(2) is True<br />
<br />
In this case the DBSession call with query, the two filter calls and the final all invocation will produce our mock result (a list of with one mock item, an object with an id attribute).<br />
<br />
<b>Brilliant!</b> And this is just one simple example: check out the official documentation for further information:<br />
<ul>
<li><a href="http://www.voidspace.org.uk/python/mock/">http://www.voidspace.org.uk/python/mock/</a> </li>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0tag:blogger.com,1999:blog-4821336113846598354.post-1141577238590235042015-02-04T23:08:00.000+01:002015-06-06T23:36:07.599+02:00Kotti CMS - workflow referenceYet another blog post about <b>Kotti CMS</b> (<a href="http://kotti.pylonsproject.org/">http://kotti.pylonsproject.org/</a>): this time I'm going to talk about <b>workflows</b> and security.<br />
<br />
Workflows in Kotti are based on <b>repoze.workflow</b>. See <a href="http://docs.repoze.org/workflow/">http://docs.repoze.org/workflow/</a> for further information. Basically you can use an xml file (zcml) in order to describe your workflow definition. You can see an example here: <a href="https://github.com/Kotti/Kotti/blob/master/kotti/workflow.zcml">https://github.com/Kotti/Kotti/blob/master/kotti/workflow.zcml</a>. A you can see it is quite straightforward adding new states, new transitions, new permissions, etc. You can easily turn your <b>2-states</b> website workflow into a <b>3-states</b> website workflow with reviewers or turn Kotti app into an <b>intranet</b> application.<br />
<br />
The default workflow definition is loaded from your project <i>.ini</i> file settings (using the <i>kotti.use_workflow</i> settings). The kotti.use_workflow setting's default value is:<br />
<blockquote class="tr_bq">
<pre>kotti.use_workflow = kotti:workflow.zcml</pre>
</blockquote>
but can change change default workflow for the whole site, register new workflows related to specific content types or disable it as well. <br />
<br />
Anyway, if you need to write a Python based CMS-ish application with hierarchical contents, custom content types, workflows, security, global and local ACL (sharing permissions), pluggable and extensible, based on relational databases, developer friendly, with a simple UI, etc... Kotti is your friend!<br />
<h3>
How to disable the default workflow</h3>
Kotti is shipped with a simple workflow implementation based on private and public states. If your particular use case does not require workflows at all, you can disable this feature with a non true value. For example:<br />
<blockquote class="tr_bq">
<pre>kotti.use_workflow = 0</pre>
</blockquote>
<h3>
How to override the Kotti's default workflow for all content types</h3>
The default workflow is quite useful for websites, but sometimes you need something of different. Just change your workflow setting and point to your zcml file:<br />
<blockquote class="tr_bq">
<pre>kotti.use_workflow = kotti_yourplugin:workflow.zcml</pre>
</blockquote>
The simplest way to deal with workflow definitions is:<br />
<ul>
<li>create a copy of the default workflow definition</li>
<li>customize it (change permissions, add new states, permissions, transitions, initial state and so on)</li>
</ul>
If your site already has content and you configure it use a workflow for the first time, or you use a different workflow than the one you used before, run the <i>kotti-reset-workflow</i> command to reset all your content's workflow. <br />
<ul>
</ul>
<h3>
How to enable the custom workflow for images and files</h3>
Images and files are not associated with the default workflow. If you need a workflow for these items you need to attach the IDefaultWorkflow marker interface.<br />
<br />
You can add the following lines in your includeme function:<br />
<blockquote class="tr_bq">
<pre>from zope.interface import implementer
from kotti.interfaces import IDefaultWorkflow
from kotti.resources import File
from kotti.resources import Image
...
def includeme(config):
...
# enable workflow for images and files
implementer(IDefaultWorkflow)(Image)
implementer(IDefaultWorkflow)(File)
...</pre>
</blockquote>
<h3>
How to assign a different workflow to a content type</h3>
[<b><i>UPDATE 20150604:</i></b><br />
We are going to use the default workflow for standard content types and a custom workflow for content providing the <i>IBoxWorkflow</i> marker interface. The custom workflow is configurable via <i>.ini</i> configuration files.<br />
Note well: the elector, the <i>config.begin/config.commit</i> and the <i>load_zcml</i> can be omitted. I'll update this section soon.<br />
<br />
<b>UPDATE 20150606:</b><br />
Confirmed: things can be done in a simpler way! So no <i>load_zcml</i>, no <i>config.begin</i>/<i>config.commit</i>, no elector.<br />
See the following video:<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/sUZGkWuj_2E" width="560"></iframe>
]<br />
<br />
In this kind of situation you want to use the default workflow for all your types and a different workflow implementation for a particular content type.<br />
<br />
You'll need to:<br />
<ul>
<li>create the new workflow definition, with a workflow elector</li>
<li>write an elector function that will returns True or False depending if the workflow should be applied (otherwise will win the default default workflow, or better, the first matching workflow without an elector)</li>
<li>load manually your zcml file in your <i>includeme</i> function</li>
</ul>
<b>.ini file (optional)</b><br />
<blockquote class="tr_bq">
<pre><span class="pl-s1"><span class="pl-pds"></span>kotti_boxes.use_workflow = kotti_boxes:workflow.zcml<span class="pl-pds"></span></span></pre>
</blockquote>
<b>__init__.py</b> <br />
<blockquote>
<pre>from pyramid.i18n import TranslationStringFactory
from kotti import FALSE_VALUES
def includeme(config):
...
workflow = config.registry.settings.get('kotti_boxes.use_workflow', None)
if workflow and workflow.lower() not in FALSE_VALUES:
config.begin()
config.hook_zca()
config.include('pyramid_zcml')
<b>config.load_zcml(workflow)</b>
config.commit()
...</pre>
</blockquote>
<b>workflow.py</b><br />
From the repoze.workflow documentation: """A workflow is unique in a system using multiple workflows if the combination of its type, its content type, its elector, and its state_attr are different than the combination of those attributes configured in any other workflow."""<br />
Depending on how specific is your combination you may need to implement an elector (a function that returns True or False for a given context).<br />
<blockquote class="tr_bq">
<pre>from kotti_boxes.interfaces import IBoxWorkflow
def elector(context):
return IBoxWorkflow.providedBy(context)</pre>
</blockquote>
<b>workflow.zcml</b><br />
<blockquote class="tr_bq">
<pre><configure xmlns="http://namespaces.repoze.org/bfg"
xmlns:i18n="http://xml.zope.org/namespaces/i18n"
i18n:domain="Kotti">
<include package="repoze.workflow" file="meta.zcml"/>
<workflow
type="security"
name="simple"
state_attr="state"
initial_state="private"
<b>content_types="kotti_boxes.interfaces.IBoxWorkflow"</b>
<b>elector='kotti_boxes.workflow.elector'</b>
permission_checker="pyramid.security.has_permission"
>
<state name="private" callback="kotti.workflow.workflow_callback">
<key name="title" value="_(u'Private')" />
<key name="order" value="1" />
<key name="inherit" value="0" />
<key name="system.Everyone" value="" />
<key name="role:viewer" value="viewbox view" />
<key name="role:editor" value="viewbox view add edit delete state_change" />
<key name="role:owner" value="viewbox view add edit delete manage state_change" />
</state>
...
<transition
name="private_to_public"
from_state="private"
to_state="public"
permission="state_change" />
...
</workflow>
</configure></pre>
</blockquote>
<h2>
All posts about Kotti</h2>
<ul>
<li><a href="http://davidemoro.blogspot.com/2015/03/pyramid-mysql-windows-good-ugly-bad.html">Pyramid, MySQL and Windows: the good, the ugly and the bad</a> </li>
<li><a href="http://davidemoro.blogspot.com/2015/03/kotti-cms-events-insert-subobjects.html">Kotti CMS events - insert subobject automatically</a> </li>
<li><a href="http://davidemoro.blogspot.it/2015/02/kotti-cms-intranet.html">Kotti CMS - how to turn your Kotti CMS into an intranet</a> </li>
<li><a href="http://davidemoro.blogspot.it/2015/02/kotti-cms-annotations-store-arbitrary-data.html">Kotti CMS - how to store arbitrary data with annotations</a> </li>
<li><a href="http://davidemoro.blogspot.com/2015/01/how-to-install-kotti-cms-on-windows.html">How to install Kotti CMS on Windows</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/01/kotti-avoid-types-addable-in-content.html">Kotti CMS - avoid types addable in content root</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-how-to-create-new-content-image.html">Kotti CMS - how to create a new content type with an image</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-workflow-reference.html">Kotti CMS - workflow reference</a></li>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com3tag:blogger.com,1999:blog-4821336113846598354.post-2347381019658291852015-02-03T23:20:00.000+01:002015-05-29T23:59:21.289+02:00Kotti CMS - how to create a new content type with an imageIf you want to create a new content type based on an existing one with <a href="http://kotti.pylonsproject.org/">Kotti</a> you need to write few lines of code and zero html for the add and edit views: it is very simple (browse Kotti's <i>resources.py</i> and <i>views</i> code).<br />
<br />
<br />
Basically you have to extend the existing content type shipped with Kotti and add your custom fields.<br />
<br />
But let's suppose you need a new content type named <b>ImageWithLink</b> with the following fields:<br />
<ul>
<li>title</li>
<li>description</li>
<li>image</li>
<li>link </li>
</ul>
In this case the implementation is more verbose compared to extend another content type (like the Document, but it is still an easy job).<br />
<br />
<b>resources.py</b><br />
<blockquote class="tr_bq">
from zope.interface import implements<br />
from kotti.resources import Image<br />
from kotti.interfaces import IImage<br />
from sqlalchemy import Column<br />
from sqlalchemy import ForeignKey<br />
from sqlalchemy import Integer<br />
from sqlalchemy import Unicode<br />
<br />
<br />
class ImageWithLink(Image):<br />
implements(IImage)<br />
<br />
id = Column(Integer, ForeignKey('images.id'), primary_key=True)<br />
link = Column(Unicode(1000))<br />
<br />
type_info = Image.type_info.copy(<br />
name=u'ImageWithLink',<br />
title=u'ImageWithLink',<br />
add_view=u'add_image_link',<br />
addable_to=['Document'],<br />
)<br />
<br />
def __init__(self, link=u"", **kwargs):<br />
super(ImageWithLink, self).__init__(**kwargs)<br />
self.link = link </blockquote>
The code is quite self-explaining: you create a new ImageWithLink class that inherits from Image. You only need to add your custom field named <i>link</i> and you initialize the link in the __init__ code after calling the super method.<br />
<br />
<b>views/content.py</b><br />
<blockquote class="tr_bq">
import colander<br />
from deform import FileData<br />
from deform.widget import FileUploadWidget<br />
from kotti.views.edit import ContentSchema<br />
from kotti.views.edit.content import ImageEditForm<br />
from kotti.views.edit.content import ImageAddForm<br />
from kotti.views.form import validate_file_size_limit<br />
from kotti.views.form import FileUploadTempStore<br />
from kotti.views.form import AddFormView<br />
from pyramid.view import view_config<br />
from kotti_yourplugin import _<br />
from kotti_yourplugin.resources import ImageWithLink<br />
from kotti_yourplugin.validators import link_validator<br />
<br />
<br />
def ImageWithLinkSchema(tmpstore):<br />
""" File schema with no set title missing binding """<br />
class ImageWithLinkSchema(ContentSchema):<br />
file = colander.SchemaNode(<br />
FileData(),<br />
title=_(u'File'),<br />
widget=FileUploadWidget(tmpstore),<br />
validator=validate_file_size_limit,<br />
)<br />
link = colander.SchemaNode(<br />
colander.String(),<br />
title=_('Link'),<br />
validator=link_validator,<br />
missing=u'',<br />
)<br />
<br />
def after_bind(node, kw):<br />
del node['tags']<br />
<br />
return ImageWithLinkSchema(after_bind=after_bind)<br />
<br />
<br />
@view_config(name='edit', permission='edit',<br />
renderer='kotti:templates/edit/node.pt')<br />
class ImageWithLinkEditForm(ImageEditForm):<br />
def schema_factory(self):<br />
tmpstore = FileUploadTempStore(self.request)<br />
return ImageWithLinkSchema(tmpstore)<br />
<br />
<br />
@view_config(name=ImageWithLink.type_info.add_view, permission='add',<br />
renderer='kotti:templates/edit/node.pt')<br />
class ImageWithLinkAddForm(ImageAddForm):<br />
item_type = _(u"Banner Box")<br />
item_class = ImageWithLink<br />
<br />
def schema_factory(self):<br />
tmpstore = FileUploadTempStore(self.request)<br />
return ImageWithLinkSchema(tmpstore)<br />
<br />
def save_success(self, appstruct):<br />
# override this method (no filename as title<br />
# like images)<br />
return AddFormView.save_success(self, appstruct)<br />
<br />
def add(self, **appstruct):<br />
# override (no tags in our form)<br />
buf = appstruct['file']['fp'].read()<br />
filename = appstruct['file']['filename']<br />
return self.item_class(<br />
title=appstruct['title'] or filename,<br />
description=appstruct['description'],<br />
data=buf,<br />
filename=filename,<br />
mimetype=appstruct['file']['mimetype'],<br />
size=len(buf),<br />
)</blockquote>
Here the code is more complex. There is a dynamic schema definition with the Kotti's temp store implementation. Both the add and the edit form refer to this schema, with some overrides because our object does not behave like files or images.<br />
<br />
<b>validators.py</b><br />
<b>UPDATE 20150211: </b>no need to write this validator. Use the url validator provided by colander instead (<a href="http://docs.pylonsproject.org/projects/colander/en/latest/api.html#colander.url">colander.url</a>). Anyway you can use all the builtin colander validators or write your own validators.<b><br /></b><br />
<blockquote class="tr_bq">
import re<br />
import colander<br />
from kotti_yourplugin import _<br />
<br />
<br />
VALID_PROTOCOLS = ('http',)<br />
URL_REGEXP = r'(%s)s?://[^\s\r\n]+' % '|'.join(VALID_PROTOCOLS)<br />
<br />
<br />
def link_validator(node, value):<br />
""" Raise a colander.Invalid exception if the provided url<br />
is not valid<br />
"""<br />
def raise_invalid_url(node, value):<br />
raise colander.Invalid(<br />
node, _(u"You must provide a valid url."))<br />
if value:<br />
if not re.match(URL_REGEXP, value):<br />
raise_invalid_url(node, value)</blockquote>
Here you can see an example of link validator based on a regular expression. This validator decorates our link field of the ImageWithLink schema.<br />
<br />
Obviously you need to add in your kotti_configure method your ImageWithLink in the <b>kotti.available_types</b> settings.<br />
<b><br /></b>
<b>__init__.py</b><br />
<blockquote class="tr_bq">
def kotti_configure(settings):<br />
settings['pyramid.includes'] += ' kotti_yourplugin'<br />
settings['kotti.available_types'] += ' kotti_yourplugin.resources.ImageWithLink' </blockquote>
<br />
and enable your configurator in your <b>.ini</b> file:<br />
<blockquote class="tr_bq">
kotti.configurators = mip_course.kotti_configure</blockquote>
<br />
And what about the default view of your content types? If you visit an ImageWithLink box it will behave like an image: it inherits the default view of the image (you should customize it adding the link on the image, very simple: not showed in this blog post), no need to deal with the image resize machinery, etc. <br />
<br />
As you can see, Kotti is a flexible solution if you need a simple but powerful CMS solution based on Python, Pyramid and SQLAlchemy. You may consider it as a simple framework (but easy to understand, don't be scared by the word framework. It is really developer friendly). If you are curious about how to manage contents with Kotti you may play with the <b>demo online</b>: <a href="http://kottidemo.danielnouri.org/">http://kottidemo.danielnouri.org/</a> (admin - qwerty).<br />
<h2>
All posts about Kotti</h2>
<ul>
<li><a href="http://davidemoro.blogspot.com/2015/03/pyramid-mysql-windows-good-ugly-bad.html">Pyramid, MySQL and Windows: the good, the ugly and the bad</a> </li>
<li><a href="http://davidemoro.blogspot.com/2015/03/kotti-cms-events-insert-subobjects.html">Kotti CMS events - insert subobject automatically</a> </li>
<li><a href="http://davidemoro.blogspot.it/2015/02/kotti-cms-intranet.html">Kotti CMS - how to turn your Kotti CMS into an intranet</a> </li>
<li><a href="http://davidemoro.blogspot.it/2015/02/kotti-cms-annotations-store-arbitrary-data.html">Kotti CMS - how to store arbitrary data with annotations</a> </li>
<li><a href="http://davidemoro.blogspot.com/2015/01/how-to-install-kotti-cms-on-windows.html">How to install Kotti CMS on Windows</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/01/kotti-avoid-types-addable-in-content.html">Kotti CMS - avoid types addable in content root</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-how-to-create-new-content-image.html">Kotti CMS - how to create a new content type with an image</a></li>
<li><a href="http://davidemoro.blogspot.com/2015/02/kotti-cms-workflow-reference.html">Kotti CMS - workflow reference</a></li>
</ul>
<h3>
All Kotti posts published by <a href="https://twitter.com/davidemoro">@davidemoro</a>:<br />
</h3>
<ul>
<li><a href="http://davidemoro.blogspot.it/search/label/kotti">http://davidemoro.blogspot.it/search/label/kotti </a></li>
</ul>
<br />
<ul>
</ul>
Davide Morohttp://www.blogger.com/profile/16551873182140400424noreply@blogger.com0