Showing posts with label yeoman. Show all posts
Showing posts with label yeoman. Show all posts

2015-08-24

Introducing substancek. A Kotti project

Let me introduce substancek, a Kotti (http://kotti.pylonsproject.org) project.

What it is substancek?

substancek is:
  • Kotti (web application framework) + decoupled admin interface
It is only an additional layer upon the following opinionated stack:
with the following motto:
"""(even) better development experience and complete frontend freedom"""
and introduces (or better promotes) the concept of private admin area (backend) decoupled from the public side (frontend) of your web applications built with Kotti.

In other words it is a set of technologies addressed under the substancek brand that let you extend Kotti in order to use it just as a private backend administration area for your application data.

So you are still using plain Kotti with an additional package (at least kotti_backend depending on what you need).

If you want to know more I've discussed here benefits and why frontend decoupled from the backend pattern. See http://davidemoro.blogspot.it/2015/07/kotti-cms-successful-story-part-1.html

substancek name

Tribute to:
  • substanced (http://www.substanced.net). Carefully designed, well documented Python web development framework based on Pyramid/ZODB. It shares the concept of management views decoupled from the retail views for each published resource
  • kotti (http://kotti.pylonsproject.org). The Kotti framework

When substancek is for you

Any project of any size (from micro to XXL) involving content management that needs:
  • RDBMS. Kotti is an opinionated framework opposed to plain Pyramid. The persistence layer is managed by SQLAlchemy (http://www.sqlalchemy.org)
  • user friendly and production ready admin/editing interface for content producers so they can immediately start adding content to your web application with copy/cut & paste, clean urls, etc
  • rapid development. Pyramid and Kotti are easy to learn and let you become productive quickly and speed up your development. Even more with the decoupled admin interface.
  • stability. Pyramid and Kotti exist since many years and they are solid rock solutions production ready
  • frontend freedom. The admin interface comes for free while it is completely up to you add what your application needs (retail views). Since they are completely two different applications there are no css/js conflicts and you can integrate your preferred frontend tool chain without get influenced by how the admin interface is built.
  • flexibility. Pyramid won't hold you back when your application is small and won't get in your way when your application becomes large.
    Even if you start small, using Pyramid you anticipate what customers will need later avoiding expensive rewrites and this can make the difference. In other words this also means successful projects in the long term. Same for Kotti. See What makes Pyramid unique.
So if you project needs (or in future iterations) one or more:
  • complex security policies
  • workflows
  • hierarchical data support
  • or even intranet/extranet like collaboration areas
you might consider even more substancek (kotti_backend + Kotti + Pyramid + SQLAlchemy).

For example:
  • very small applications. For example a just one view public json endpoint for published news-like resources consumed by a third party app with an admin interface for editing contents
  • heavy Javascript based applications with modern frontend tools (eg: SPA + REST) with a decoupled admin interface
  • content management solutions
    • blog
    • ecommerce
    • intranets
    • large CMS-ish applications
Note well: if you don't need workflows don't be scared because there is no overkill. You can use a one state workflow or no workflow at all for example. No hierarchical data? Use not nestable resources and so on. If it comes out later that you need them it will be quite easy converting your code.

Alternatives

You can use plain Kotti, without the substancek's kotti_backend setup. Or if you prefer noSQL try the excellent substanced (substanced + Pyramid + ZODB). Both solutions are lightweight, tested, well documented and easy to learn. Alternatively if you need really a minimal and unopinionated solution you might use plain Pyramid.

Do you need something more? You might consider to use Plone (https://plone.org) as a framework.

Anyway the good news is that Python is plenty of good options.

substancek architecture details

As already told you the private admin area (backend) and the rest of the application (frontend) are two complete different applications with different settings, different views and shared authentication.

Assuming you are going to use PasteDeploy to run your application, let's consider the following configuration files setup:
  • backend-dev.ini, configures the private admin interface based on Kotti thanks to the kotti_backend plugin
  • frontend-dev.ini, configures your application you are developing (a full CMS frontend implementation or a microapp with just one retail view).
  • development.ini (optional), mount the backend-dev.ini and frontend-dev.ini applications in the same process (/admin for the private admin interface and / for your application). Alternatively you can run the frontend and backend using two processes waiting for requests on different ports and play with rewrite rules.

backend-dev.ini

[app:kotti]
use = egg:kotti

...
pyramid.includes =
    pyramid_debugtoolbar
    pyramid_tm
    kotti_backend.views.override_root_view


kotti.configurators = kotti_tinymce.kotti_configure
    kotti_backend.kotti_configure

kotti.use_workflow = kotti_backend:workflows/simple_backend.zcml

kotti_backend.goto_frontend = 1 
This is a normal Kotti setup with:
  • enabled the (optional) root view override for our admin interface. In other words the default root view will be @@contents instead of the standard Kotti's hello page (see pyramid.includes)
  • added kotti_backend to kotti.configurators
  • override the Kotti's default workflow with the one provided by kotti_backend (kotti.use_workflow). Playing with the additional pview permission you can decide your resources visibility on the frontend. See the workflow definition here https://github.com/Kotti/kotti_backend/blob/master/kotti_backend/workflows/simple_backend.zcml
  • enable a "Goto frontend" link (kotti_backend.goto_frontend) for easy switching from admin interface and frontend 
See more options on the kotti_backend's README file:

frontend-dev.ini

[app:main]
use = egg:Kotti
...

kotti.use_workflow = kotti_backend:workflows/simple_backend.zcml

kotti.configurators =
    your_package.kotti_configure

kotti.base_includes =
    kotti
    kotti.views
On the frontend configuration file we share the same workflow in use on the admin interface (kotti.use_workflow).

One of the most important configuration is the kotti.base_includes override: here we decide what will be loaded on our application. We omit all the Kotti views loaded by default in the standard setup and we load what we want to include where:
  • kotti, loads the kotti "core"
  • kotti.views (optional), load some view discriminators and utils defined by Kotti if you need them
  • your_plugin.your_includes, load your includes registering the views needed by your application
The kotti.configurators typically auto includes your package and tell what should be included in your application (pyramid.includes). See the Kotti documentation for more info.

In other words:
"what is not loaded, it doesn't exist"
so the final result is that there is nothing exposed on the frontend except what you decide to load and extreme control. You can register just only one view application or configure a complex setup for a CMS-like application: it's up to you registering only the views your application needs and no more. This way you can use completely different frontend frameworks, different versions of Javascript libraries, you have no css/js conflicts and no need to hide unneeded things and you decide which resources will be published on the frontend.

See also another advance usage pattern "Using Kotti as a library" http://kotti.readthedocs.org/en/latest/developing/advanced/as-a-library.html

development.ini

# See http://pythonpaste.org/deploy/#paste-composite-factory
[composite:main]
use = egg:Paste#urlmap
/ = config:frontend-dev.ini
/admin = config:backend-dev.ini

[server:main]
use = egg:waitress#main
host = 127.0.0.1
port = 5000
The (optional) development.ini shows how to configure a composite application with different mount points. You can change /admin with /cms or /whateveryouwant depending on your needs.

Examples

You can checkout the https://github.com/substancek/substancek_cms_theme package if you want to see in action a (quite complex) example.

I'm going to provide more and simpler examples (eg: a pretend micro application), see the roadmap.

What are the substancek related packages

Here you can see the whole substancek ecosystem:
  • kotti_backend, generic package that turns Kotti to a private admin area. This is the keystone for every substancek like project
  • CMS-like applications
    • substancek_cms (roadmap), a Kotti CMS distribution with an alternative frontend theme based on SASS, html/templates minification and assets optimization based on Yeoman tools. It will be based on the kotti_project experiment available here https://github.com/davidemoro/kotti_project
    • substancek_cms_theme, an example public side CMS implementation based on Kotti built with SASS and html/templates minification
    • kotti_actions, link actions backend implementation for header, footer and main navigation links
    • kotti_boxes, portlets and box backend implementation
    • kotti_es, elastic search support (to be refactored)
  • common utilities
    • pyramid_html_minifier, generic Pyramid package that introduces Chameleon templates minification with no overhead. Required by substancek_cms_theme.
    • build_commands, generic Python package that defines a set of initialization commands on install (eg: python setup.py npm/bower/gulp) for frontend stuff. Required by substancek_cms_theme.

Who's using substancek technologies

MIP - graduate school of business

The MIP (Politecnico di Milano graduate school of business - www.mip.polimi.it/en) uses substancek technology inside for the private admin interface. This approach is so flexible that let you use Kotti as a private admin content management area and even implement your public views using other frameworks or non Python languages (for example PHP+Symfony2).

See:

substancek_cms_theme

This is a work in progress opinionated CMS frontend implementation that reuses existing Kotti templates and logics.

The look and feel is the same you get with a standard Kotti installation but it shows how to integrate and distribute a Python package integrated with a Yeoman setup (http://yeoman.io) that provides:
  • production vs development setup. In production mode will be used assets and templates from the dist
  • Chameleon templates minification with no overhead thanks to pyramid_html_minifier when you are in "production" mode
  • SASS
  • image/assets optimization
  • build commands for npm/bower/gulp initialization thanks to the generic package build_commands
Once installed you'll see the admin interface visiting http://localhost:5000/cms.
See the code here: https://github.com/substancek/substancek_cms_theme

Next steps

If you want to contribute there is a lot to do:
  • contributions to the Kotti core (for example @@contents pagination, REST, etc and new ideas)
  • creation of the substancek_cms package that puts together the substancek_cms_theme default theme plus the most useful third party Kotti plugins like news, events, etc with decoupled admin interface and Vagrant/Ansible automated setup based on kotti_project for easy installation/evaluation
  • creation of simple pretend packages, add more examples (microapp, blog)
  • a toolbar available on the frontend just of editor users for improved usability (the same shown in the MIP case history) implemented with a Pyramid tween
  • more work on substancek_cms_theme, implement advanced features shown in the MIP case history like link managers and portlets and refactor the Yeoman folder removing what is not strictly needed
  • scaffolding, help people create new projects with decoupled admin interface using Kotti as a framework
  • substancek dedicated github page
Contributions, feedback or pings like "hey, I'm going to use Pyramid/Kotti for the my next project" will be very appreciated!

Documentation

All Kotti posts published by @davidemoro

Twitter links

2014-10-26

Linuxday2014 Torino

Il 25 ottobre ho avuto l'onore di presentare un talk al Linuxday2014 a Torino presso i locali del Politecnico dal titolo...

Strumenti  per lo sviluppo web

Automatizza operazioni ripetitive e noiose:
Migliora la tua esperienza di sviluppatore web adottando strumenti avanzati per automatizzare operazioni ripetitive e noiose: scaffolding nuovi progetti, css performance tooling, live reload browser, javascript linting, merge e minificazione asset, uncss, cdn, integrazione con framework web e tematiche legate al deploy.
L'ospite d'onore del talk è stato Yeoman, con numerosi cenni a Bower, Npm, Grunt e naturalmente Yo (e i suoi generatori più famosi come per esempio generator-webapp e generator-angular).

Inoltre per terminare sono stati presentati una serie di esempi di integrazioni di Yeoman con framework web di diverse tipologie (basati su Python - come non citare Plone o Pyramid - e NodeJS - portando come esempio il framework loopback.io con la sua AngularJS SDK) ed illustrate alcune delle (molte) tecniche avanzate per l'ottimizzazione del front end (es: grunt-uncss).

Insomma, argomenti molto vari ed è stata dura riuscire a condensare tutto in una sola ora di talk a disposizione!

Per quanto riguarda gli altri talk posso dire che il livello era molto alto e, purtroppo, avrei voluto assistere ad altre due presentazioni ma erano in parallelo con la mia (Arduino e virtualizzazione con Docker)... Da segnalare una buona partecipazione da parte dei pinguini torinesi, l'ottima organizzazione, location veramente azzeccata (comoda con i trasporti, aule capienti, buona acustica), la simpatia dei volontari allo stand dei gadget. Talk preferito? Scrivere driver per il kernel, presentato dal simpaticissimo e ultra competente kernel hacker Alessandro Rubini (complimenti!!!).

Come promesso ecco le slide.

Slide

Le slide sono disponibili su https://speakerdeck.com/davidemoro/linux-day-2014-torino-strumenti-per-lo-sviluppo-web

Domande e risposte

Riporto una serie di domande e risposte emerse alla fine del talk, purtroppo c'è stato poco tempo a disposizione per portare avanti la discussione e ho aggiunto delle integrazioni e precisazioni postume con aggiunta di riferimenti. Vado a memoria e spero di non aver dimenticato nulla.

Q: La tecnica dell'uncss sembra essere un anti-pattern rispetto a soluzioni Compass-sass
A: Mmm, no. Questo pattern è stato recentemente presentato alla csseuconf dall'autore di grunt-uncss che lavora presso Google, ma il concetto esisteva già ed è stato già applicato in diversi contesti. Per maggiori informazioni puoi approfondire la questione leggendo l'articolo originale dell'autore di grunt-uncss http://addyosmani.com/blog/removing-unused-css/

Q: Si poteva snellire allo stesso modo il css di Twitter boostrap con Compass-Sass sfruttando la modularità di Twitter Bootstrap (pattern migliore)
A: Sì, ma solo parzialmente. Grunt-uncss è uno strumento diverso e consente comunque di ottenere delle performance migliori. Inolte uncss può essere applicato a qualsiasi css o framework UI indipendentemente dal supporto o meno di sass. Per maggiori informazioni fai riferimento all'articolo dell'autore http://addyosmani.com/blog/removing-unused-css/

Q: Hai parlato spesso di AngularJS nel tuo talk, c'è una correlazione con Yeoman o è solo un caso? Perchè non emberjs o altri framework.
A: No, non c'è correlazione, giusto per fare un esempio con del codice Javascript. Yeoman è totalmente indipendente dal framework Javascript usato. Avrei benissimo potuto usare una qualsiasi altra tecnologia. La scelta di riportare AngularJS come esempio è ricaduta semplicemente in base a una mia preferenza personale, ma i concetti visti sono assolutamente generici.

Q: Yo funziona anche sotto Windows?

A: In realtà ho la fortuna di usare solo Linux da diversi anni per cui non so esprimermi di preciso, ma viste le tecnologie usate immagino di sì. Dopo aver fatto una breve ricerca a casa posso affermare che la risposta è certamente positiva, anche se non ho mai provato in prima persona. Vedi annuncio su http://addyosmani.com/blog/yeoman-update-announcing-1-0-beta-4

Q: Esistono delle soluzioni per il deploy e l'hosting?

A: Ci sono numerosi servizi Paas (Platform As A Service) che consentono di fare il deploy della propria applicazione NodeJS, Python, ecc tramite una comoda CLI (Command Line Interface) ed offrono una serie di servizi aggiuntivi molto comodi. Basta cercare su Google per trovare molte alternative. Per quanto riguarda Plone invece ci si può rivolgere a uno dei provider o installare un proprio VPS, vedi https://plone.org/support/providers

2014-09-25

collective.angularstarter (Plone + AngularJS + Yeoman kickstarter project)

Get started with Plone + AngularJS without any of the normal headaches associated with a manual setup of useful tools that let you improve your development experience and the deploy of your application.

Since I have been using AngularJS on Plone, I decided to create a reusable starter scaffold (or something or similar) based on Yeoman that let me save precious time.

That's why I created:
This is a plugin that let you bootstrap single page web applications (or heavy Javascript logics) based on Plone+AngularJS+Yeoman.

Yeoman workflow benefits

collective.angularstarter is powered by the Yeoman workflow. If you want to see what are the Yeoman benefits due to an integration with a framework you might have a look at:
Next sections will talk about what you can build with Plone if you are not familiar with it, in particular heavily dynamic Javascript based verticalizations built with collective.angularstarter or similar techniques.

Plone

Plone is not only a CMS but a framework built with the Python programming language that let you build complex web applications, intranet, websites with strong focus on:
  • contents
  • security and sharing
  • workflow
  • searchability
You can extend the features provided by default by Plone thanks to a considerable number third party plugins.

You can see a couple of examples about you can build with Plone verticalizations with collective.angularstarter.

Coworking application

You can create custom content types for meeting rooms, private desks or common desks seats.

Basically the main object you are sharing is a folderish with metadata that let you configure the resource, for example:
  • image, title, description and rich text widget. You can describe the resource you are sharing
  • configure time slots configuration and hourly costs
  • configure time slots for group of hours with costs (ex: morning, afternoon)
  • daily cost or month cost, depending on the type of resource
  • number of available seats available in parallel
Once the resource has been published, users can buy the suitable time slots depending on the type of resource and availability.

For private or common desks you can choose to search for multiple seats for multiple days or months. For meeting rooms you can buy just one unit time slot, multiple hours, group of hours (ex: morning) or the full day.

For example if you are searching for a meeting room you can choose partial days mode and select the available day you want to select:



The infinite scrolling shows you only the days that fits what you are searching for (2 private desks)
One selected the day, you can choose one or more available slots (for example morning):
And then: buy it!
The reservation objects are based on event types, since the reservation has a start and end datetime. So you can easily perform non-expensive catalog queries in order to search for slots available.

The project itself it is more complex because Plone it is integrated with an external invoice management software, a Paypal interface that let you buy more credits and a personal user box that let the users to see invoice PDF files and other notifications.

Advanced search forms

You can also use plone as a backend for a highly dynamic single page web application.
 
You can mix Plone data with external resources provided by third party server in order to build a complex search form.

For example the main search prompt to the user a master-slave AJAX widget
where the vocabulary of slave selects depend on the value of the previous one): 

or search for different criteria:
In this particular case results will appear after you fill all the needed information, but it is quite easy to implement a live search.

If you need more info about how to create a master-select widget component with Angular you may have a look at this article http://davidemoro.blogspot.com/2014/09/angularjs-master-slave-select-with.html.

What collective.angularstarter is

The collective.angularstarter plugin is:
  1. a Plone + AngularJS kickstarter project. You can use this package when you want to develop a single page web applications powered by Angular using Plone as backend. With all the benefits of the Yeoman workflow
  2. scaffolding tool that let you extend this package, add more features and then clone it creating a more sophisticated application. You can redistribute it with another name. Or you can develop a rapid prototype of your reusable application and after create a new zopeskel or yo package generator with one or more options. The clone hack might fail in some corner case but it should help you to convert an existing package to another. Anyway if something goes wrong you can easily correct the problems by hand. I get used to apply a similar script when me or other colleagues chose a very ugly package name and then you have to rename it. Maurizio, remember?! How many days we saved with this script? 
Anyway when you install collective.angularstarter and visit the @@angularstarter browser view it shows an example of AngularJS app with enabled by default:
Here you can see how the @@angularstarter view looks like:
collective.angularstarter screenshot. Fill the input text and you'll see the page instantly updated
After that it's up to you coding with AngularJS and Plone!

Results

The following screenshot show you what happens if you analyze the network section of Firebug when you are in development mode:
or in production more:

Wait a moment! The resulting resultim bootstrap.css weights in at only 3,2 KB?! That's the power of minification and uncss tasks
A you can see you'll get (see the part 1 article of Pyramid starter seed project for further details about uncss and other tips explained):
  • html minified (experimental, disable for real project)
  • lighter images (no asset images in collective.angularstarter)
  • most popular Javascript resources automatically cdn-ified
  • css files concatenated, uncssed and minified
  • javascript concatenated and uglified
  • [update 20140926] assets automatically revved (avoid nasty caching problems)
  • ... you can do more installing additional grunt tasks
How did I did it? Basically I played with Plone's resource registrations and layers.  See https://github.com/collective/collective.angularstarter/blob/master/collective/angularstarter/browser/configure.zcml

collective.angularstarter wraps a modified Yeoman AngularJS project (browser/angular): asset paths modified, bower_components folder renamed and a couple and other local changes to the Gruntfile.js file.

Hope you'll find collective.angularstarter useful. Feedback will be very appreciated!

UPDATE 20150303: I forgot to mention that collective.angularstarter plays well with Diazo (http://docs.diazo.org/en/latest/). This way you can write a pure static mocks with lots of javascripts and then using the html as the diazo theme with a bunch xml rules, with the backend decoupled with the frontend. Easy and tested on production! Probably I'll write a new write up.

2014-09-24

Pyramid starter seed template powered by Yeoman (part 3)

In the previous articles we have seen:
  • what are the benefits of the Yeoman workflow applied to Pyramid, a traditional web framework (part 1)
  • how to install pyramid_starter_seed, a Pyramid+Yeoman flavoured starter template (part 2)
Once installed pyramid_starter_seed, we will see now:
  • how it works under the hood
  • how to manage things with grunt
  • how to create and share other templates based on pyramid_starter_seed

How it works under the hood (narrative)

pyramid_starter_seed registers only one route (home -> /) and static assets views.

The home route is associated to a view callable with a webapp/%s/index.html renderer.
from pyramid.view import view_config

@view_config(route_name='home', renderer='webapp/%s/index.html')
def my_view(request):
    return {'project': 'pyramid_starter_seed'}
Views can be associated to routes imperatively or through a scan.

Wait, there is no .html renderer handled by default in Pyramid! The pyramid_starter_seed will register a .html renderer that will replace the string token with app or dist depending on the production settings and it calls the original .pt renderer.

You can choose between production vs development mode running your pyramid app with the appropriate .ini file provided by pyramid_starter_seed.

Here you can see the relevant parts of the production.ini:
[app:main]
use = egg:pyramid_starter_seed

PRODUCTION = true
minify = dist
...
and development.ini:
[app:main]
use = egg:pyramid_starter_seed

PRODUCTION = false
minify = app

...
As you can imagine the PRODUCTION configuration tells the application to switch between production or development. The minify configuration tells the .html renderer how to construct templates paths.

Let's see how looks like the index.html template. When you write css or javascript files you want to keep things separated on different modules when you are in development mode, but in production mode you might want a unique concatenated and minified/uglified resource. Here you can see how you can do that for the Bootstrap javascripts modules you might want to enable:
<!doctype html>
<html class="no-js"
      lang="${request.locale_name}"
      tal:define="minify python:request.registry.settings['minify'];
                  production python:request.registry.settings.get('PRODUCTION', 'false') == 'true'">
...

         <tal:production tal:condition="production">
            <script src="${request.static_url('pyramid_starter_seed:webapp/%s/scripts/plugins.js' % minify)}"></script>
        </tal:production>
        <tal:not_production tal:condition="not:production">
            <script src="${request.static_url('pyramid_starter_seed:webapp/%s/bower_components/bootstrap/js/alert.js' % minify)}"></script>
            <script src="${request.static_url('pyramid_starter_seed:webapp/%s/bower_components/bootstrap/js/dropdown.js' % minify)}"></script>

        </tal:not_production>
        <!-- build:js scripts/plugins.js -->
        <tal:comment replace="nothing">
            <!-- DO NOT REMOVE this block (minifier) -->
            <script src="./bower_components/bootstrap/js/alert.js"></script>
            <script src="./bower_components/bootstrap/js/dropdown.js"></script>

        </tal:comment>
        <!-- endbuild -->
...
</html>
So in development mode you will have two separate javascript files: alert.js and dropdown.js. When you are in production mode it will served a unique concatenated and uglyfied scripts/plugins.js. At first time it might seem a bit complicated or a task with a too verbose setup, but it is a simple and very powerful mechanism.

For example: do you want to add another javascript file to the plugins.js bundle? Just add two lines and you are ok!

It works without any other configuration thanks to the start and end comments blocks build:js and endbuild that groups assets groups.

Simple, isn't it? Same thing for css files.

And if you have to include images is even more simple with:
<img class="logo img-responsive" src="${request.static_url('pyramid_starter_seed:webapp/%s/images/pyramid.png' % minify)}" alt="pyramid web framework">

How to manage things with grunt

The concatenation, minification/uglyfication and image optimization (with other tasks specified in the grunt's pipeline) it is automatically performed just running the following command:
$ grunt build
This is just one of the possible implementations. Feel free to contribute and improve pyramid_starter_seed.

How to clone pyramid_starter_seed 

Fetch pyramid_starter_seed, personalize it and then clone it!

Pyramid starter seed can be fetched, personalized and released with another name. So other developer can bootstrap, build, release and distribute their own starter templates without having to write a new package template generator. For example you could create a more opinionated starter seed based on SQLAlchemy, ZODB nosql or powered by a javascript framework like AngularJS and so on.

The clone method should speed up the process of creation of new more evoluted packages based on Pyramid, also people that are not keen on writing their own reusable scaffold templates.

So if you want to release your own customized template based on pyramid_starter_seed you'll have to call a console script named pyramid_starter_seed_clone with the following syntax (obviously you'll have to call this command outside the root directory of pyramid_starter_seed):
$ YOUR_VIRTUALENV_PYTHON_PATH/bin/pyramid_starter_seed_clone new_template
and you'll get as a result a perfect renamed clone new_template:
A new starter template cloned from pyramid_starter_seed
If you provide tests you can check immediately if something went wrong during the cloning process.

In effect the clone console script it might not work in some corner cases just in case you choose a new package name that contains reserved words or the name of a dependency of your plugin, but it should be quite easy to fix by hand or improving the console script. Anyway this mechanism has been tested several years: I have built this script years ago because I was fed up with ugly package names chose by me or other colleagues of mine and allowed me to save a lot of time.


If you want to disable the console script on your new template (for example: new_template_clone) drop from setup.py the console script configuration.

So it sounds like a viral extension mechanism (I hope).

End of story?


In the next future I'd like to create a new Pyramid starter seed for single page web apps based on SQLAlchemy and powered by AngularJS. So if you similar plans... together is better: if you want to share your thoughts, improvements, feedback in general, or if you are going to create your own template based on pyramid_starter_seed please contact me (Twitter, Google+, Linkedin)!


Anyway I hope you'll save time with pyramid_starter_seed.

Links

2014-09-18

Pyramid starter seed template powered by Yeoman (part 2)

In the previous blog post we have seen what are the benefits of using the Yeoman workflow fully integrated with a web development framework like Pyramid. See:

Now we'll add more technical details about:
  • how to install pyramid_starter_seed and its prerequisites

How to install pyramid_starter_seed

    Prerequisites

    As you can imagine, nodejs tools are required.

    I strongly suggest to:
    • avoid system packages because they are too old
    • install nodejs with nvm (Node Version Manager)
    I won't cover the nodejs installation but it is quite simple if you follow these instructions provided by this useful link:
    The nvm achronym stands for NodeJS Version Manager. Once installed nvm, installing nodejs it is as simple as typing nvm install VERSION (at this time of writing 0.10.32).

    Nodejs is shipped with the command line utility named npm (Nodejs Package Manager) and we will use npm for installing what we need.

    We need to install our global (-g option) dev dependencies, so just type:
    $ npm install -g bower
    $ npm install -g grunt-cli
    $ npm install -g karma

    Starter seed project installation

    Create an isolated Python environment as explained in the official Pyramid documentation and instal Pyramid.

    Once installed you can clone pyramid_starter_seed from github:
    $ git clone git@github.com:davidemoro/pyramid_starter_seed.git
    $ cd pyramid_starter_seed
    $ YOUR_VIRTUALENV_PYTHON_PATH/bin/python setup.py develop
    Not finished yet, continue.

    Yeoman initialization

    Go to the folder where it lives our Yeoman project and initialize it.

    These are the standard commands (but, wait a moment, see the "Notes and known issues" subsection):
    $ cd pyramid_starter_seed/webapp
    $ bower install
    $ npm install --loglevel verbose
    Known issues:
    • if you are behind a proxy you'll have to configure properly npm
    • if you have a slow internet connection you might experience timeout problems.

    Build phase

    Just type:
    $ grunt
    and... probably it will fail because of a couple of known issues shipped with the latest version of generator-webapp or its dependencies.

    Probably these issues will be fixed in newer generator-webapp releases. However here it is how to solve these problems, so don't worry:
    1. grunt-contrib-imagemin fix
      Problem with grunt:
      Warning: Running "imagemin:dist" (imagemin) task
      Warning: Bad argument Use --force to continue.
      Solution:
      $ npm cache clean
      $ rm -fR node_modules       # not sure it is needed, don't remember
      $ npm install grunt-contrib-imagemin
    2. Mocha/PhantomJS issue
    Problem with Mocha/PhantomJS launching grunt
    Warning: PhantomJS timed out, possibly due to a missing Mocha run() call. Use --force to continue.
    Solution:
    $ cd test
    $ bower install
    Run bower install in the test directory of webapp (pyramid_starter_seed/webapp/test). This is a known issue, see https://github.com/yeoman/generator-webapp/issues/446.

    Run you Pyramid app

    Now can choose to run Pyramid in development or production mode.
    Just type:
    $ YOUR_VIRTUALENV_PYTHON_PATH/bin/pserve development.ini
    or:
    $ YOUR_VIRTUALENV_PYTHON_PATH/bin/pserve production.ini
    Done!


    In the next blog post with topic Pyramid + Yeoman (coming soon) I'm going to talk about:
    • how to manage things with grunt and personalize pyramid_starter_seed registering other assets
    • how to clone pyramid_starter_seed. Yes, you can easily customize it creating something of more sophisticated and create your own starter seed with another name. Without having to write a package generator
     So stay tuned..

    Links

    2014-09-16

    Pyramid starter seed template powered by Yeoman (part 1)

    Book of the month I'm reading this summer: Pylons/Pyramid (http://docs.pylonsproject.org/en/latest).


    Pyramid (http://www.pylonsproject.org) is a minimal Python-based web development framework that let you "start small and finish big".

    It stole a lot of (good) ideas and concepts from other mature Python web frameworks and it is build with the pluggable and extensible concepts in mind. Read: no need to fork applications.

    Furthermore Pyramid is database and template engine agnostic: you are free.

    From the very beginning Pyramid allows you to become productive quickly. So why not start with something of useful?

    Pyramid + Yeoman

    The goal of this experiment is integrate yeoman with Pyramid (or other frameworks like NodeJs/Express with AngularJS or Plone as already did), preserving the yeoman's workflow.

    UPDATE 20140926: here you can see a Plone + AngularJS + Yeoman article (collective.angularstarter)

    In this article I'll talk about what are the benefits you get integrating your Pyramid app with Yeoman, in future posts I'll discuss how they work under the hood with additional technical details omitted here (each used component deserves an entire blog post).

    Yeoman
    You might wonder why? Because of the importance of tooling. Since it is very important build an effective developer tooling ecosystem, I want to integrate the simple starter demo app with commonly used tools to help you stay productive. So this simple application prototype it is just an experiment that should help you to integrate with modern web development tools provided by the yeoman workflow stack (http://yeoman.io).

    Choosing the right tools is very important for the best develop experience and I cannot work anymore without Yeoman, especially when coding with Javascript.

    Grunt
    Yeoman it is internally based on three important components (nodejs powered):
    • yo, scaffolding tool like pcreate, paster or zopeskel. It is widely adopted by a large and trasversal community
    • grunt, system used for build, preview and test your software. Gulp is another popular option
    • bower, used for dependency management, so that you no longer have to manually download and manage your scripts
    Bower

    So with the yeoman's tools you can just code, avoid annoying repetitive tasks and don't worry about:
    • javascript testing setup
    • javascript code linting
    • javascript/css minification and merging
    • image minification
    • html minification
    • switch to CDN versions of you vendor plugins in production mode
    • auto-reload browser
    • much much more
    So let's see together what happened to our pyramid starter demo template created with pcreate -t starter integrated with a yeoman's generator-webapp project.

    The result will be a Pyramid starter seed project integrated with modern non Python-based web development tools.

    Goals

    Management of third party assets

    You no longer have to manually download and manage your scripts with the Bower package manager.

    From http://bower.io:
    """Bower works by fetching and installing packages from all over, taking care of hunting, finding, downloading, and saving the stuff you’re looking for."""
    So just type something like: bower install angular-translate --save and you'll get the rigth resource with pinning support.

    Tasks automation

    Automation, automation, automation.

    From http://gruntjs.com:
    """Why use a task runner? In one word: automation. The less work you have to do when performing repetitive tasks like minification, compilation, unit testing, linting, etc, the easier your job becomes. After you've configured it, a task runner can do most of that mundane work for you—and your team—with basically zero effort."""
    Examples:
    • grunt serve
    • grunt test
    • grunt build
    • grunt YOUR TASK 
    • etc

    Jslint

    No more deploy Javascript code with bad indentation, syntax errors or bad code practices.

    All syntax errors or bad practise will be found.

    Image minification

    The build process will detect and minify automatically all your asset images.

    Uncss task

    Modern (and heavy) UI frameworks like Twitter Bootstrap provide an excellent solution for prototyping your initial project, but most of the times you are using a very minimal subset of their functionalities.

    https://twitter.com/davidemoroThis inspiring Addy Osmani's blog post helps you to remove unused css in your pages with a grunt task named grunt-uncss (https://github.com/addyosmani/grunt-uncss):
    The original not-minified bootstrap.css weights in at 120 kB before removing unused rule.

    Css concat and minification

    You can split your css code into different files and then the build process will concat and minify them creating a unique app.css file. This way you write modular and better readable css files, reducing the number of browser requests.

    The theme.css file is quite small but in real projects you can save more. In this case:
    The configured build pipeline is concat, uncss and cssmin. 122.85 kB (original bootstrap.css) -> 4.64 kB (uncss) -> 3.45 kB (minification)

    Automatic CDN-ification

    It is handy using unminified versions of third party javascript libraries during development and switch to CDN versions in production mode with well known benefits for your website.

    Don't worry: the cdnify task will take care about this boring issue. Automatically.

    You save a boring manual and error-prone configuration.

    Composable bootstrap.js version

    The Pyramid starter project is based on Twitter Bootstrap.

    Twitter Bootstrap
    Depending on your project you can load the whole Twitter Bootstrap Javascript code at once or including individual plugins.

    As you can see the Javascript component of Twitter Bootstrap is very modular: http://getbootstrap.com/javascript. So if you don't use a particular feature, just don't include it.

    This way in development mode you will have all individual plugins splitted in different files, in production it will served a unique concatenated and minified Javascript file built automatically.

    So if you just need alert.js and dropdown.js you can get a 2.79 kB plugins.js:

    The concatenation of alert.js and dropdown.js produces a 7.06 kB, that weight in at 2.79 kB after minification instead of the 8.9 kB (gzipped) bootstrap-min.js corresponding to not gzipped 27.2 kB.

    Html (template) minification

    Since the ZPT/Chameleon templating language is an extension of HTML with xml syntax,

    Brower are able to display unrendered ZPT/Chameleon templates
    theorically it can play well with html minificators.

    I know, template minification can lead to potential unexpected problems due to minification issues on template files... but this is my personal playground, so let me play please!

    So... why not switch to a pre-compiled minified template of your ZPT/Chameleon files when you are in "production mode"?

    Obviously during development you will use the original template files.

    The interesting side of this approach is that there is no overhead at response time, since the minification task runs just one time before deploying your application. It might be an option if you want just your html minified and you cannot feasibly add to your site or project additional optimization tools at web server level.

    Anyway I have tried this mad experiment and... if you don't use too aggressive minification params, it seems to work fine with good results. Try it at your own risk or just disable it. Here you can the effects on the generated index.html used in production:
    Template minified (7.62 kB -> 4.16 kB)

    Result: a lighter Pyramid

    Same results but a lighter Pyramid app:

    Let's see how it behave the standard Pyramid starter project:
    Standard Pyramid starter project (production.ini)
    And the Pyramid starter seed:
    Pyramid starter seed (production.ini)
    As you can see the seed version is ~38 Kb smaller and more performant.

    Useful links

    That's all?

    No, you can do more, for example:
    • reduce the  number or requests (for example you can merge vendor.css and app.css)
    • create and keep updated css sprites with grunt (https://github.com/Ensighten/grunt-spritesmith)
    • manage and upload all your assets to professional services like Amazon AWS (for example you can serve up all your images, styles and scripts from a S3 bucket + CloudFront). This way Pyramid will be able to handle more requests. Pyramid let you put static media on a separate webserver during production with static_url() in conjunction with add_static_view(), without having to change your templates code
    • generate static gzipped assets with Grunt and let your webserver serve them
    • install and configure dedicated performance modules at webserver level (Apache's mod_pagespeed)
    Let me know what you think about that, please. Hope soon I will manage to write the second part of this blog post explaining how I did it. In the mean time you can:

    Links

    2014-04-07

    yeoman generator mongodb interactive installer


    In the previous post we have seen how to create your own yeoman generators using custom _.templateSettings rules.

    However if you want to dive into yeoman generators you can play with generator-generator, have a look at the code of existing generators and... why not? Create your own whatever-you-want yeoman generator!

    I chose to implement a simple generator, based on custom _.templateSettings seen in the previous post, that let you automate the mongodb installation.

    Its name is generator-mongodb (https://github.com/davidemoro/generator-mongodb).

    Useful or not useful, this is my first whatever-i-want-generator, made just for fun. There is no tests coverage (I got stuck on a problem but the solution is on his way) and I won't maintain it. So that's just an example, anyway I hope you'll find useful hints.

    How it is implemented this generator under the hood

    This generator is internally based on the buildout build system (http://www.buildout.org).

    What is buildout? Buildout is a widely Python-based build system for creating, assembling and deploying applications from multiple parts, some of which may be non-Python-based. It lets you create a buildout configuration and reproduce the same software later.

    Don't worry: we are not going to write any line of python code! We need python because it is required by our install system).

    Buildout configurations are text-based, versionable, fine-grained reproducible environments that you can share with your team or create software installers (one example is the Plone CMS unified installer - see http://plone.org/products/plone).

    There are dozens of published buildout recipes ready to be used. For example:
    So with buildout you can enhance your productivity and automate your build process providind a light out of the box solution, instead of relying on virtual machines and huge downloads.

    This helps when introducing multiple developers to a project: you can reduce setup times and enhance productivity. This way we can ensure that the whole team we are collaborating with has the right environment and you can save a lot of time.

    Another example: if you are the creator of a nice open source project (for example a Node.js CMS based on MongoDB), you shoud consider that your download&install page will be visited by people just interested in evaluating a CMS or specific app solution without any knowledge or interest about the technology involved because they want to install and play with it locally. So you can provide simpler install instructions thanks to buildout (or any other automated system), for example you can replace "be sure you have installed node.js version x.y.z and MongoDB" with "checkout this folder and run this command: done!".

    If you don't provide similar easy way to install your solution you are assuming that your users are comfortable with the underlying technologies, they know how to install the external components you need to run your software.  - If you want to explore a different approach you can have a look at http://www.vagrantup.com/ with its virtual environments.

    Let's come back to the buildout. Buildout files can be extended, we can build different profiles (production, development, etc) and a very simple buildout.cfg file looks like the following:
    [buildout]
    parts = mongodb

    [mongodb]
    recipe = rod.recipe.mongodb
    darwin-32bit-url = http://downloads.mongodb.org/osx/mongodb-osx-i386-1.6.5.tgz
    ...
    How it works? Just one command and the buildout process starts, for example:
    $ ./bin/buildout [-c buildout.cfg]
    So thanks to buildout and existing recipes our generator workflow will be very simple:
    1. read one or more user inputs (version of mongodb, but you can add more advanced options if you want)
    2. render the buildout.cfg following the steps described here http://davidemoro.blogspot.it/2014/03/yeoman-generator-template-settings-lodash-underscore-js.html
    3. setup the right environment (it creates a local and isolated python environment that does not pollute the global python installed on your system)
    4. runs the buildout command for you
    At the end of the process you'll find a local environment with all you need. Since each buildout folder is an isolated, self-contained environment, you can install also different MongoDB versions on your machine as well.

    For advanced option you can see the rod.recipe.mongodb reference: https://pypi.python.org/pypi/rod.recipe.mongodb.

    How to use generator-mongodb

    Be sure you have installed node.js (see https://github.com/creationix/nvm), python and virtualenv (>= 1.11.4).

    If you are not comfortable with python you can install it with a couple of commands:
    $ sudo apt-get install python2.7
    $ sudo apt-get install python-pip
    $ [update 20140410:  not safe, follow the official doc] sudo pip install --upgrade virtualenv
    The install yo:
    $ npm install -g yo
    If generator-mongodb were released, installing it it would as simple as typing:
    $ npm install -g generator-mongodb
    Since this package is not released, you'll have to clone the github repository:
    $ git clone git@github.com:davidemoro/generator-mongodb.git
    $ cd generator-mongodb
    $ npm install
    $ npm link
    $ cd
    Then move where you want to install your local mongodb environment and type:
    $ mkdir yourmongodir
    $ cd yourmongodir
    $ yo mongodb
    And you'll see something of similar.

    Type the mongodb version and wait for the end of the install process (it might take a while).

    Once finished you'll find the mongodb executables:
    $ ls bin -1 | grep mongo
    mongo
    mongod
    ...
    Update 20140415: actually ./bin/mongod doesn't work because I missed to set the db path option in buildout.cfg. Just type mkdir -p data/db in your buildout root and launch ./bin/mongod --dbpath data/db. If you configure well the mongodb recipe it will work without typing any option.

    Done!

    Links


    2014-03-28

    yeoman-generator's template() now accept a forth parameter being the _.templateSettings

    We are talking about yeoman generators with custom _templateSettings rules.

    Sometimes when you are using underscore.js or lodash templates you have to change how parameters and expressions will be interpolated.

    You might want to do so because the syntax of your generated file does not fit the templateSettings defaults (example: you get a lodash/underscore template error like "SyntaxError: Unexpected token :") or you just want to use another syntax to interpolate vars or expressions.

    How can you do that with yeoman?

    Starting from 0.17.0-pre.1 prerelease version the yeoman-generator's template() accept a forth parameter being the _.templateSettings (see https://github.com/yeoman/generator/releases/tag/v0.17.0-pre.1).

    Here you can find an example about how to use custom _.templateSettings rules in a custom yeoman generator created with generator-generator (see http://yeoman.io/generators.html).

    Note well: this post won't talk about how to create a custom generator. You can find more information about this topic following the official yeoman guide. Anyway installing generator-generator is as simple as typing the following command:
    $ npm install -g yo generator-generator
    (after doing that you can have a coffee and/or brush your cat’s fur since it might requires a long time depending on your network or machine)

    So let's see how it works!

    app/index.js

    The template method usage is similar to the older versions of yeoman-generator except a richer signature:
    function template(source, destination, data, options)
    where the fourth parameter is an object to be used as _.templateSettings. For example here it is a working example:
    this.template('_buildout.cfg',
                  'buildout.cfg',
                   this,
        {
            evaluate: /\{\{([\s\S]+?)\}\}/g,
            interpolate: /\{\{=([\s\S]+?)\}\}/g,
            escape: /\{\{-([\s\S]+?)\}\}/g
        });

    app/templates/_buildout.cfg

    This is our _buildout.cfg template, you can populate values using the {{}} syntax. In this particular case we are building the download url of mongodb depending on the user input (mongodbVersion). For example:
    [buildout]
    ...
    # Do not remove this: <%= mongodbVersion %>
    recipe = rod.recipe.mongodb
    darwin-32bit-url = http://downloads.mongodb.org/...-{{=mongodbVersion}}.tgz
    You might wonder why there is still a line with <%= mongodbVersion %>. It seems to be necessary because without this line this prerelease version of yeoman-generator will produce the unrendered output. See https://github.com/yeoman/generator/issues/517.

    With this workaround all works fine, probably it won't be necessary in future versions.

    Example usage


    Results

    The resulting buildout.cfg contents will be similar to the following lines, depending on the user input:
    [buildout]
    ...
    [mongodb]
    ...
    # Do not remove this: <%= mongodbVersion %>
    recipe = rod.recipe.mongodb
    darwin-32bit-url = http://downloads.mongodb.org/osx/mongodb-osx-i386-2.4.9.tgz
    The rendered file will be used to install and configure automatically the given version of mongodb.

    Done

    First of all: feedback, please!

    And... I think to write a second part of this blog post talking about how this particular generator works under the hood (a yeoman based mongodb installer based internally on a widely used build system for creating, assembling, configuring and deploying applications). So if you are pretty curious stay tuned on @davidemoro for future write ups.

    20140407: see http://davidemoro.blogspot.com/2014/04/yeoman-generator-mongodb-installer.html fur further information.