2014-11-25

Dr. Jekyll and Mr. GitHub Pages

Time ago I started using the GitHub Pages service just for fun:
With GitHub Pages you can write documentation, create a landing page for your account or organization and create unlimited project sites for you and your projects.

Websites, blogs, landing pages, live demos.

Things are hosted directly from your GitHub repo. So just edit, push and your changes are live (and under version control and easy collaboration with other team members).

You can generate a site using one of the pre-built themes or create a site from scratch.

Who is using GitHub Pages?
We'll cover these topics in this article:
  • type of sites you can build with GitHub Pages
    • user or organization site
    • project site
  • site generation methods
    • pre-built themes
    • from scratch
  • advanced blogging tools (with Jekyll)

User or organization site from scratch (master)

You can create your personal page or organization site for free.

First: create a new repository named username.github.io, where username is your username (or organization name) on GitHub.

Then clone your repo:
$ git clone https://github.com/username/username.github.io
$ cd username.github.io
Create a demo hello world page:
$ echo "Hello World" > index.html
Push:
$ git add --all
$ git commit -m "Initial commit"
$ git push
Visit your page (you'll have to wait a bit the very first time you update your site):
  • http://username.github.io

Project site (gh-pages branch)

For Project sites, you have the option to generate a site with one of the available pre-built themes, or to create a site from scratch.

Generate a site (pre-built themes)

Go to the settings panel of you repository and click on the Automatic Page Generator.

You can use the editor to add content and then pick a theme and publish!

Start from scratch

Follow this guide:

Jekyll

GitHub pages can be used in conjunction with Jekyll http://jekyllrb.com/.

What you can do with Jekyll:
"""Using Jekyll, you can blog using beautiful Markdown syntax, and without having to deal with any databases.

One of Jekyll’s best aspects is that it is “blog aware”. What does this mean, exactly? Well, simply put, it means that blogging is baked into Jekyll’s functionality. If you write articles and publish them online, this means that you can publish and maintain a blog simply by managing a folder of text-files on your computer. Compared to the hassle of configuring and maintaining databases and web-based CMS systems, this will be a welcome change!"""
Here you can find some useful resources about Jekyll and GitHub Pages:
Prerequisites: Ruby. Your Ruby version should begin with 1.9.3 or 2.0.0.

User or organization site

Bootstrap a new dir name USER_OR_ORGANIZATION.github.io
$ jekyll new USER_OR_ORGANIZATION.github.io
$ cd USER_OR_ORGANIZATION.github.io
Basically you'll have to install Jekyll and Bundle as suggested on the GitHub official doc. Create a Gemfile file with this content:
source 'https://rubygems.org'
gem 'github-pages'
Check your url and baseurl options of your _config.yml file.
_config.yml for a project/repository site:
        ...
        baseurl: "/REPOSITORY" # the subpath of your site, e.g. /blog/
        url: "http://USER_OR_ORGANIZATION.github.io" # the base hostname & protocol for your site

Invoke Jekyll with Bundler:
$ bundle install
$ bundle exec jekyll build
The .gitignore file is already generated.

Now you can create the repo online on github and push your contents (.gitignore file already generated). Before pushing it is suggested to run Jekyll locally:
$ bundle exec jekyll serve
Result:
Now you can add new posts, change your page title, etc. See the _config.yml dir, the _posts dir and the official documentation online.

Now you can push your changes and they will display online.

Repo/Project site (with Jekyll)

You should create a gh-pages branch of your repository.

Follow this guide:
but instead of putting plain html files, initialize your branch with Jekyll (with bundle exec jekyll new .).

And then check your _config.yml. The user/organization site requires a different setup:
        ...
        baseurl: "" # the subpath of your site, e.g. /blog/
        baseurl: "/REPOSITORY" # the subpath of your site, e.g. /blog/
        url: "http://USER_OR_ORGANIZATION.github.io" # the base hostname & protocol for your site

Now you can build things and upload to GitHub. Results will displayed on http://USER_OR_ORGANIZATION.github.io/REPOSITORY.

Jekyll troubleshooting

Error 1 (LoadError):
/var/lib/gems/1.9.1/gems/jekyll-2.5.1/bin/jekyll:21:in `block in <top (required)>': cannot load such file -- jekyll/version (LoadError)
Solution (see https://github.com/jekyll/jekyll/issues/3084):
$ bundle exec jekyll COMMAND ("serve" or "new .")
Error 2 (Gemfile does not exist and directory is not empty)
I got two strange errors initializing an empty dir with Jekyll with the "bundle exec jekyll new ." command. Something like "error: Gemfile does not exist". Ok, let's create a Gemfile and "error: directory is not empty". If worked for me "bundle exec jekyll new USER_OR_ORGANIZATION.github.io"

2014-10-26

Linuxday2014 Torino

Il 25 ottobre ho avuto l'onore di presentare un talk al Linuxday2014 a Torino presso i locali del Politecnico dal titolo...

Strumenti  per lo sviluppo web

Automatizza operazioni ripetitive e noiose:
Migliora la tua esperienza di sviluppatore web adottando strumenti avanzati per automatizzare operazioni ripetitive e noiose: scaffolding nuovi progetti, css performance tooling, live reload browser, javascript linting, merge e minificazione asset, uncss, cdn, integrazione con framework web e tematiche legate al deploy.
L'ospite d'onore del talk è stato Yeoman, con numerosi cenni a Bower, Npm, Grunt e naturalmente Yo (e i suoi generatori più famosi come per esempio generator-webapp e generator-angular).

Inoltre per terminare sono stati presentati una serie di esempi di integrazioni di Yeoman con framework web di diverse tipologie (basati su Python - come non citare Plone o Pyramid - e NodeJS - portando come esempio il framework loopback.io con la sua AngularJS SDK) ed illustrate alcune delle (molte) tecniche avanzate per l'ottimizzazione del front end (es: grunt-uncss).

Insomma, argomenti molto vari ed è stata dura riuscire a condensare tutto in una sola ora di talk a disposizione!

Per quanto riguarda gli altri talk posso dire che il livello era molto alto e, purtroppo, avrei voluto assistere ad altre due presentazioni ma erano in parallelo con la mia (Arduino e virtualizzazione con Docker)... Da segnalare una buona partecipazione da parte dei pinguini torinesi, l'ottima organizzazione, location veramente azzeccata (comoda con i trasporti, aule capienti, buona acustica), la simpatia dei volontari allo stand dei gadget. Talk preferito? Scrivere driver per il kernel, presentato dal simpaticissimo e ultra competente kernel hacker Alessandro Rubini (complimenti!!!).

Come promesso ecco le slide.

Slide

Le slide sono disponibili su https://speakerdeck.com/davidemoro/linux-day-2014-torino-strumenti-per-lo-sviluppo-web

Domande e risposte

Riporto una serie di domande e risposte emerse alla fine del talk, purtroppo c'è stato poco tempo a disposizione per portare avanti la discussione e ho aggiunto delle integrazioni e precisazioni postume con aggiunta di riferimenti. Vado a memoria e spero di non aver dimenticato nulla.

Q: La tecnica dell'uncss sembra essere un anti-pattern rispetto a soluzioni Compass-sass
A: Mmm, no. Questo pattern è stato recentemente presentato alla csseuconf dall'autore di grunt-uncss che lavora presso Google, ma il concetto esisteva già ed è stato già applicato in diversi contesti. Per maggiori informazioni puoi approfondire la questione leggendo l'articolo originale dell'autore di grunt-uncss http://addyosmani.com/blog/removing-unused-css/

Q: Si poteva snellire allo stesso modo il css di Twitter boostrap con Compass-Sass sfruttando la modularità di Twitter Bootstrap (pattern migliore)
A: Sì, ma solo parzialmente. Grunt-uncss è uno strumento diverso e consente comunque di ottenere delle performance migliori. Inolte uncss può essere applicato a qualsiasi css o framework UI indipendentemente dal supporto o meno di sass. Per maggiori informazioni fai riferimento all'articolo dell'autore http://addyosmani.com/blog/removing-unused-css/

Q: Hai parlato spesso di AngularJS nel tuo talk, c'è una correlazione con Yeoman o è solo un caso? Perchè non emberjs o altri framework.
A: No, non c'è correlazione, giusto per fare un esempio con del codice Javascript. Yeoman è totalmente indipendente dal framework Javascript usato. Avrei benissimo potuto usare una qualsiasi altra tecnologia. La scelta di riportare AngularJS come esempio è ricaduta semplicemente in base a una mia preferenza personale, ma i concetti visti sono assolutamente generici.

Q: Yo funziona anche sotto Windows?

A: In realtà ho la fortuna di usare solo Linux da diversi anni per cui non so esprimermi di preciso, ma viste le tecnologie usate immagino di sì. Dopo aver fatto una breve ricerca a casa posso affermare che la risposta è certamente positiva, anche se non ho mai provato in prima persona. Vedi annuncio su http://addyosmani.com/blog/yeoman-update-announcing-1-0-beta-4

Q: Esistono delle soluzioni per il deploy e l'hosting?

A: Ci sono numerosi servizi Paas (Platform As A Service) che consentono di fare il deploy della propria applicazione NodeJS, Python, ecc tramite una comoda CLI (Command Line Interface) ed offrono una serie di servizi aggiuntivi molto comodi. Basta cercare su Google per trovare molte alternative. Per quanto riguarda Plone invece ci si può rivolgere a uno dei provider o installare un proprio VPS, vedi https://plone.org/support/providers

2014-10-23

Sqlite array type and Python SQLAlchemy

I need to write up things just for remembering how I solved a particular issue if occurs in the future.

Sqlite (with http://sqlitebrowser.org) is great for rapid prototypes development but it lacks some useful implementations provided by Postgresql (for example the sqlalchemy.dialects.postgresql.ARRAY type).

I solved implementing a SQLAlchemy TypeDecorator with a json serialization:   
Here it is the self-explaining code:
from sqlalchemy.schema import Column
from sqlalchemy.types import (
    Integer,
    String,
    TypeDecorator,
    )
from sqlalchemy import Sequence
from pyramid_sqlalchemy import BaseObject as Base
import json

class ArrayType(TypeDecorator):
    """ Sqlite-like does not support arrays.
        Let's use a custom type decorator.

        See http://docs.sqlalchemy.org/en/latest/core/types.html#sqlalchemy.types.TypeDecorator
    """
    impl = String

    def process_bind_param(self, value, dialect):
        return json.dumps(value)

    def process_result_value(self, value, dialect):
        return json.loads(value)

    def copy(self):
        return ArrayType(self.impl.length)

class Element(Base):
    __tablename__ = 'elements'

    id = Column(Integer(),
                Sequence('element_id_seq'),
                primary_key = True)
    # ...
    myarray = Column(ArrayType())
If you are not using Pyramid just replace the pyramid_sqlalchemy's Base wrapper import with:
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()

2014-09-25

collective.angularstarter (Plone + AngularJS + Yeoman kickstarter project)

Get started with Plone + AngularJS without any of the normal headaches associated with a manual setup of useful tools that let you improve your development experience and the deploy of your application.

Since I have been using AngularJS on Plone, I decided to create a reusable starter scaffold (or something or similar) based on Yeoman that let me save precious time.

That's why I created:
This is a plugin that let you bootstrap single page web applications (or heavy Javascript logics) based on Plone+AngularJS+Yeoman.

Yeoman workflow benefits

collective.angularstarter is powered by the Yeoman workflow. If you want to see what are the Yeoman benefits due to an integration with a framework you might have a look at:
Next sections will talk about what you can build with Plone if you are not familiar with it, in particular heavily dynamic Javascript based verticalizations built with collective.angularstarter or similar techniques.

Plone

Plone is not only a CMS but a framework built with the Python programming language that let you build complex web applications, intranet, websites with strong focus on:
  • contents
  • security and sharing
  • workflow
  • searchability
You can extend the features provided by default by Plone thanks to a considerable number third party plugins.

You can see a couple of examples about you can build with Plone verticalizations with collective.angularstarter.

Coworking application

You can create custom content types for meeting rooms, private desks or common desks seats.

Basically the main object you are sharing is a folderish with metadata that let you configure the resource, for example:
  • image, title, description and rich text widget. You can describe the resource you are sharing
  • configure time slots configuration and hourly costs
  • configure time slots for group of hours with costs (ex: morning, afternoon)
  • daily cost or month cost, depending on the type of resource
  • number of available seats available in parallel
Once the resource has been published, users can buy the suitable time slots depending on the type of resource and availability.

For private or common desks you can choose to search for multiple seats for multiple days or months. For meeting rooms you can buy just one unit time slot, multiple hours, group of hours (ex: morning) or the full day.

For example if you are searching for a meeting room you can choose partial days mode and select the available day you want to select:



The infinite scrolling shows you only the days that fits what you are searching for (2 private desks)
One selected the day, you can choose one or more available slots (for example morning):
And then: buy it!
The reservation objects are based on event types, since the reservation has a start and end datetime. So you can easily perform non-expensive catalog queries in order to search for slots available.

The project itself it is more complex because Plone it is integrated with an external invoice management software, a Paypal interface that let you buy more credits and a personal user box that let the users to see invoice PDF files and other notifications.

Advanced search forms

You can also use plone as a backend for a highly dynamic single page web application.
 
You can mix Plone data with external resources provided by third party server in order to build a complex search form.

For example the main search prompt to the user a master-slave AJAX widget
where the vocabulary of slave selects depend on the value of the previous one): 

or search for different criteria:
In this particular case results will appear after you fill all the needed information, but it is quite easy to implement a live search.

If you need more info about how to create a master-select widget component with Angular you may have a look at this article http://davidemoro.blogspot.com/2014/09/angularjs-master-slave-select-with.html.

What collective.angularstarter is

The collective.angularstarter plugin is:
  1. a Plone + AngularJS kickstarter project. You can use this package when you want to develop a single page web applications powered by Angular using Plone as backend. With all the benefits of the Yeoman workflow
  2. scaffolding tool that let you extend this package, add more features and then clone it creating a more sophisticated application. You can redistribute it with another name. Or you can develop a rapid prototype of your reusable application and after create a new zopeskel or yo package generator with one or more options. The clone hack might fail in some corner case but it should help you to convert an existing package to another. Anyway if something goes wrong you can easily correct the problems by hand. I get used to apply a similar script when me or other colleagues chose a very ugly package name and then you have to rename it. Maurizio, remember?! How many days we saved with this script? 
Anyway when you install collective.angularstarter and visit the @@angularstarter browser view it shows an example of AngularJS app with enabled by default:
Here you can see how the @@angularstarter view looks like:
collective.angularstarter screenshot. Fill the input text and you'll see the page instantly updated
After that it's up to you coding with AngularJS and Plone!

Results

The following screenshot show you what happens if you analyze the network section of Firebug when you are in development mode:
or in production more:

Wait a moment! The resulting resultim bootstrap.css weights in at only 3,2 KB?! That's the power of minification and uncss tasks
A you can see you'll get (see the part 1 article of Pyramid starter seed project for further details about uncss and other tips explained):
  • html minified (experimental, disable for real project)
  • lighter images (no asset images in collective.angularstarter)
  • most popular Javascript resources automatically cdn-ified
  • css files concatenated, uncssed and minified
  • javascript concatenated and uglified
  • [update 20140926] assets automatically revved (avoid nasty caching problems)
  • ... you can do more installing additional grunt tasks
How did I did it? Basically I played with Plone's resource registrations and layers.  See https://github.com/collective/collective.angularstarter/blob/master/collective/angularstarter/browser/configure.zcml

collective.angularstarter wraps a modified Yeoman AngularJS project (browser/angular): asset paths modified, bower_components folder renamed and a couple and other local changes to the Gruntfile.js file.

Hope you'll find collective.angularstarter useful. Feedback will be very appreciated!

UPDATE 20150303: I forgot to mention that collective.angularstarter plays well with Diazo (http://docs.diazo.org/en/latest/). This way you can write a pure static mocks with lots of javascripts and then using the html as the diazo theme with a bunch xml rules, with the backend decoupled with the frontend. Easy and tested on production! Probably I'll write a new write up.

2014-09-24

Pyramid starter seed template powered by Yeoman (part 3)

In the previous articles we have seen:
  • what are the benefits of the Yeoman workflow applied to Pyramid, a traditional web framework (part 1)
  • how to install pyramid_starter_seed, a Pyramid+Yeoman flavoured starter template (part 2)
Once installed pyramid_starter_seed, we will see now:
  • how it works under the hood
  • how to manage things with grunt
  • how to create and share other templates based on pyramid_starter_seed

How it works under the hood (narrative)

pyramid_starter_seed registers only one route (home -> /) and static assets views.

The home route is associated to a view callable with a webapp/%s/index.html renderer.
from pyramid.view import view_config

@view_config(route_name='home', renderer='webapp/%s/index.html')
def my_view(request):
    return {'project': 'pyramid_starter_seed'}
Views can be associated to routes imperatively or through a scan.

Wait, there is no .html renderer handled by default in Pyramid! The pyramid_starter_seed will register a .html renderer that will replace the string token with app or dist depending on the production settings and it calls the original .pt renderer.

You can choose between production vs development mode running your pyramid app with the appropriate .ini file provided by pyramid_starter_seed.

Here you can see the relevant parts of the production.ini:
[app:main]
use = egg:pyramid_starter_seed

PRODUCTION = true
minify = dist
...
and development.ini:
[app:main]
use = egg:pyramid_starter_seed

PRODUCTION = false
minify = app

...
As you can imagine the PRODUCTION configuration tells the application to switch between production or development. The minify configuration tells the .html renderer how to construct templates paths.

Let's see how looks like the index.html template. When you write css or javascript files you want to keep things separated on different modules when you are in development mode, but in production mode you might want a unique concatenated and minified/uglified resource. Here you can see how you can do that for the Bootstrap javascripts modules you might want to enable:
<!doctype html>
<html class="no-js"
      lang="${request.locale_name}"
      tal:define="minify python:request.registry.settings['minify'];
                  production python:request.registry.settings.get('PRODUCTION', 'false') == 'true'">
...

         <tal:production tal:condition="production">
            <script src="${request.static_url('pyramid_starter_seed:webapp/%s/scripts/plugins.js' % minify)}"></script>
        </tal:production>
        <tal:not_production tal:condition="not:production">
            <script src="${request.static_url('pyramid_starter_seed:webapp/%s/bower_components/bootstrap/js/alert.js' % minify)}"></script>
            <script src="${request.static_url('pyramid_starter_seed:webapp/%s/bower_components/bootstrap/js/dropdown.js' % minify)}"></script>

        </tal:not_production>
        <!-- build:js scripts/plugins.js -->
        <tal:comment replace="nothing">
            <!-- DO NOT REMOVE this block (minifier) -->
            <script src="./bower_components/bootstrap/js/alert.js"></script>
            <script src="./bower_components/bootstrap/js/dropdown.js"></script>

        </tal:comment>
        <!-- endbuild -->
...
</html>
So in development mode you will have two separate javascript files: alert.js and dropdown.js. When you are in production mode it will served a unique concatenated and uglyfied scripts/plugins.js. At first time it might seem a bit complicated or a task with a too verbose setup, but it is a simple and very powerful mechanism.

For example: do you want to add another javascript file to the plugins.js bundle? Just add two lines and you are ok!

It works without any other configuration thanks to the start and end comments blocks build:js and endbuild that groups assets groups.

Simple, isn't it? Same thing for css files.

And if you have to include images is even more simple with:
<img class="logo img-responsive" src="${request.static_url('pyramid_starter_seed:webapp/%s/images/pyramid.png' % minify)}" alt="pyramid web framework">

How to manage things with grunt

The concatenation, minification/uglyfication and image optimization (with other tasks specified in the grunt's pipeline) it is automatically performed just running the following command:
$ grunt build
This is just one of the possible implementations. Feel free to contribute and improve pyramid_starter_seed.

How to clone pyramid_starter_seed 

Fetch pyramid_starter_seed, personalize it and then clone it!

Pyramid starter seed can be fetched, personalized and released with another name. So other developer can bootstrap, build, release and distribute their own starter templates without having to write a new package template generator. For example you could create a more opinionated starter seed based on SQLAlchemy, ZODB nosql or powered by a javascript framework like AngularJS and so on.

The clone method should speed up the process of creation of new more evoluted packages based on Pyramid, also people that are not keen on writing their own reusable scaffold templates.

So if you want to release your own customized template based on pyramid_starter_seed you'll have to call a console script named pyramid_starter_seed_clone with the following syntax (obviously you'll have to call this command outside the root directory of pyramid_starter_seed):
$ YOUR_VIRTUALENV_PYTHON_PATH/bin/pyramid_starter_seed_clone new_template
and you'll get as a result a perfect renamed clone new_template:
A new starter template cloned from pyramid_starter_seed
If you provide tests you can check immediately if something went wrong during the cloning process.

In effect the clone console script it might not work in some corner cases just in case you choose a new package name that contains reserved words or the name of a dependency of your plugin, but it should be quite easy to fix by hand or improving the console script. Anyway this mechanism has been tested several years: I have built this script years ago because I was fed up with ugly package names chose by me or other colleagues of mine and allowed me to save a lot of time.


If you want to disable the console script on your new template (for example: new_template_clone) drop from setup.py the console script configuration.

So it sounds like a viral extension mechanism (I hope).

End of story?


In the next future I'd like to create a new Pyramid starter seed for single page web apps based on SQLAlchemy and powered by AngularJS. So if you similar plans... together is better: if you want to share your thoughts, improvements, feedback in general, or if you are going to create your own template based on pyramid_starter_seed please contact me (Twitter, Google+, Linkedin)!


Anyway I hope you'll save time with pyramid_starter_seed.

Links

2014-09-18

Pyramid starter seed template powered by Yeoman (part 2)

In the previous blog post we have seen what are the benefits of using the Yeoman workflow fully integrated with a web development framework like Pyramid. See:

Now we'll add more technical details about:
  • how to install pyramid_starter_seed and its prerequisites

How to install pyramid_starter_seed

    Prerequisites

    As you can imagine, nodejs tools are required.

    I strongly suggest to:
    • avoid system packages because they are too old
    • install nodejs with nvm (Node Version Manager)
    I won't cover the nodejs installation but it is quite simple if you follow these instructions provided by this useful link:
    The nvm achronym stands for NodeJS Version Manager. Once installed nvm, installing nodejs it is as simple as typing nvm install VERSION (at this time of writing 0.10.32).

    Nodejs is shipped with the command line utility named npm (Nodejs Package Manager) and we will use npm for installing what we need.

    We need to install our global (-g option) dev dependencies, so just type:
    $ npm install -g bower
    $ npm install -g grunt-cli
    $ npm install -g karma

    Starter seed project installation

    Create an isolated Python environment as explained in the official Pyramid documentation and instal Pyramid.

    Once installed you can clone pyramid_starter_seed from github:
    $ git clone git@github.com:davidemoro/pyramid_starter_seed.git
    $ cd pyramid_starter_seed
    $ YOUR_VIRTUALENV_PYTHON_PATH/bin/python setup.py develop
    Not finished yet, continue.

    Yeoman initialization

    Go to the folder where it lives our Yeoman project and initialize it.

    These are the standard commands (but, wait a moment, see the "Notes and known issues" subsection):
    $ cd pyramid_starter_seed/webapp
    $ bower install
    $ npm install --loglevel verbose
    Known issues:
    • if you are behind a proxy you'll have to configure properly npm
    • if you have a slow internet connection you might experience timeout problems.

    Build phase

    Just type:
    $ grunt
    and... probably it will fail because of a couple of known issues shipped with the latest version of generator-webapp or its dependencies.

    Probably these issues will be fixed in newer generator-webapp releases. However here it is how to solve these problems, so don't worry:
    1. grunt-contrib-imagemin fix
      Problem with grunt:
      Warning: Running "imagemin:dist" (imagemin) task
      Warning: Bad argument Use --force to continue.
      Solution:
      $ npm cache clean
      $ rm -fR node_modules       # not sure it is needed, don't remember
      $ npm install grunt-contrib-imagemin
    2. Mocha/PhantomJS issue
    Problem with Mocha/PhantomJS launching grunt
    Warning: PhantomJS timed out, possibly due to a missing Mocha run() call. Use --force to continue.
    Solution:
    $ cd test
    $ bower install
    Run bower install in the test directory of webapp (pyramid_starter_seed/webapp/test). This is a known issue, see https://github.com/yeoman/generator-webapp/issues/446.

    Run you Pyramid app

    Now can choose to run Pyramid in development or production mode.
    Just type:
    $ YOUR_VIRTUALENV_PYTHON_PATH/bin/pserve development.ini
    or:
    $ YOUR_VIRTUALENV_PYTHON_PATH/bin/pserve production.ini
    Done!


    In the next blog post with topic Pyramid + Yeoman (coming soon) I'm going to talk about:
    • how to manage things with grunt and personalize pyramid_starter_seed registering other assets
    • how to clone pyramid_starter_seed. Yes, you can easily customize it creating something of more sophisticated and create your own starter seed with another name. Without having to write a package generator
     So stay tuned..

    Links

    2014-09-16

    Pyramid starter seed template powered by Yeoman (part 1)

    Book of the month I'm reading this summer: Pylons/Pyramid (http://docs.pylonsproject.org/en/latest).


    Pyramid (http://www.pylonsproject.org) is a minimal Python-based web development framework that let you "start small and finish big".

    It stole a lot of (good) ideas and concepts from other mature Python web frameworks and it is build with the pluggable and extensible concepts in mind. Read: no need to fork applications.

    Furthermore Pyramid is database and template engine agnostic: you are free.

    From the very beginning Pyramid allows you to become productive quickly. So why not start with something of useful?

    Pyramid + Yeoman

    The goal of this experiment is integrate yeoman with Pyramid (or other frameworks like NodeJs/Express with AngularJS or Plone as already did), preserving the yeoman's workflow.

    UPDATE 20140926: here you can see a Plone + AngularJS + Yeoman article (collective.angularstarter)

    In this article I'll talk about what are the benefits you get integrating your Pyramid app with Yeoman, in future posts I'll discuss how they work under the hood with additional technical details omitted here (each used component deserves an entire blog post).

    Yeoman
    You might wonder why? Because of the importance of tooling. Since it is very important build an effective developer tooling ecosystem, I want to integrate the simple starter demo app with commonly used tools to help you stay productive. So this simple application prototype it is just an experiment that should help you to integrate with modern web development tools provided by the yeoman workflow stack (http://yeoman.io).

    Choosing the right tools is very important for the best develop experience and I cannot work anymore without Yeoman, especially when coding with Javascript.

    Grunt
    Yeoman it is internally based on three important components (nodejs powered):
    • yo, scaffolding tool like pcreate, paster or zopeskel. It is widely adopted by a large and trasversal community
    • grunt, system used for build, preview and test your software. Gulp is another popular option
    • bower, used for dependency management, so that you no longer have to manually download and manage your scripts
    Bower

    So with the yeoman's tools you can just code, avoid annoying repetitive tasks and don't worry about:
    • javascript testing setup
    • javascript code linting
    • javascript/css minification and merging
    • image minification
    • html minification
    • switch to CDN versions of you vendor plugins in production mode
    • auto-reload browser
    • much much more
    So let's see together what happened to our pyramid starter demo template created with pcreate -t starter integrated with a yeoman's generator-webapp project.

    The result will be a Pyramid starter seed project integrated with modern non Python-based web development tools.

    Goals

    Management of third party assets

    You no longer have to manually download and manage your scripts with the Bower package manager.

    From http://bower.io:
    """Bower works by fetching and installing packages from all over, taking care of hunting, finding, downloading, and saving the stuff you’re looking for."""
    So just type something like: bower install angular-translate --save and you'll get the rigth resource with pinning support.

    Tasks automation

    Automation, automation, automation.

    From http://gruntjs.com:
    """Why use a task runner? In one word: automation. The less work you have to do when performing repetitive tasks like minification, compilation, unit testing, linting, etc, the easier your job becomes. After you've configured it, a task runner can do most of that mundane work for you—and your team—with basically zero effort."""
    Examples:
    • grunt serve
    • grunt test
    • grunt build
    • grunt YOUR TASK 
    • etc

    Jslint

    No more deploy Javascript code with bad indentation, syntax errors or bad code practices.

    All syntax errors or bad practise will be found.

    Image minification

    The build process will detect and minify automatically all your asset images.

    Uncss task

    Modern (and heavy) UI frameworks like Twitter Bootstrap provide an excellent solution for prototyping your initial project, but most of the times you are using a very minimal subset of their functionalities.

    https://twitter.com/davidemoroThis inspiring Addy Osmani's blog post helps you to remove unused css in your pages with a grunt task named grunt-uncss (https://github.com/addyosmani/grunt-uncss):
    The original not-minified bootstrap.css weights in at 120 kB before removing unused rule.

    Css concat and minification

    You can split your css code into different files and then the build process will concat and minify them creating a unique app.css file. This way you write modular and better readable css files, reducing the number of browser requests.

    The theme.css file is quite small but in real projects you can save more. In this case:
    The configured build pipeline is concat, uncss and cssmin. 122.85 kB (original bootstrap.css) -> 4.64 kB (uncss) -> 3.45 kB (minification)

    Automatic CDN-ification

    It is handy using unminified versions of third party javascript libraries during development and switch to CDN versions in production mode with well known benefits for your website.

    Don't worry: the cdnify task will take care about this boring issue. Automatically.

    You save a boring manual and error-prone configuration.

    Composable bootstrap.js version

    The Pyramid starter project is based on Twitter Bootstrap.

    Twitter Bootstrap
    Depending on your project you can load the whole Twitter Bootstrap Javascript code at once or including individual plugins.

    As you can see the Javascript component of Twitter Bootstrap is very modular: http://getbootstrap.com/javascript. So if you don't use a particular feature, just don't include it.

    This way in development mode you will have all individual plugins splitted in different files, in production it will served a unique concatenated and minified Javascript file built automatically.

    So if you just need alert.js and dropdown.js you can get a 2.79 kB plugins.js:

    The concatenation of alert.js and dropdown.js produces a 7.06 kB, that weight in at 2.79 kB after minification instead of the 8.9 kB (gzipped) bootstrap-min.js corresponding to not gzipped 27.2 kB.

    Html (template) minification

    Since the ZPT/Chameleon templating language is an extension of HTML with xml syntax,

    Brower are able to display unrendered ZPT/Chameleon templates
    theorically it can play well with html minificators.

    I know, template minification can lead to potential unexpected problems due to minification issues on template files... but this is my personal playground, so let me play please!

    So... why not switch to a pre-compiled minified template of your ZPT/Chameleon files when you are in "production mode"?

    Obviously during development you will use the original template files.

    The interesting side of this approach is that there is no overhead at response time, since the minification task runs just one time before deploying your application. It might be an option if you want just your html minified and you cannot feasibly add to your site or project additional optimization tools at web server level.

    Anyway I have tried this mad experiment and... if you don't use too aggressive minification params, it seems to work fine with good results. Try it at your own risk or just disable it. Here you can the effects on the generated index.html used in production:
    Template minified (7.62 kB -> 4.16 kB)

    Result: a lighter Pyramid

    Same results but a lighter Pyramid app:

    Let's see how it behave the standard Pyramid starter project:
    Standard Pyramid starter project (production.ini)
    And the Pyramid starter seed:
    Pyramid starter seed (production.ini)
    As you can see the seed version is ~38 Kb smaller and more performant.

    Useful links

    That's all?

    No, you can do more, for example:
    • reduce the  number or requests (for example you can merge vendor.css and app.css)
    • create and keep updated css sprites with grunt (https://github.com/Ensighten/grunt-spritesmith)
    • manage and upload all your assets to professional services like Amazon AWS (for example you can serve up all your images, styles and scripts from a S3 bucket + CloudFront). This way Pyramid will be able to handle more requests. Pyramid let you put static media on a separate webserver during production with static_url() in conjunction with add_static_view(), without having to change your templates code
    • generate static gzipped assets with Grunt and let your webserver serve them
    • install and configure dedicated performance modules at webserver level (Apache's mod_pagespeed)
    Let me know what you think about that, please. Hope soon I will manage to write the second part of this blog post explaining how I did it. In the mean time you can:

    Links

    AngularJS master-slave select with $location.search() params

    In this post we will see how to implement a simple master-slave select controls in AngularJS.

    This kind of controls are useful when your users selection depends on the previous select.

    For example:
    • continent (Europe, America, Asia, Oceania)
    • country (USA, ...)
    • state (California, ...)
    • etc
    The implementation is a bit more complex because I want to support urls that reflect each user input (example: /search -> /search?continent=America -> /search?continent=America&country=USA -> etc). This way the user will be able to share the url of an intermediate state of the application without having to retype all form inputs. Sometimes if you paste the url of a single page application heavily based on AJAX you loose the context.

    The key concepts based on this kind of implementation are:
    • $location.search()
    • use reloadOnSearch: false in conjunction on $routeProvider.when. It just updates query params without a reload

    Code

    In bold the relevant blocks.

    Controller

    'use strict';

    angular
      .module('angularApp', []);
     
      angular.module('angularApp')
      .controller('MainCtrl', ['$scope', '$location', 'queryFactory', function ($scope, $location, queryFactory) {

        var params = $location.search();   // the current params dict

        var initParams = function () {
          /* init controllers conf params */
          angular.forEach(params, function (value, key) {
            if ($scope.conf.search[key]) {
              $scope.conf.search[key].selected = value;
            }
          });
        };

        /* The model configuration */
        $scope.conf = {
          initsearch: 'select1',   // the master of all selects
          search: {
            select1: {
              values: [], selected: undefined, slave: 'select2', updater: queryFactory.getSelect1
            },
            select2: {
              values: [], selected: undefined, slave: undefined, updater: queryFactory.getSelect2
            },
          }
        };

        /* Master select initialization (lookup of options) */  $scope.conf.search[$scope.conf.initsearch].updater($scope.conf.search)
          .success(function(data) {
            $scope.conf.search[$scope.conf.initsearch].values = data;
          });

        /* Init $scope.conf.search depending on search params.
         * */

        initParams();

        // not yet available $watchGroup on angularjs 1.2
        angular.forEach($scope.conf.search, function (value, key) { // jshint ignore:line
          $scope.$watch('conf.search.' + key + '.selected', function(newValue, oldValue) { // jshint ignore:line
            var slave = $scope.conf.search[key].slave;

            params[key] = newValue;
            $location.search(params);

            if (newValue) {
              if ($scope.conf.search[slave]) {
                $scope.conf.search[slave].updater($scope.conf.search)
                  .success(function(data) {
                    $scope.conf.search[slave].values = data;
                  });
              }
              else {
                // slave is undefined
                if ($scope.conf.search[key].selected) {
                  // end of chain, do something
                  // TODO
                }
              }
            }
            else {
              // removed selected and values
              if ($scope.conf.search[slave]) {
                $scope.conf.search[slave].values = [];
                if ($scope.conf.search[slave].selected) {
                  $scope.conf.search[slave].selected = undefined;
                }
              }
            }
          });

        });
      }]);

    Template

    <!DOCTYPE html>
    <html ng-app="angularApp">

      <head>
        <script data-require="angular.js@*" data-semver="1.2.22" src="https://code.angularjs.org/1.2.22/angular.js"></script>
        <link href="style.css" rel="stylesheet" />
        <script src="script.js"></script>
      </head>

      <body>
        <div ng-controller="MainCtrl">
          <h2>Master select widget example with AngularJS</h2>
          Launch in a separate window in order to see the .search() usage: click on the blue button on your top-right of the demo.<br/>
         
          <select ng-model="conf.search.select1.selected" ng-options="item.id as item.title for item in conf.search.select1.values">
            <option value="">--</option>
          </select>

          <select ng-model="conf.search.select2.selected" ng-options="item.id as item.title for item in conf.search.select2.values">
            <option value="">--</option>
          </select>

        </div>
      </body>

    </html>

    Service (performs $http queries)

    angular.module('angularApp')
      .factory('queryFactory', ['$q', function ($q) {
        // Service logic (mock)
        // You should put here your $http calls, for a working example see http://davidemoro.blogspot.it/2014/09/angularjs-how-to-test-http-calls.html


        // Public API here
        return {
          getSelect1: function (conf) {  // jshint ignore:line
            var promise = $q.when([{id: '1', title: '1'}, {id: '2', title: '2'}]);
            promise.success = function(fn) {
              promise.then(function(response) {
                fn(response);
              });
              return promise;
            };

            return promise;
          },
          getSelect2: function (conf) { // jshint ignore:line
            var promise;
            if (conf['select1'].selected === '1') {
                promise = $q.when([{id: '1.1', title: '1.1'}, {id: '1.2', title: '1.2'}]);
            } else {
                promise = $q.when([{id: '2.1', title: '2.1'}, {id: '2.2', title: '2.2'}]);
            }
            promise.success = function(fn) {
              promise.then(function(response) {
                fn(response);
              });
              return promise;
            };

            return promise;
          }
        };
      }]);

    How to implement $http tests

    You can see an example of tests for a service $http-based decoupled from the controller logic:

    Results

    Selecting 1 on the first select you'll get 1.X values on the second one and so on

    Plunkr demo

    I have made a Plunkr demo available at this url:
    If you open the Plunkr link in fullscreen mode you'll see the query params will change for each input. If you copy and paste the intermediate url you'll get still a compiled form.

    Generic master-slave directive?

    I'd like to implement a generic and more reusable master-slave directive. What are your suggestions about the best design strategy and how to build things following the Angular way?

    2014-09-11

    AngularJS - how to test $http calls (with tests example)


    AngularJS templates no longer automatically unwrap promises (AngularJS >= 1.2). See https://code.angularjs.org/1.2.24/docs/guide/migration#templates-no-longer-automatically-unwrap-promises

    This feature is now deprecated but if you absolutely need it, it can be reenabled for now via the $parseProvider.unwrapPromises(true) API.

    In this post we will see how to:
    • separate data lookup from controllers logic. It is way way better implement $http calls into dedicated angular services for separation of concerns, good design and software testability
    • how to test $http calls with mocks
    This is just a reminder for me but I hope other developers will save time if they don't figure out why their code does not how expected.

    Meaningful code in bold, feedback please!

    Controller

    Controller's code

    scripts/controllers/car.js:
    'use strict';

    angular.module('angularApp')
      .controller('CarCtrl', ['$scope', '$routeParams', 'carFactory', function ($scope, $routeParams, carFactory) {

        // route params (example: )
        $scope.params = $routeParams;

        carFactory.get($scope.params.carId)
          .success(function(data) {
            $scope.data = data;
          });
      }]);

    Controller's test code

    test/spec/controllers/car.js:
    'use strict';

    describe('Controller: CarCtrl', function () {

      // load the controller's module
      beforeEach(module('angularApp'));

      var CarCtrl,
        scope,
        success;

      // Initialize the controller and a mock scope
      beforeEach(inject(function ($controller, $rootScope, $q) {

        var promise = $q.when({id: '1', title: 'Audi'});
        promise.success = function(fn) {
          promise.then(function(response) {
            fn(response);
          });
          return promise;
        };

        scope = $rootScope.$new();
        CarCtrl = $controller('CarCtrl', {
          $scope: scope,
          $routeParams: {carId: 'audi'},
          carFactory: {get: function () {return promise;}}
        });
      }));


      it('should have routeParams into params', function () {
        expect(scope.params.carId).toBe('audi');
      });

      it('should have carFactory data', function () {
        scope.$digest();
        expect(!!scope.data).toBe(true);
      });
    });

    Service

    Service code

    The base url of my remote endpoint can be injected with:
    .value('apiPrefix', 'http://localhost:3001/')
    if you are using a service and it is a value shared with other components.
    Otherwise can use a configurable service (provider).

    Using value or a configurable service helps you to mock things injecting different endpoint urls during development.

    scripts/services/carfactory.js):
    'use strict';

    angular.module('angularApp')
      .factory('carFactory', ['$http', 'apiPrefix', function ($http, apiPrefix) {
        // Service logic
        // ...
        var base = apiPrefix ? apiPrefix : '';

        // Public API here
        return {
          get: function (carId) {
            var promise;

            promise = $http.jsonp(base + 'cars/' + carId + '?callback=JSON_CALLBACK');
            return promise;
          }
        };
      }]);

    Service test code

    test/spec/services/carfactory.js:
    'use strict';

    describe('Service: carFactory', function () {

      // load the service's module
      beforeEach(module('angularApp'));

      // instantiate service
      var carFactory, $httpBackend, apiPrefix;
      beforeEach(inject(function (_carFactory_, _$httpBackend_, _apiPrefix_) {
        carFactory = _carFactory_;
        $httpBackend = _$httpBackend_;
        apiPrefix = _apiPrefix_;

        $httpBackend.whenJSONP(apiPrefix + 'cars/audi?callback=JSON_CALLBACK').respond({
          id: 'audi',
          title: 'Audi'
        });
      }));

      afterEach(function() {
        $httpBackend.verifyNoOutstandingExpectation();
        $httpBackend.verifyNoOutstandingRequest();
      });

      it('should do something', function () {
        expect(!!carFactory).toBe(true);
      });

      it('test get data', function () {
        var data, promise;
        promise = carFactory.get('audi');
        promise.success(function(res) {
          data = res;
        });
        $httpBackend.flush();

        expect(data.title).toBe('Audi');
      });

    });

    2014-04-16

    Keystonejs CMS

    I've just installed KeystoneJS (http://keystonejs.com), a Nodejs CMS based on Mongodb and optional storage policies for binaries like images or files.

    First impression? I like it :)

    You only need to install NodeJs (NVM is your friend) and a MongoDB instance. After that you will be able to install it locally on your machine typing a couple of commands if you use the official generator-keystone Yeoman generator. Otherwise you can start from scratch as well.

    After that no extra work is needed, open http://localhost:3000 in your browser and add new categories or blob posts. Or if you want you can deploy and host your application into the cloud on Paas platform.


    Shortly:
    • you can create your own content types with a defined schema
    • admin UI automatically generated (add forms, edit forms)
    • supports logged in users and session management
    • form validations
    • dynamic routes
    • email sending
    • very basic workflow support (just what you need for a simple publishing experience)
    • you can switch to alternative template engines (default: Jade... nooo, I don't like it!)
    • easy customizable templates for object views and listing (just html)
    • mobile ready
    • indexing and search support (no advanced features like pdf-text extraction and indexing)
    • Paas ready for hosting solutions
    • no folder-aware conceps (for example blog items are just reachable from the /blob/posts/blog-slug, you cannot nest items in folders. But it is ok for a wide range of applications)
    • editable slug urls
    • different storage policies for images and files
      • CloudinaryImage(s) for automatically manages images stored in Cloudinary, including uploading, resizing and deleting. See http://cloudinary.com/
      • LocalFile, this field type is not compatible with PAAS hosts like Heroku https://www.heroku.com because it relies on the local file system
      • S3, automatically manages files stored in Amazon S3, including uploading and deleting (http://aws.amazon.com/s3/)
      • AzureFile, automatically manages files stored in Windows Azure Storage, including uploading and deleting
    Not supported (or not yet):
    • advanced, configurable and generic workflow engine
    • field level security on content types
    • configurable portlet items or user defined boxes
    • not sure, no out of the box minification assets solutions for css and javascripts when you switch to production mode? (more or less something like already discussed on this old blog post hack http://davidemoro.blogspot.it/2013/08/yeoman-express-and-angularjs.html)
    So if you compare its features with enterprise level CMS like Plone (http://plone.org) you might think that it is a very simple CMS solution.

    But you should not consider it just like a CMS but as a simple but flexible framework for creating NodeJS/ExpressJS/MongoDB based generic web applications with an autogenerated UI. You can save a lot of time!

    It is a very young project and I don't know if it will become the nodejs killer app but the approach sounds very good and probably it is the most advanced CMS solution based on NodeJS at this time of writing.

    Very cool guys, keep up the good work!

    2014-04-07

    yeoman generator mongodb interactive installer


    In the previous post we have seen how to create your own yeoman generators using custom _.templateSettings rules.

    However if you want to dive into yeoman generators you can play with generator-generator, have a look at the code of existing generators and... why not? Create your own whatever-you-want yeoman generator!

    I chose to implement a simple generator, based on custom _.templateSettings seen in the previous post, that let you automate the mongodb installation.

    Its name is generator-mongodb (https://github.com/davidemoro/generator-mongodb).

    Useful or not useful, this is my first whatever-i-want-generator, made just for fun. There is no tests coverage (I got stuck on a problem but the solution is on his way) and I won't maintain it. So that's just an example, anyway I hope you'll find useful hints.

    How it is implemented this generator under the hood

    This generator is internally based on the buildout build system (http://www.buildout.org).

    What is buildout? Buildout is a widely Python-based build system for creating, assembling and deploying applications from multiple parts, some of which may be non-Python-based. It lets you create a buildout configuration and reproduce the same software later.

    Don't worry: we are not going to write any line of python code! We need python because it is required by our install system).

    Buildout configurations are text-based, versionable, fine-grained reproducible environments that you can share with your team or create software installers (one example is the Plone CMS unified installer - see http://plone.org/products/plone).

    There are dozens of published buildout recipes ready to be used. For example:
    So with buildout you can enhance your productivity and automate your build process providind a light out of the box solution, instead of relying on virtual machines and huge downloads.

    This helps when introducing multiple developers to a project: you can reduce setup times and enhance productivity. This way we can ensure that the whole team we are collaborating with has the right environment and you can save a lot of time.

    Another example: if you are the creator of a nice open source project (for example a Node.js CMS based on MongoDB), you shoud consider that your download&install page will be visited by people just interested in evaluating a CMS or specific app solution without any knowledge or interest about the technology involved because they want to install and play with it locally. So you can provide simpler install instructions thanks to buildout (or any other automated system), for example you can replace "be sure you have installed node.js version x.y.z and MongoDB" with "checkout this folder and run this command: done!".

    If you don't provide similar easy way to install your solution you are assuming that your users are comfortable with the underlying technologies, they know how to install the external components you need to run your software.  - If you want to explore a different approach you can have a look at http://www.vagrantup.com/ with its virtual environments.

    Let's come back to the buildout. Buildout files can be extended, we can build different profiles (production, development, etc) and a very simple buildout.cfg file looks like the following:
    [buildout]
    parts = mongodb

    [mongodb]
    recipe = rod.recipe.mongodb
    darwin-32bit-url = http://downloads.mongodb.org/osx/mongodb-osx-i386-1.6.5.tgz
    ...
    How it works? Just one command and the buildout process starts, for example:
    $ ./bin/buildout [-c buildout.cfg]
    So thanks to buildout and existing recipes our generator workflow will be very simple:
    1. read one or more user inputs (version of mongodb, but you can add more advanced options if you want)
    2. render the buildout.cfg following the steps described here http://davidemoro.blogspot.it/2014/03/yeoman-generator-template-settings-lodash-underscore-js.html
    3. setup the right environment (it creates a local and isolated python environment that does not pollute the global python installed on your system)
    4. runs the buildout command for you
    At the end of the process you'll find a local environment with all you need. Since each buildout folder is an isolated, self-contained environment, you can install also different MongoDB versions on your machine as well.

    For advanced option you can see the rod.recipe.mongodb reference: https://pypi.python.org/pypi/rod.recipe.mongodb.

    How to use generator-mongodb

    Be sure you have installed node.js (see https://github.com/creationix/nvm), python and virtualenv (>= 1.11.4).

    If you are not comfortable with python you can install it with a couple of commands:
    $ sudo apt-get install python2.7
    $ sudo apt-get install python-pip
    $ [update 20140410:  not safe, follow the official doc] sudo pip install --upgrade virtualenv
    The install yo:
    $ npm install -g yo
    If generator-mongodb were released, installing it it would as simple as typing:
    $ npm install -g generator-mongodb
    Since this package is not released, you'll have to clone the github repository:
    $ git clone git@github.com:davidemoro/generator-mongodb.git
    $ cd generator-mongodb
    $ npm install
    $ npm link
    $ cd
    Then move where you want to install your local mongodb environment and type:
    $ mkdir yourmongodir
    $ cd yourmongodir
    $ yo mongodb
    And you'll see something of similar.

    Type the mongodb version and wait for the end of the install process (it might take a while).

    Once finished you'll find the mongodb executables:
    $ ls bin -1 | grep mongo
    mongo
    mongod
    ...
    Update 20140415: actually ./bin/mongod doesn't work because I missed to set the db path option in buildout.cfg. Just type mkdir -p data/db in your buildout root and launch ./bin/mongod --dbpath data/db. If you configure well the mongodb recipe it will work without typing any option.

    Done!

    Links