Dr. Jekyll and Mr. GitHub Pages

Time ago I started using the GitHub Pages service just for fun:

• http://davidemoro.github.io/tour_knight_angularjs (just my first demo app based on AngularJS)

With GitHub Pages you can write documentation, create a landing page for your account or organization and create unlimited project sites for you and your projects.

Websites, blogs, landing pages, live demos.

Things are hosted directly from your GitHub repo. So just edit, push and your changes are live (and under version control and easy collaboration with other team members).

You can generate a site using one of the pre-built themes or create a site from scratch.

Who is using GitHub Pages?

• Foundation - http://foundation.zurb.com/
• Twitter bootstrap - http://getbootstrap.com/
• Yeoman - http://yeoman.io/
• Octopress - http://octopress.org/
• many others

We'll cover these topics in this article:

• type of sites you can build with GitHub Pages
• user or organization site
• project site
• site generation methods
• pre-built themes
• from scratch
• advanced blogging tools (with Jekyll)

User or organization site from scratch (master)

You can create your personal page or organization site for free.

First: create a new repository named username.github.io, where username is your username (or organization name) on GitHub.

Then clone your repo:

$ git clone https://github.com/username/username.github.io
$ cd username.github.io

Create a demo hello world page:

$ echo "Hello World" > index.html

Push:

$ git add --all
$ git commit -m "Initial commit"
$ git push

Visit your page (you'll have to wait a bit the very first time you update your site):

• http://username.github.io

Project site (gh-pages branch)

For Project sites, you have the option to generate a site with one of the available pre-built themes, or to create a site from scratch.

Generate a site (pre-built themes)

Go to the settings panel of you repository and click on the Automatic Page Generator.

You can use the editor to add content and then pick a theme and publish!

Start from scratch

Follow this guide:

• https://help.github.com/articles/creating-project-pages-manually/

Jekyll

GitHub pages can be used in conjunction with Jekyll http://jekyllrb.com/.

What you can do with Jekyll:

"""Using Jekyll, you can blog using beautiful Markdown syntax, and without having to deal with any databases.

One of Jekyll’s best aspects is that it is “blog aware”. What does this mean, exactly? Well, simply put, it means that blogging is baked into Jekyll’s functionality. If you write articles and publish them online, this means that you can publish and maintain a blog simply by managing a folder of text-files on your computer. Compared to the hassle of configuring and maintaining databases and web-based CMS systems, this will be a welcome change!"""

Here you can find some useful resources about Jekyll and GitHub Pages:

• https://help.github.com/articles/using-jekyll-with-pages/ 
• https://help.github.com/articles/troubleshooting-github-pages-build-failures/

Prerequisites: Ruby. Your Ruby version should begin with 1.9.3 or 2.0.0.

User or organization site

Bootstrap a new dir name USER_OR_ORGANIZATION.github.io: 

$ jekyll new USER_OR_ORGANIZATION.github.io
$ cd USER_OR_ORGANIZATION.github.io

Basically you'll have to install Jekyll and Bundle as suggested on the GitHub official doc. Create a Gemfile file with this content:

source 'https://rubygems.org'
gem 'github-pages'

Check your url and baseurl options of your _config.yml file.
_config.yml for a project/repository site:
        ...
        baseurl: "/REPOSITORY" # the subpath of your site, e.g. /blog/
        url: "http://USER_OR_ORGANIZATION.github.io" # the base hostname & protocol for your site

Invoke Jekyll with Bundler:

$ bundle install
$ bundle exec jekyll build

The .gitignore file is already generated.

Now you can create the repo online on github and push your contents (.gitignore file already generated). Before pushing it is suggested to run Jekyll locally:

$ bundle exec jekyll serve

Result:

Now you can add new posts, change your page title, etc. See the _config.yml dir, the _posts dir and the official documentation online.

Now you can push your changes and they will display online.

Repo/Project site (with Jekyll)

You should create a gh-pages branch of your repository.

Follow this guide:

• https://help.github.com/articles/creating-project-pages-manually/

but instead of putting plain html files, initialize your branch with Jekyll (with bundle exec jekyll new .).

And then check your _config.yml. The user/organization site requires a different setup:
        ...
        baseurl: "" # the subpath of your site, e.g. /blog/
        baseurl: "/REPOSITORY" # the subpath of your site, e.g. /blog/
        url: "http://USER_OR_ORGANIZATION.github.io" # the base hostname & protocol for your site

Now you can build things and upload to GitHub. Results will displayed on http://USER_OR_ORGANIZATION.github.io/REPOSITORY.

Jekyll troubleshooting

Error 1 (LoadError):

/var/lib/gems/1.9.1/gems/jekyll-2.5.1/bin/jekyll:21:in `block in <top (required)>': cannot load such file -- jekyll/version (LoadError)

Solution (see https://github.com/jekyll/jekyll/issues/3084):

$ bundle exec jekyll COMMAND ("serve" or "new .")

Error 2 (Gemfile does not exist and directory is not empty)

I got two strange errors initializing an empty dir with Jekyll with the "bundle exec jekyll new ." command. Something like "error: Gemfile does not exist". Ok, let's create a Gemfile and "error: directory is not empty". If worked for me "bundle exec jekyll new USER_OR_ORGANIZATION.github.io"

Linuxday2014 Torino

Il 25 ottobre ho avuto l'onore di presentare un talk al Linuxday2014 a Torino presso i locali del Politecnico dal titolo...

Strumenti  per lo sviluppo web

Automatizza operazioni ripetitive e noiose:

Migliora la tua esperienza di sviluppatore web adottando strumenti avanzati per automatizzare operazioni ripetitive e noiose: scaffolding nuovi progetti, css performance tooling, live reload browser, javascript linting, merge e minificazione asset, uncss, cdn, integrazione con framework web e tematiche legate al deploy.

L'ospite d'onore del talk è stato Yeoman, con numerosi cenni a Bower, Npm, Grunt e naturalmente Yo (e i suoi generatori più famosi come per esempio generator-webapp e generator-angular).

Inoltre per terminare sono stati presentati una serie di esempi di integrazioni di Yeoman con framework web di diverse tipologie (basati su Python - come non citare Plone o Pyramid - e NodeJS - portando come esempio il framework loopback.io con la sua AngularJS SDK) ed illustrate alcune delle (molte) tecniche avanzate per l'ottimizzazione del front end (es: grunt-uncss).

Insomma, argomenti molto vari ed è stata dura riuscire a condensare tutto in una sola ora di talk a disposizione!

Per quanto riguarda gli altri talk posso dire che il livello era molto alto e, purtroppo, avrei voluto assistere ad altre due presentazioni ma erano in parallelo con la mia (Arduino e virtualizzazione con Docker)... Da segnalare una buona partecipazione da parte dei pinguini torinesi, l'ottima organizzazione, location veramente azzeccata (comoda con i trasporti, aule capienti, buona acustica), la simpatia dei volontari allo stand dei gadget. Talk preferito? Scrivere driver per il kernel, presentato dal simpaticissimo e ultra competente kernel hacker Alessandro Rubini (complimenti!!!).

Come promesso ecco le slide.

Slide

Le slide sono disponibili su https://speakerdeck.com/davidemoro/linux-day-2014-torino-strumenti-per-lo-sviluppo-web

Domande e risposte

Riporto una serie di domande e risposte emerse alla fine del talk, purtroppo c'è stato poco tempo a disposizione per portare avanti la discussione e ho aggiunto delle integrazioni e precisazioni postume con aggiunta di riferimenti. Vado a memoria e spero di non aver dimenticato nulla.

Q: La tecnica dell'uncss sembra essere un anti-pattern rispetto a soluzioni Compass-sass
A: Mmm, no. Questo pattern è stato recentemente presentato alla csseuconf dall'autore di grunt-uncss che lavora presso Google, ma il concetto esisteva già ed è stato già applicato in diversi contesti. Per maggiori informazioni puoi approfondire la questione leggendo l'articolo originale dell'autore di grunt-uncss http://addyosmani.com/blog/removing-unused-css/

Q: Si poteva snellire allo stesso modo il css di Twitter boostrap con Compass-Sass sfruttando la modularità di Twitter Bootstrap (pattern migliore)
A: Sì, ma solo parzialmente. Grunt-uncss è uno strumento diverso e consente comunque di ottenere delle performance migliori. Inolte uncss può essere applicato a qualsiasi css o framework UI indipendentemente dal supporto o meno di sass. Per maggiori informazioni fai riferimento all'articolo dell'autore http://addyosmani.com/blog/removing-unused-css/

Q: Hai parlato spesso di AngularJS nel tuo talk, c'è una correlazione con Yeoman o è solo un caso? Perchè non emberjs o altri framework.
A: No, non c'è correlazione, giusto per fare un esempio con del codice Javascript. Yeoman è totalmente indipendente dal framework Javascript usato. Avrei benissimo potuto usare una qualsiasi altra tecnologia. La scelta di riportare AngularJS come esempio è ricaduta semplicemente in base a una mia preferenza personale, ma i concetti visti sono assolutamente generici.

Q: Yo funziona anche sotto Windows?

A: In realtà ho la fortuna di usare solo Linux da diversi anni per cui non so esprimermi di preciso, ma viste le tecnologie usate immagino di sì. Dopo aver fatto una breve ricerca a casa posso affermare che la risposta è certamente positiva, anche se non ho mai provato in prima persona. Vedi annuncio su http://addyosmani.com/blog/yeoman-update-announcing-1-0-beta-4

Q: Esistono delle soluzioni per il deploy e l'hosting?

A: Ci sono numerosi servizi Paas (Platform As A Service) che consentono di fare il deploy della propria applicazione NodeJS, Python, ecc tramite una comoda CLI (Command Line Interface) ed offrono una serie di servizi aggiuntivi molto comodi. Basta cercare su Google per trovare molte alternative. Per quanto riguarda Plone invece ci si può rivolgere a uno dei provider o installare un proprio VPS, vedi https://plone.org/support/providers

Sqlite array type and Python SQLAlchemy

I need to write up things just for remembering how I solved a particular issue if occurs in the future.

Sqlite (with http://sqlitebrowser.org) is great for rapid prototypes development but it lacks some useful implementations provided by Postgresql (for example the sqlalchemy.dialects.postgresql.ARRAY type).

I solved implementing a SQLAlchemy TypeDecorator with a json serialization:   

• http://docs.sqlalchemy.org/en/latest/core/types.html#sqlalchemy.types.TypeDecorator

Here it is the self-explaining code:

from sqlalchemy.schema import Column
from sqlalchemy.types import (
    Integer,
    String,
    TypeDecorator,
    )
from sqlalchemy import Sequence
from pyramid_sqlalchemy import BaseObject as Base
import json

class ArrayType(TypeDecorator):
    """ Sqlite-like does not support arrays.
        Let's use a custom type decorator.

        See http://docs.sqlalchemy.org/en/latest/core/types.html#sqlalchemy.types.TypeDecorator
    """
    impl = String

    def process_bind_param(self, value, dialect):
        return json.dumps(value)

    def process_result_value(self, value, dialect):
        return json.loads(value)

    def copy(self):
        return ArrayType(self.impl.length)

class Element(Base):
    __tablename__ = 'elements'

    id = Column(Integer(),
                Sequence('element_id_seq'),
                primary_key = True)
    # ...
    myarray = Column(ArrayType())

If you are not using Pyramid just replace the pyramid_sqlalchemy's Base wrapper import with:

from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()

collective.angularstarter (Plone + AngularJS + Yeoman kickstarter project)

Get started with Plone + AngularJS without any of the normal headaches associated with a manual setup of useful tools that let you improve your development experience and the deploy of your application.

Since I have been using AngularJS on Plone, I decided to create a reusable starter scaffold (or something or similar) based on Yeoman that let me save precious time.

That's why I created:

• https://github.com/collective/collective.angularstarter

This is a plugin that let you bootstrap single page web applications (or heavy Javascript logics) based on Plone+AngularJS+Yeoman.

Yeoman workflow benefits

collective.angularstarter is powered by the Yeoman workflow. If you want to see what are the Yeoman benefits due to an integration with a framework you might have a look at:

• Pyramid, Yeoman. On this link I deeply described the Yeoman benefits and its main components and tools (bower, grunt, etc)
• NodeJS, Express, AngularJS, Yeoman. A NodeJS + Express experiment with Yeoman and AngularJS

Next sections will talk about what you can build with Plone if you are not familiar with it, in particular heavily dynamic Javascript based verticalizations built with collective.angularstarter or similar techniques.

Plone

Plone is not only a CMS but a framework built with the Python programming language that let you build complex web applications, intranet, websites with strong focus on:

• contents
• security and sharing
• workflow
• searchability

You can extend the features provided by default by Plone thanks to a considerable number third party plugins.

You can see a couple of examples about you can build with Plone verticalizations with collective.angularstarter.

Coworking application

You can create custom content types for meeting rooms, private desks or common desks seats.

Basically the main object you are sharing is a folderish with metadata that let you configure the resource, for example:

• image, title, description and rich text widget. You can describe the resource you are sharing
• configure time slots configuration and hourly costs
• configure time slots for group of hours with costs (ex: morning, afternoon)
• daily cost or month cost, depending on the type of resource
• number of available seats available in parallel

Once the resource has been published, users can buy the suitable time slots depending on the type of resource and availability.

For private or common desks you can choose to search for multiple seats for multiple days or months. For meeting rooms you can buy just one unit time slot, multiple hours, group of hours (ex: morning) or the full day.

For example if you are searching for a meeting room you can choose partial days mode and select the available day you want to select:

The infinite scrolling shows you only the days that fits what you are searching for (2 private desks)

One selected the day, you can choose one or more available slots (for example morning):

And then: buy it!

The reservation objects are based on event types, since the reservation has a start and end datetime. So you can easily perform non-expensive catalog queries in order to search for slots available.

The project itself it is more complex because Plone it is integrated with an external invoice management software, a Paypal interface that let you buy more credits and a personal user box that let the users to see invoice PDF files and other notifications.

Advanced search forms

You can also use plone as a backend for a highly dynamic single page web application.
 
You can mix Plone data with external resources provided by third party server in order to build a complex search form.

For example the main search prompt to the user a master-slave AJAX widget

where the vocabulary of slave selects depend on the value of the previous one): 

or search for different criteria:

In this particular case results will appear after you fill all the needed information, but it is quite easy to implement a live search.

If you need more info about how to create a master-select widget component with Angular you may have a look at this article http://davidemoro.blogspot.com/2014/09/angularjs-master-slave-select-with.html.

What collective.angularstarter is

The collective.angularstarter plugin is:

1. a Plone + AngularJS kickstarter project. You can use this package when you want to develop a single page web applications powered by Angular using Plone as backend. With all the benefits of the Yeoman workflow
2. scaffolding tool that let you extend this package, add more features and then clone it creating a more sophisticated application. You can redistribute it with another name. Or you can develop a rapid prototype of your reusable application and after create a new zopeskel or yo package generator with one or more options. The clone hack might fail in some corner case but it should help you to convert an existing package to another. Anyway if something goes wrong you can easily correct the problems by hand. I get used to apply a similar script when me or other colleagues chose a very ugly package name and then you have to rename it. Maurizio, remember?! How many days we saved with this script? 

Anyway when you install collective.angularstarter and visit the @@angularstarter browser view it shows an example of AngularJS app with enabled by default:

• i18n support (http://angular-translate.github.io/)
• the AngularJS default routing support (but you can swith to more powerful alternatives. See https://github.com/angular-ui/ui-router)
• a test example
• assets management like image optimization, concat, minification/uglification of assets, unused css rules removal (see Addi Osmani's https://github.com/addyosmani/grunt-uncss) and so on
• support for production vs development mode based on two different installation mode
• Twitter Bootstrap ready

Here you can see how the @@angularstarter view looks like:

collective.angularstarter screenshot. Fill the input text and you'll see the page instantly updated

After that it's up to you coding with AngularJS and Plone!

Results

The following screenshot show you what happens if you analyze the network section of Firebug when you are in development mode:

or in production more:

Wait a moment! The resulting resultim bootstrap.css weights in at only 3,2 KB?! That's the power of minification and uncss tasks

A you can see you'll get (see the part 1 article of Pyramid starter seed project for further details about uncss and other tips explained):

• html minified (experimental, disable for real project)
• lighter images (no asset images in collective.angularstarter)
• most popular Javascript resources automatically cdn-ified
• css files concatenated, uncssed and minified
• javascript concatenated and uglified
• [update 20140926] assets automatically revved (avoid nasty caching problems)
• ... you can do more installing additional grunt tasks

How did I did it? Basically I played with Plone's resource registrations and layers.  See https://github.com/collective/collective.angularstarter/blob/master/collective/angularstarter/browser/configure.zcml

collective.angularstarter wraps a modified Yeoman AngularJS project (browser/angular): asset paths modified, bower_components folder renamed and a couple and other local changes to the Gruntfile.js file.

Hope you'll find collective.angularstarter useful. Feedback will be very appreciated!

UPDATE 20150303: I forgot to mention that collective.angularstarter plays well with Diazo (http://docs.diazo.org/en/latest/). This way you can write a pure static mocks with lots of javascripts and then using the html as the diazo theme with a bunch xml rules, with the backend decoupled with the frontend. Easy and tested on production! Probably I'll write a new write up.

Pyramid starter seed template powered by Yeoman (part 3)

In the previous articles we have seen:

• what are the benefits of the Yeoman workflow applied to Pyramid, a traditional web framework (part 1)
• how to install pyramid_starter_seed, a Pyramid+Yeoman flavoured starter template (part 2)

Once installed pyramid_starter_seed, we will see now:

• how it works under the hood
• how to manage things with grunt
• how to create and share other templates based on pyramid_starter_seed

How it works under the hood (narrative)

pyramid_starter_seed registers only one route (home -> /) and static assets views.

The home route is associated to a view callable with a webapp/%s/index.html renderer.

from pyramid.view import view_config

@view_config(route_name='home', renderer='webapp/%s/index.html')
def my_view(request):
    return {'project': 'pyramid_starter_seed'}

Views can be associated to routes imperatively or through a scan.

Wait, there is no .html renderer handled by default in Pyramid! The pyramid_starter_seed will register a .html renderer that will replace the string token with app or dist depending on the production settings and it calls the original .pt renderer.

You can choose between production vs development mode running your pyramid app with the appropriate .ini file provided by pyramid_starter_seed.

Here you can see the relevant parts of the production.ini:

[app:main]
use = egg:pyramid_starter_seed

PRODUCTION = true
minify = dist

...

and development.ini:

[app:main]
use = egg:pyramid_starter_seed

PRODUCTION = false
minify = app
...

As you can imagine the PRODUCTION configuration tells the application to switch between production or development. The minify configuration tells the .html renderer how to construct templates paths.

Let's see how looks like the index.html template. When you write css or javascript files you want to keep things separated on different modules when you are in development mode, but in production mode you might want a unique concatenated and minified/uglified resource. Here you can see how you can do that for the Bootstrap javascripts modules you might want to enable:

<!doctype html>
<html class="no-js"
      lang="${request.locale_name}"
      tal:define="minify python:request.registry.settings['minify'];
                  production python:request.registry.settings.get('PRODUCTION', 'false') == 'true'">
...

         <tal:production tal:condition="production">
            <script src="${request.static_url('pyramid_starter_seed:webapp/%s/scripts/plugins.js' % minify)}"></script>
        </tal:production>
        <tal:not_production tal:condition="not:production">
            <script src="${request.static_url('pyramid_starter_seed:webapp/%s/bower_components/bootstrap/js/alert.js' % minify)}"></script>
            <script src="${request.static_url('pyramid_starter_seed:webapp/%s/bower_components/bootstrap/js/dropdown.js' % minify)}"></script>
        </tal:not_production>
        <!-- build:js scripts/plugins.js -->
        <tal:comment replace="nothing">
            <!-- DO NOT REMOVE this block (minifier) -->
            <script src="./bower_components/bootstrap/js/alert.js"></script>
            <script src="./bower_components/bootstrap/js/dropdown.js"></script>
        </tal:comment>
        <!-- endbuild -->
...
</html>

So in development mode you will have two separate javascript files: alert.js and dropdown.js. When you are in production mode it will served a unique concatenated and uglyfied scripts/plugins.js. At first time it might seem a bit complicated or a task with a too verbose setup, but it is a simple and very powerful mechanism.

For example: do you want to add another javascript file to the plugins.js bundle? Just add two lines and you are ok!

It works without any other configuration thanks to the start and end comments blocks build:js and endbuild that groups assets groups.

Simple, isn't it? Same thing for css files.

And if you have to include images is even more simple with:

<img class="logo img-responsive" src="${request.static_url('pyramid_starter_seed:webapp/%s/images/pyramid.png' % minify)}" alt="pyramid web framework">

How to manage things with grunt

The concatenation, minification/uglyfication and image optimization (with other tasks specified in the grunt's pipeline) it is automatically performed just running the following command:

$ grunt build

This is just one of the possible implementations. Feel free to contribute and improve pyramid_starter_seed.

How to clone pyramid_starter_seed 

Fetch pyramid_starter_seed, personalize it and then clone it!

Pyramid starter seed can be fetched, personalized and released with another name. So other developer can bootstrap, build, release and distribute their own starter templates without having to write a new package template generator. For example you could create a more opinionated starter seed based on SQLAlchemy, ZODB nosql or powered by a javascript framework like AngularJS and so on.

The clone method should speed up the process of creation of new more evoluted packages based on Pyramid, also people that are not keen on writing their own reusable scaffold templates.

So if you want to release your own customized template based on pyramid_starter_seed you'll have to call a console script named pyramid_starter_seed_clone with the following syntax (obviously you'll have to call this command outside the root directory of pyramid_starter_seed):

$ YOUR_VIRTUALENV_PYTHON_PATH/bin/pyramid_starter_seed_clone new_template

and you'll get as a result a perfect renamed clone new_template:

A new starter template cloned from pyramid_starter_seed

If you provide tests you can check immediately if something went wrong during the cloning process.

In effect the clone console script it might not work in some corner cases just in case you choose a new package name that contains reserved words or the name of a dependency of your plugin, but it should be quite easy to fix by hand or improving the console script. Anyway this mechanism has been tested several years: I have built this script years ago because I was fed up with ugly package names chose by me or other colleagues of mine and allowed me to save a lot of time.


If you want to disable the console script on your new template (for example: new_template_clone) drop from setup.py the console script configuration.

So it sounds like a viral extension mechanism (I hope).

End of story?

In the next future I'd like to create a new Pyramid starter seed for single page web apps based on SQLAlchemy and powered by AngularJS. So if you similar plans... together is better: if you want to share your thoughts, improvements, feedback in general, or if you are going to create your own template based on pyramid_starter_seed please contact me (Twitter, Google+, Linkedin)!


Anyway I hope you'll save time with pyramid_starter_seed.

Links

• pyramid_starter_seed (github repo): https://github.com/davidemoro/pyramid_starter_seed
• part 1 (benefits of yeoman fully integrated with a web framework like Pyramid): http://davidemoro.blogspot.com/2014/09/pyramid-starter-seed-yeomam-part-1.html
• part 2 (how to install pyramid_starter_seed ant its prerequisites, this blog post) - http://davidemoro.blogspot.com/2014/09/pyramid-starter-seed-yeoman-part-2.html
• part 3 (how to manage things with grunt, personalize pyramid_starter_seed and create your own starter template - this blog post) - http://davidemoro.blogspot.com/2014/09/pyramid-starter-seed-yeoman-part-3.html

Pyramid starter seed template powered by Yeoman (part 2)

In the previous blog post we have seen what are the benefits of using the Yeoman workflow fully integrated with a web development framework like Pyramid. See:

• http://davidemoro.blogspot.com/2014/09/pyramid-starter-seed-yeomam-part-1.html

Now we'll add more technical details about:

• how to install pyramid_starter_seed and its prerequisites

How to install pyramid_starter_seed

Prerequisites

As you can imagine, nodejs tools are required.

I strongly suggest to:

• avoid system packages because they are too old
• install nodejs with nvm (Node Version Manager)

I won't cover the nodejs installation but it is quite simple if you follow these instructions provided by this useful link:

• https://github.com/creationix/nvm

The nvm achronym stands for NodeJS Version Manager. Once installed nvm, installing nodejs it is as simple as typing nvm install VERSION (at this time of writing 0.10.32).

Nodejs is shipped with the command line utility named npm (Nodejs Package Manager) and we will use npm for installing what we need.

We need to install our global (-g option) dev dependencies, so just type:

$ npm install -g bower
$ npm install -g grunt-cli
$ npm install -g karma

Starter seed project installation

Create an isolated Python environment as explained in the official Pyramid documentation and instal Pyramid.

Once installed you can clone pyramid_starter_seed from github:

$ git clone git@github.com:davidemoro/pyramid_starter_seed.git
$ cd pyramid_starter_seed
$ YOUR_VIRTUALENV_PYTHON_PATH/bin/python setup.py develop

Not finished yet, continue.

Yeoman initialization

Go to the folder where it lives our Yeoman project and initialize it.

These are the standard commands (but, wait a moment, see the "Notes and known issues" subsection):

$ cd pyramid_starter_seed/webapp
$ bower install
$ npm install --loglevel verbose

Known issues:

• if you are behind a proxy you'll have to configure properly npm
• if you have a slow internet connection you might experience timeout problems.

Build phase

Just type:

$ grunt

and... probably it will fail because of a couple of known issues shipped with the latest version of generator-webapp or its dependencies.

Probably these issues will be fixed in newer generator-webapp releases. However here it is how to solve these problems, so don't worry:

1. grunt-contrib-imagemin fix
   Problem with grunt:
   

   Warning: Running "imagemin:dist" (imagemin) task
   Warning: Bad argument Use --force to continue.

   Solution:

   $ npm cache clean
   $ rm -fR node_modules       # not sure it is needed, don't remember
   $ npm install grunt-contrib-imagemin

2. Mocha/PhantomJS issue

Problem with Mocha/PhantomJS launching grunt

Warning: PhantomJS timed out, possibly due to a missing Mocha run() call. Use --force to continue.

Solution:

$ cd test
$ bower install

Run bower install in the test directory of webapp (pyramid_starter_seed/webapp/test). This is a known issue, see https://github.com/yeoman/generator-webapp/issues/446.

Run you Pyramid app

Now can choose to run Pyramid in development or production mode.
Just type:

$ YOUR_VIRTUALENV_PYTHON_PATH/bin/pserve development.ini

or:

$ YOUR_VIRTUALENV_PYTHON_PATH/bin/pserve production.ini

Done!

In the next blog post with topic Pyramid + Yeoman (coming soon) I'm going to talk about:

• how to manage things with grunt and personalize pyramid_starter_seed registering other assets
• how to clone pyramid_starter_seed. Yes, you can easily customize it creating something of more sophisticated and create your own starter seed with another name. Without having to write a package generator

 So stay tuned..

Links

• pyramid_starter_seed (github repo): https://github.com/davidemoro/pyramid_starter_seed
• part 1 (benefits of yeoman fully integrated with a web framework like Pyramid): http://davidemoro.blogspot.com/2014/09/pyramid-starter-seed-yeomam-part-1.html
• part 2 (how to install pyramid_starter_seed ant its prerequisites, this blog post) - http://davidemoro.blogspot.com/2014/09/pyramid-starter-seed-yeoman-part-2.html
• part 3 (how to manage things with grunt, personalize pyramid_starter_seed and create your own starter template) - http://davidemoro.blogspot.com/2014/09/pyramid-starter-seed-yeoman-part-3.html

Pyramid starter seed template powered by Yeoman (part 1)

Book of the month I'm reading this summer: Pylons/Pyramid (http://docs.pylonsproject.org/en/latest).


Pyramid (http://www.pylonsproject.org) is a minimal Python-based web development framework that let you "start small and finish big".

It stole a lot of (good) ideas and concepts from other mature Python web frameworks and it is build with the pluggable and extensible concepts in mind. Read: no need to fork applications.

Furthermore Pyramid is database and template engine agnostic: you are free.

From the very beginning Pyramid allows you to become productive quickly. So why not start with something of useful?

Pyramid + Yeoman

The goal of this experiment is integrate yeoman with Pyramid (or other frameworks like NodeJs/Express with AngularJS or Plone as already did), preserving the yeoman's workflow.

UPDATE 20140926: here you can see a Plone + AngularJS + Yeoman article (collective.angularstarter)

In this article I'll talk about what are the benefits you get integrating your Pyramid app with Yeoman, in future posts I'll discuss how they work under the hood with additional technical details omitted here (each used component deserves an entire blog post).

Yeoman

You might wonder why? Because of the importance of tooling. Since it is very important build an effective developer tooling ecosystem, I want to integrate the simple starter demo app with commonly used tools to help you stay productive. So this simple application prototype it is just an experiment that should help you to integrate with modern web development tools provided by the yeoman workflow stack (http://yeoman.io).

Choosing the right tools is very important for the best develop experience and I cannot work anymore without Yeoman, especially when coding with Javascript.

Grunt

Yeoman it is internally based on three important components (nodejs powered):

• yo, scaffolding tool like pcreate, paster or zopeskel. It is widely adopted by a large and trasversal community
• grunt, system used for build, preview and test your software. Gulp is another popular option
• bower, used for dependency management, so that you no longer have to manually download and manage your scripts

Bower

So with the yeoman's tools you can just code, avoid annoying repetitive tasks and don't worry about:

• javascript testing setup
• javascript code linting
• javascript/css minification and merging
• image minification
• html minification
• switch to CDN versions of you vendor plugins in production mode
• auto-reload browser
• much much more

So let's see together what happened to our pyramid starter demo template created with pcreate -t starter integrated with a yeoman's generator-webapp project.

The result will be a Pyramid starter seed project integrated with modern non Python-based web development tools.

Goals

Management of third party assets

You no longer have to manually download and manage your scripts with the Bower package manager.

From http://bower.io:

"""Bower works by fetching and installing packages from all over, taking care of hunting, finding, downloading, and saving the stuff you’re looking for."""

So just type something like: bower install angular-translate --save and you'll get the rigth resource with pinning support.

Tasks automation

Automation, automation, automation.

From http://gruntjs.com:

"""Why use a task runner? In one word: automation. The less work you have to do when performing repetitive tasks like minification, compilation, unit testing, linting, etc, the easier your job becomes. After you've configured it, a task runner can do most of that mundane work for you—and your team—with basically zero effort."""

Examples:

• grunt serve
• grunt test
• grunt build
• grunt YOUR TASK 
• etc

Jslint

No more deploy Javascript code with bad indentation, syntax errors or bad code practices.

All syntax errors or bad practise will be found.

Image minification

The build process will detect and minify automatically all your asset images.

Uncss task

Modern (and heavy) UI frameworks like Twitter Bootstrap provide an excellent solution for prototyping your initial project, but most of the times you are using a very minimal subset of their functionalities.

This inspiring Addy Osmani's blog post helps you to remove unused css in your pages with a grunt task named grunt-uncss (https://github.com/addyosmani/grunt-uncss):

• http://addyosmani.com/blog/removing-unused-css

The original not-minified bootstrap.css weights in at 120 kB before removing unused rule.

Css concat and minification

You can split your css code into different files and then the build process will concat and minify them creating a unique app.css file. This way you write modular and better readable css files, reducing the number of browser requests.

The theme.css file is quite small but in real projects you can save more. In this case:

The configured build pipeline is concat, uncss and cssmin. 122.85 kB (original bootstrap.css) -> 4.64 kB (uncss) -> 3.45 kB (minification)

Automatic CDN-ification

It is handy using unminified versions of third party javascript libraries during development and switch to CDN versions in production mode with well known benefits for your website.

Don't worry: the cdnify task will take care about this boring issue. Automatically.

You save a boring manual and error-prone configuration.

Composable bootstrap.js version

The Pyramid starter project is based on Twitter Bootstrap.

Twitter Bootstrap

Depending on your project you can load the whole Twitter Bootstrap Javascript code at once or including individual plugins.

As you can see the Javascript component of Twitter Bootstrap is very modular: http://getbootstrap.com/javascript. So if you don't use a particular feature, just don't include it.

This way in development mode you will have all individual plugins splitted in different files, in production it will served a unique concatenated and minified Javascript file built automatically.

So if you just need alert.js and dropdown.js you can get a 2.79 kB plugins.js:

The concatenation of alert.js and dropdown.js produces a 7.06 kB, that weight in at 2.79 kB after minification instead of the 8.9 kB (gzipped) bootstrap-min.js corresponding to not gzipped 27.2 kB.

Html (template) minification

Since the ZPT/Chameleon templating language is an extension of HTML with xml syntax,

Brower are able to display unrendered ZPT/Chameleon templates

theorically it can play well with html minificators.

I know, template minification can lead to potential unexpected problems due to minification issues on template files... but this is my personal playground, so let me play please!

So... why not switch to a pre-compiled minified template of your ZPT/Chameleon files when you are in "production mode"?

Obviously during development you will use the original template files.

The interesting side of this approach is that there is no overhead at response time, since the minification task runs just one time before deploying your application. It might be an option if you want just your html minified and you cannot feasibly add to your site or project additional optimization tools at web server level.

Anyway I have tried this mad experiment and... if you don't use too aggressive minification params, it seems to work fine with good results. Try it at your own risk or just disable it. Here you can the effects on the generated index.html used in production:

Template minified (7.62 kB -> 4.16 kB)

Result: a lighter Pyramid

Same results but a lighter Pyramid app:

Let's see how it behave the standard Pyramid starter project:

Standard Pyramid starter project (production.ini)

And the Pyramid starter seed:

Pyramid starter seed (production.ini)

As you can see the seed version is ~38 Kb smaller and more performant.

Useful links

• uncss - http://addyosmani.com/blog/removing-unused-css/
• alternative installation of nodejs - http://greenfinity.hu/en/blog/installing-javascript-from-zc-buildout
• performance optimization techniques - http://www.smashingmagazine.com/2014/09/08/improving-smashing-magazine-performance-case-study
• @CSSconfeu CSS performance tooling slides (updated 20140919) - https://speakerdeck.com/addyosmani/css-performance-tooling 

That's all?

No, you can do more, for example:

• reduce the  number or requests (for example you can merge vendor.css and app.css)
• create and keep updated css sprites with grunt (https://github.com/Ensighten/grunt-spritesmith)
• manage and upload all your assets to professional services like Amazon AWS (for example you can serve up all your images, styles and scripts from a S3 bucket + CloudFront). This way Pyramid will be able to handle more requests. Pyramid let you put static media on a separate webserver during production with static_url() in conjunction with add_static_view(), without having to change your templates code
• generate static gzipped assets with Grunt and let your webserver serve them
  
• install and configure dedicated performance modules at webserver level (Apache's mod_pagespeed)

Let me know what you think about that, please. Hope soon I will manage to write the second part of this blog post explaining how I did it. In the mean time you can:

• have a look at the pyramid_starter_seed repository on Github https://github.com/davidemoro/pyramid_starter_seed
• follow me on Twitter https://twitter.com/davidemoro
• keep in touch with me on Google+ (https://plus.google.com/+davidemoro) or Linkedin (https://www.linkedin.com/in/davidemoro82)

Links

• pyramid_starter_seed (github repo): https://github.com/davidemoro/pyramid_starter_seed
• part 1 (benefits of yeoman fully integrated with a web framework like Pyramid, this blob post): http://davidemoro.blogspot.com/2014/09/pyramid-starter-seed-yeomam-part-1.html
• part 2 (how to install pyramid_starter_seed ant its prerequisites) - http://davidemoro.blogspot.com/2014/09/pyramid-starter-seed-yeoman-part-2.html
• part 3 (how pyramid_starter_seed works under the hood - how to achieve these results and manage things with grunt, personalize pyramid_starter_seed and create your own starter template) - http://davidemoro.blogspot.com/2014/09/pyramid-starter-seed-yeoman-part-3.html

AngularJS master-slave select with $location.search() params

In this post we will see how to implement a simple master-slave select controls in AngularJS.

This kind of controls are useful when your users selection depends on the previous select.

For example:

• continent (Europe, America, Asia, Oceania)
• country (USA, ...)
• state (California, ...)
• etc

The implementation is a bit more complex because I want to support urls that reflect each user input (example: /search -> /search?continent=America -> /search?continent=America&country=USA -> etc). This way the user will be able to share the url of an intermediate state of the application without having to retype all form inputs. Sometimes if you paste the url of a single page application heavily based on AJAX you loose the context.

The key concepts based on this kind of implementation are:

• $location.search()
• use reloadOnSearch: false in conjunction on $routeProvider.when. It just updates query params without a reload

Code

In bold the relevant blocks.

Controller

'use strict';

angular
  .module('angularApp', []);
 
  angular.module('angularApp')
  .controller('MainCtrl', ['$scope', '$location', 'queryFactory', function ($scope, $location, queryFactory) {

    var params = $location.search();   // the current params dict

    var initParams = function () {
      /* init controllers conf params */
      angular.forEach(params, function (value, key) {
        if ($scope.conf.search[key]) {
          $scope.conf.search[key].selected = value;
        }
      });
    };
    /* The model configuration */
    $scope.conf = {
      initsearch: 'select1',   // the master of all selects
      search: {
        select1: {
          values: [], selected: undefined, slave: 'select2', updater: queryFactory.getSelect1
        },
        select2: {
          values: [], selected: undefined, slave: undefined, updater: queryFactory.getSelect2
        },
      }
    };

    /* Master select initialization (lookup of options) */  $scope.conf.search[$scope.conf.initsearch].updater($scope.conf.search)
      .success(function(data) {
        $scope.conf.search[$scope.conf.initsearch].values = data;
      });

    /* Init $scope.conf.search depending on search params.
     * */
    initParams();

    // not yet available $watchGroup on angularjs 1.2
    angular.forEach($scope.conf.search, function (value, key) { // jshint ignore:line
      $scope.$watch('conf.search.' + key + '.selected', function(newValue, oldValue) { // jshint ignore:line
        var slave = $scope.conf.search[key].slave;

        params[key] = newValue;
        $location.search(params);

        if (newValue) {
          if ($scope.conf.search[slave]) {
            $scope.conf.search[slave].updater($scope.conf.search)
              .success(function(data) {
                $scope.conf.search[slave].values = data;
              });
          }
          else {
            // slave is undefined
            if ($scope.conf.search[key].selected) {
              // end of chain, do something
              // TODO
            }
          }
        }
        else {
          // removed selected and values
          if ($scope.conf.search[slave]) {
            $scope.conf.search[slave].values = [];
            if ($scope.conf.search[slave].selected) {
              $scope.conf.search[slave].selected = undefined;
            }
          }
        }
      });

    });
  }]);

Template

<!DOCTYPE html>
<html ng-app="angularApp">

  <head>
    <script data-require="angular.js@*" data-semver="1.2.22" src="https://code.angularjs.org/1.2.22/angular.js"></script>
    <link href="style.css" rel="stylesheet" />
    <script src="script.js"></script>
  </head>

  <body>
    <div ng-controller="MainCtrl">
      <h2>Master select widget example with AngularJS</h2>
      Launch in a separate window in order to see the .search() usage: click on the blue button on your top-right of the demo.<br/>
     
      <select ng-model="conf.search.select1.selected" ng-options="item.id as item.title for item in conf.search.select1.values">
        <option value="">--</option>
      </select>

      <select ng-model="conf.search.select2.selected" ng-options="item.id as item.title for item in conf.search.select2.values">
        <option value="">--</option>
      </select>

    </div>
  </body>

</html>

Service (performs $http queries)

angular.module('angularApp')
  .factory('queryFactory', ['$q', function ($q) {
    // Service logic (mock)
    // You should put here your $http calls, for a working example see http://davidemoro.blogspot.it/2014/09/angularjs-how-to-test-http-calls.html

    // Public API here
    return {
      getSelect1: function (conf) {  // jshint ignore:line
        var promise = $q.when([{id: '1', title: '1'}, {id: '2', title: '2'}]);
        promise.success = function(fn) {
          promise.then(function(response) {
            fn(response);
          });
          return promise;
        };

        return promise;
      },
      getSelect2: function (conf) { // jshint ignore:line
        var promise;
        if (conf['select1'].selected === '1') {
            promise = $q.when([{id: '1.1', title: '1.1'}, {id: '1.2', title: '1.2'}]);
        } else {
            promise = $q.when([{id: '2.1', title: '2.1'}, {id: '2.2', title: '2.2'}]);
        }
        promise.success = function(fn) {
          promise.then(function(response) {
            fn(response);
          });
          return promise;
        };

        return promise;
      }
    };
  }]);

How to implement $http tests

You can see an example of tests for a service $http-based decoupled from the controller logic:

• http://davidemoro.blogspot.it/2014/09/angularjs-how-to-test-http-calls.html

Results

Selecting 1 on the first select you'll get 1.X values on the second one and so on

Plunkr demo

I have made a Plunkr demo available at this url:

• http://plnkr.co/edit/yLelIW?p=info

If you open the Plunkr link in fullscreen mode you'll see the query params will change for each input. If you copy and paste the intermediate url you'll get still a compiled form.

Generic master-slave directive?

I'd like to implement a generic and more reusable master-slave directive. What are your suggestions about the best design strategy and how to build things following the Angular way?

AngularJS - how to test $http calls (with tests example)

AngularJS templates no longer automatically unwrap promises (AngularJS >= 1.2). See https://code.angularjs.org/1.2.24/docs/guide/migration#templates-no-longer-automatically-unwrap-promises

This feature is now deprecated but if you absolutely need it, it can be reenabled for now via the $parseProvider.unwrapPromises(true) API.

In this post we will see how to:

• separate data lookup from controllers logic. It is way way better implement $http calls into dedicated angular services for separation of concerns, good design and software testability
• how to test $http calls with mocks

This is just a reminder for me but I hope other developers will save time if they don't figure out why their code does not how expected.

Meaningful code in bold, feedback please!

Controller

Controller's code

scripts/controllers/car.js:

'use strict';

angular.module('angularApp')
  .controller('CarCtrl', ['$scope', '$routeParams', 'carFactory', function ($scope, $routeParams, carFactory) {

    // route params (example: )
    $scope.params = $routeParams;

    carFactory.get($scope.params.carId)
      .success(function(data) {
        $scope.data = data;
      });
  }]);

Controller's test code

test/spec/controllers/car.js:

'use strict';

describe('Controller: CarCtrl', function () {

  // load the controller's module
  beforeEach(module('angularApp'));

  var CarCtrl,
    scope,
    success;

  // Initialize the controller and a mock scope
  beforeEach(inject(function ($controller, $rootScope, $q) {

    var promise = $q.when({id: '1', title: 'Audi'});
    promise.success = function(fn) {
      promise.then(function(response) {
        fn(response);
      });
      return promise;
    };

    scope = $rootScope.$new();
    CarCtrl = $controller('CarCtrl', {
      $scope: scope,
      $routeParams: {carId: 'audi'},
      carFactory: {get: function () {return promise;}}
    });
  }));


  it('should have routeParams into params', function () {
    expect(scope.params.carId).toBe('audi');
  });

  it('should have carFactory data', function () {
    scope.$digest();
    expect(!!scope.data).toBe(true);
  });
});

Service

Service code

The base url of my remote endpoint can be injected with:

.value('apiPrefix', 'http://localhost:3001/')

if you are using a service and it is a value shared with other components.
Otherwise can use a configurable service (provider).

Using value or a configurable service helps you to mock things injecting different endpoint urls during development.

scripts/services/carfactory.js):

'use strict';

angular.module('angularApp')
  .factory('carFactory', ['$http', 'apiPrefix', function ($http, apiPrefix) {
    // Service logic
    // ...
    var base = apiPrefix ? apiPrefix : '';

    // Public API here
    return {
      get: function (carId) {
        var promise;

        promise = $http.jsonp(base + 'cars/' + carId + '?callback=JSON_CALLBACK');
        return promise;
      }
    };
  }]);

Service test code

test/spec/services/carfactory.js:

'use strict';

describe('Service: carFactory', function () {

  // load the service's module
  beforeEach(module('angularApp'));

  // instantiate service
  var carFactory, $httpBackend, apiPrefix;
  beforeEach(inject(function (_carFactory_, _$httpBackend_, _apiPrefix_) {
    carFactory = _carFactory_;
    $httpBackend = _$httpBackend_;
    apiPrefix = _apiPrefix_;

    $httpBackend.whenJSONP(apiPrefix + 'cars/audi?callback=JSON_CALLBACK').respond({
      id: 'audi',
      title: 'Audi'
    });
  }));

  afterEach(function() {
    $httpBackend.verifyNoOutstandingExpectation();
    $httpBackend.verifyNoOutstandingRequest();
  });

  it('should do something', function () {
    expect(!!carFactory).toBe(true);
  });

  it('test get data', function () {
    var data, promise;
    promise = carFactory.get('audi');
    promise.success(function(res) {
      data = res;
    });
    $httpBackend.flush();

    expect(data.title).toBe('Audi');
  });

});