Pyramid exceptions logging

If you want to log exceptions with Pyramid (http://www.pylonsproject.org/) you should start reading carefully the following resources:

• http://docs.pylonsproject.org/docs/pyramid/en/latest/narr/logging.html
• http://pyramid-exclog.readthedocs.org/en/latest/

and once enabled pyramid_exclog check your tween chain order as explained here http://pyramid-exclog.readthedocs.org/en/latest/#explicit-tween-configuration.

If you get in trouble some some reason, maybe you'll find this article helpful in some way. Depending on how you serve your application, it might happen that your .ini configuration logging settings are not considered at all.

Covered arguments in this post:

• how to log exceptions based on PasteDeploy configuration .ini files
• running your app with uwsgi
• or with CherryPy (see the how to deploy pyramid applications on windows and the Windows service section on http://pyramid-cookbook.readthedocs.org/en/latest/deployment/windows.html)

How to configure your PasteDeploy .ini file

In the following example I'm using the handlers.RotatingFileHandler (a python logrotate implementation), but feel free to use FileHandler (without handlers.).
You can configure your production.ini like the following one:

...
###
# logging configuration
# http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/logging.html
###

[loggers]
keys = root, mip_project, exc_logge

[handlers]
keys = console, exc_handle

[formatters]
keys = generic, exc_formatter

[logger_root]
level = WARN
handlers = console

[logger_mip_project]
level = WARN
handlers =
qualname = mip_project

[logger_exc_logger]
level = ERROR
handlers = exc_handler
qualname = exc_logger

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[handler_exc_handler]
class = handlers.RotatingFileHandler
args = ('exception.log', 'a', 10000000, 10)
level = ERROR
formatter = exc_formatter

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s

[formatter_exc_formatter]
format = %(asctime)s %(message)s

[uwsgi]
plugin = python
...
master = true
processes = 1
threads = 4
virtualenv = %d/your_python
chdir = %d
home = %d/your_python

In the above example it is important omitting %(here)s/ in the RotatingFileHandler args, otherwise it won't be resolved running your app with uwsgi.

If uwsgi complains about no paste.script.util.logging_config is available

If your logger configuration is not considered by uwsgi and you see a warning more or less like a missing "paste.script.util.logging_config" module, just install PasteScript in your virtualenv:

(your_python) $ pip install PasteScript

And make sure your [uwsgi] section is pointing to your virtualenv (see previous section).

Anyway, once installed PasteScript (I've tested the 1.7.5 version), all went fine:

$ uwsgi --ini-paste-logged production.ini

Unfortunately I wasn't able to get exception logs enabled if I call the above command in a supervisord configuration file. Any suggestions?

See also:

• http://lists.unbit.it/pipermail/uwsgi/2012-February/003619.html
• http://lists.unbit.it/pipermail/uwsgi/2012-February/003620.html 

Windows service and  CherryPy

If you want to serve a Pyramid application with CherryPy (or creating a CherryPy based Windows service) with exception logging enabled you'll have to setup logging by hand because CherryPy does not consider your logging configuration.

See also:

• http://pyramid-cookbook.readthedocs.org/en/latest/deployment/windows.html
• http://davidemoro.blogspot.com/2015/03/pyramid-mysql-windows-good-ugly-bad.html

You can follow the above guides and add the bold lines in your pyramidsvc.py in order to have exceptions logged:

from cherrypy import wsgiserver
from pyramid.paster import get_app
from pyramid.paster import setup_logging
import os

... 

    def SvcDoRun(self):

 
        path = os.path.dirname(os.path.abspath(__file__))

        os.chdir(path)

        app = get_app(CONFIG_FILE)
        setup_logging(CONFIG_FILE)

        .... 

Links

A very interesting overview on Python logging and syslog:

• https://hynek.me/articles/taking-some-pain-out-of-python-logging/

Pyramid, MySQL and Windows: the good, the ugly and the bad

Pyramid, MySQL and Windows: the good (Pyramid), the ugly and the bad. This title does not fit perfectly the main characters of this blog post because some of theme are both ugly and bad, but it doesn't matter.

Just in case you are going to set up a Pyramid project based on MySQL and Windows (sometimes you have to)... there are a couple of things useful to know. But let's start with a chronologically order.

Day 0 - morning

You feel like the brave Steeve McQueen:

Evening - day 0

At the end of the day you'll feel also like Steeve McQueen, but a little more proved:

What happened?

Random problems with normal transactions considered too large, thread disconnected, error log entries like:

• InterfaceError
• OperationalError
• MySQL Server has gone away
• database connection failure
• TimeoutError: QueuePool limit of size ... overflow ... reached, connection timed out, timeout ...
• pyramid process hangs
• etc

The solution

1 - adjust your my.ini options like that:

[mysqld]
max_allowed_packet = 64MB    # adjust this parameter according to your situation
wait_timeout = 28800
interactive_timeout = 2880 

2 - be sure your production.ini file looks like the following one (with Python 2):

sqlalchemy.url = mysql+mysqldb://USER:PASSWORD@127.0.0.1:3306/YOURDB?charset=utf8&use_unicode=0
# For Mysql "MySQL Connection Timeout and SQLAlchemy Connection Pool Recycling" issues see:
# http://docs.sqlalchemy.org/en/latest/core/pooling.html#setting-pool-recycle
# http://douglatornell.ca/blog/2012/01/08/staying-alive/
sqlalchemy.pool_recycle = 3600

3 - you can schedule a restart of your application once a day.

4 - [OPTIONAL, not only Windows related] adjust your SqlAlchemy configuration parameters according to how many threads your server runs. For example (production.ini):

sqlalchemy.pool_size = 20
sqlalchemy.max_overflow = 10

5 - if you are using CherryPy as a Windows service, be sure your  'engine.autoreload.on' option is set to False.


6 - [UPDATE20150401] check your sql-mode if you want to prevent not handled ProxyError exceptions due to input longer than the max size provided on the field model. See https://github.com/Kotti/Kotti/issues/404

 Results

No more exceptions or odd behaviours!

Links

• http://docs.sqlalchemy.org/en/rel_0_9/dialects/mysql.html#unicode-encoding-decoding
• http://docs.sqlalchemy.org/en/rel_0_9/dialects/mysql.html#module-sqlalchemy.dialects.mysql.mysqldb
• http://docs.sqlalchemy.org/en/latest/core/pooling.html#setting-pool-recycle
• http://douglatornell.ca/blog/2012/01/08/staying-alive/

Kotti CMS events - insert subobjects automatically

Yet another small recipe for Kotti CMS: how to initialize automatically a new object once inserted with events, for example adding a subobject.

Use case? When someone creates a UserProfile object /users/name-surname, an event should create automatically a profile image in /users/name-surname/photo (a Kotti image instance).

It is quite simple. Let's see our your_package/events.py module, where IMAGE_ID is equals to 'photo':

import os
from kotti.events import (
    ObjectInsert,
    subscribe,
    notify,
    )
from kotti.resources import Image
from your_package.resources import UserProfile
from your_package.config import IMAGE_ID


@subscribe(ObjectInsert, UserProfile)
def user_profile_added(event):
    obj = event.object

    if IMAGE_ID not in obj.keys():
        image_path = os.path.join(
            os.path.dirname(__file__),
            'data', 'fallback.png'
            )
        with open(image_path, 'rb') as image_file:
            obj[IMAGE_ID] = image_obj = Image(
                title='Image',
                in_navigation=False,
                data=image_file.read(),
                filename=u'fallback.png',
                mimetype=u'image/png',
                )
            notify(ObjectInsert(image_obj, event.request)) 

Notes:

1. the subscribe decorator will register our handler when a UserProfile resource will be inserted
2. we should check if IMAGE_ID is already instanciated (prevent errors on paste)
3. if you want your code will work both for Unix-like or under Windows, use os.path.join instead of a plain data/fallback.png path (path separator issues)
4. the final b in the open is important if you want to write code that works under Windows, see https://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files. On Unix, it doesn’t hurt to append a 'b' to the mode, so you can use it platform-independently for all binary files
5. notify the image insertion

And your_package/__init__.py (in this example I've used the scan method but you could also register your event handlers imperatively):

...
def includeme(config):
    """ Don't add this to your ``pyramid_includes``, but add the
    ``kotti_configure`` above to your ``kotti.configurators`` instead.

    :param config: Pyramid configurator object.
    :type config: :class:`pyramid.config.Configurator`
    """
    ...
    config.scan(__name__)

And... tests of course (your_package/tests/test_events.py):

from pytest import fixture

from kotti.testing import DummyRequest


@fixture
def user_profile(db_session, config, root):
    """ returns dummy UserProfile.
    """
    config.include('your_package')

    from your_package.resources import UserProfile
    root['userprofile'] = userprofile = UserProfile(title='UserProfile')
    from kotti.events import notify
    from kotti.events import ObjectInsert
    notify(ObjectInsert(course, DummyRequest()))    return userprofile


class TestUserProfileWithEvents:

    def test_assert_userprofile_image(self, user_profile):
        from your_package.config import IMAGE_ID
        assert IMAGE_ID in user_profile.keys()
    ... 

You can test also if everything goes right after a copy/paste action (see the Kotti's tests).

Done!

All posts about Kotti

• Pyramid, MySQL and Windows: the good, the ugly and the bad
• Kotti CMS events - insert subobject automatically
• Kotti CMS - how to turn your Kotti CMS into an intranet
• Kotti CMS - how to store arbitrary data with annotations
• How to install Kotti CMS on Windows
• Kotti CMS - avoid types addable in content root
• Kotti CMS - how to create a new content type with an image
• Kotti CMS - workflow reference

All Kotti posts published by @davidemoro:

• http://davidemoro.blogspot.it/search/label/kotti

How to install Kotti CMS on Windows

Yes, as expected, you can install Kotti CMS also on Windows if you have this constraint!

What is Kotti

From the official doc:

"""A high-level, Pythonic web application framework based on Pyramid and SQLAlchemy. It includes an extensible Content Management System called the Kotti CMS.

Kotti is most useful when you are developing applications that:

• have complex security requirements
• use workflows, and/or
• work with hierarchical data

"""

It is developer friendly and with a good user interface. You can easily extend it, develop new features or install one of the available third party modules (search for Kotti on https://pypi.python.org/pypi if you want to browse existing modules ready to be used). Heavily inspired by Plone (http://plone.org).

If you want to evaluate Kotti you can install it locally (no database installation is required, you can use SQLlite during evaluation or development).

Otherwise if you are particular lazy there is a working demo online with admin / qwerty administrator credentials: 

• http://kottidemo.danielnouri.org

Prerequisites

• python (tested with python 2.7.9 but it should work also on newer versions)
• Microsoft Visual C++ 9.0 available on the following url http://aka.ms/vcpython27 (needed for an issue with bcrypt)
• virtualenv (suggested)

Installation steps

Once you have installed python from http://www.python.org you can start installing Kotti. I assume in this article that your Python installation path is C:\Python27.

Now create a new folder (it doesn't matter the name, in this article my folder name is just kotti):

> mkdir kotti
> cd kotti

Install virtualenv and create a new isolated python environment in your kotti dir:

> C:\Python27\Scripts\pip.exe install virtualenv> C:\Python27\Scripts\virtualenv.exe --no-site-packages .

Install Kotti and its requirements:

> Scripts\pip.exe install -r https://raw.github.com/Kotti/Kotti/stable/requirements.txt
> Scripts\pip.exe install Kotti

Put inside your kotti dir the app.ini file downloaded from:

• https://raw.github.com/Kotti/Kotti/stable/app.ini

Runs Kotti:

Scripts\pserve.exe app.ini
Starting server in PID 2452
serving on http://127.0.0.1:5000

Done!

Update 20150219: if you want to install Kotti as a standard Windows service see this tutorial: http://pyramid-cookbook.readthedocs.org/en/latest/deployment/windows.html.

Troubleshooting (tested on Windows Vista)

If Microsoft Visual C++ Compiler for Python 2.7 is not installed on your environment you'll get an error during the requirements installation phase (only on Windows):

> Scripts\pip.exe install -r https://raw.githubusercontent.co
m/Kotti/Kotti/stable/requirements.txt
....
  Running setup.py install for py-bcrypt
    building 'bcrypt._bcrypt' extension
    error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat).
Get it from http://aka.ms/vcpython27
    Complete output from command C:\Users\dmoro\kotti\Scripts\python.exe -c "imp
ort setuptools, tokenize;__file__='c:\\users\\dmoro\\appdata\\local\\temp\\pip-b
uild-mact2r\\py-bcrypt\\setup.py';exec(compile(getattr(tokenize, 'open', open)(_
_file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\u
sers\dmoro\appdata\local\temp\pip-wcmy6c-record\install-record.txt --single-vers
ion-externally-managed --compile --install-headers C:\Users\dmoro\kotti\include\
site\python2.7:
    running install

    running build

    running build_py

    creating build

    creating build\lib.win-amd64-2.7

    creating build\lib.win-amd64-2.7\bcrypt

    copying bcrypt\__init__.py -> build\lib.win-amd64-2.7\bcrypt

    running build_ext

    building 'bcrypt._bcrypt' extension

    error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat).
Get it from http://aka.ms/vcpython27

You just need to install this requirement and all will work fine.

Troubleshooting 2 (tested on Windows 2008 R2 Server - updated 20150219)

You might experience other compilation errors on Windows due to different compiler versions, role management tool configuration, missing DLLs, environment variables (vcvars32.bat), missing header files, etc. The same C code that compiles fine on a Windows machine, on a different version of Windows could produce a compilation error (compiling under Windows is a pain).

Anyway the following links helped me a lot a install py-bcrypt under Windows 2008 R2 Server with Visual Studio 2008 Express (free version downloadable from http://go.microsoft.com/?linkid=7729279):

• https://code.google.com/p/py-bcrypt/issues/detail?id=18
• http://stackoverflow.com/questions/10773732/compilation-error-in-visual-studio-linked-with-python26
• http://stackoverflow.com/questions/2817869/error-unable-to-find-vcvarsall-bat
• http://stackoverflow.com/questions/3047542/building-lxml-for-python-2-7-on-windows/5122521#5122521
• http://stackoverflow.com/questions/17108102/download-and-install-visual-studio-2008
• http://blogs.msdn.com/b/vijaysk/archive/2009/08/16/you-must-use-the-role-management-tool-to-install-or-configure-microsoft-net-framework-3-5.aspx
• https://msahputra.wordpress.com/2009/04/13/clexe-for-vc9-visual-studio-2008/
• https://msahputra.wordpress.com/2009/04/13/clexe-for-vc9-visual-studio-2008/
• https://code.google.com/p/py-bcrypt/issues/detail?id=15
• http://msinttypes.googlecode.com/svn/trunk/stdint.h

Links

• http://kotti.pylonsproject.org/
• https://kotti.readthedocs.org/en/latest/
• https://kotti.readthedocs.org/en/latest/first_steps/installation.html
• http://kottidemo.danielnouri.org
• http://python.org
• http://www.pylonsproject.org/projects/pyramid/about
• https://twitter.com/KottiCMS

Screenshots

Kotti's front page (from the public demo online):

Kotti's folder contents (from the public demo online), requires authentication:

All posts about Kotti

• Pyramid, MySQL and Windows: the good, the ugly and the bad
• Kotti CMS - how to turn your Kotti CMS into an intranet
• Kotti CMS - how to store arbitrary data with annotations
• How to install Kotti CMS on Windows
• Kotti CMS - avoid types addable in content root
• Kotti CMS - how to create a new content type with an image
• Kotti CMS - workflow reference

All Kotti posts published by @davidemoro:

• http://davidemoro.blogspot.it/search/label/kotti