Pyramid exceptions logging

If you want to log exceptions with Pyramid (http://www.pylonsproject.org/) you should start reading carefully the following resources:

• http://docs.pylonsproject.org/docs/pyramid/en/latest/narr/logging.html
• http://pyramid-exclog.readthedocs.org/en/latest/

and once enabled pyramid_exclog check your tween chain order as explained here http://pyramid-exclog.readthedocs.org/en/latest/#explicit-tween-configuration.

If you get in trouble some some reason, maybe you'll find this article helpful in some way. Depending on how you serve your application, it might happen that your .ini configuration logging settings are not considered at all.

Covered arguments in this post:

• how to log exceptions based on PasteDeploy configuration .ini files
• running your app with uwsgi
• or with CherryPy (see the how to deploy pyramid applications on windows and the Windows service section on http://pyramid-cookbook.readthedocs.org/en/latest/deployment/windows.html)

How to configure your PasteDeploy .ini file

In the following example I'm using the handlers.RotatingFileHandler (a python logrotate implementation), but feel free to use FileHandler (without handlers.).
You can configure your production.ini like the following one:

...
###
# logging configuration
# http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/logging.html
###

[loggers]
keys = root, mip_project, exc_logge

[handlers]
keys = console, exc_handle

[formatters]
keys = generic, exc_formatter

[logger_root]
level = WARN
handlers = console

[logger_mip_project]
level = WARN
handlers =
qualname = mip_project

[logger_exc_logger]
level = ERROR
handlers = exc_handler
qualname = exc_logger

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[handler_exc_handler]
class = handlers.RotatingFileHandler
args = ('exception.log', 'a', 10000000, 10)
level = ERROR
formatter = exc_formatter

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s

[formatter_exc_formatter]
format = %(asctime)s %(message)s

[uwsgi]
plugin = python
...
master = true
processes = 1
threads = 4
virtualenv = %d/your_python
chdir = %d
home = %d/your_python

In the above example it is important omitting %(here)s/ in the RotatingFileHandler args, otherwise it won't be resolved running your app with uwsgi.

If uwsgi complains about no paste.script.util.logging_config is available

If your logger configuration is not considered by uwsgi and you see a warning more or less like a missing "paste.script.util.logging_config" module, just install PasteScript in your virtualenv:

(your_python) $ pip install PasteScript

And make sure your [uwsgi] section is pointing to your virtualenv (see previous section).

Anyway, once installed PasteScript (I've tested the 1.7.5 version), all went fine:

$ uwsgi --ini-paste-logged production.ini

Unfortunately I wasn't able to get exception logs enabled if I call the above command in a supervisord configuration file. Any suggestions?

See also:

• http://lists.unbit.it/pipermail/uwsgi/2012-February/003619.html
• http://lists.unbit.it/pipermail/uwsgi/2012-February/003620.html 

Windows service and  CherryPy

If you want to serve a Pyramid application with CherryPy (or creating a CherryPy based Windows service) with exception logging enabled you'll have to setup logging by hand because CherryPy does not consider your logging configuration.

See also:

• http://pyramid-cookbook.readthedocs.org/en/latest/deployment/windows.html
• http://davidemoro.blogspot.com/2015/03/pyramid-mysql-windows-good-ugly-bad.html

You can follow the above guides and add the bold lines in your pyramidsvc.py in order to have exceptions logged:

from cherrypy import wsgiserver
from pyramid.paster import get_app
from pyramid.paster import setup_logging
import os

... 

    def SvcDoRun(self):

 
        path = os.path.dirname(os.path.abspath(__file__))

        os.chdir(path)

        app = get_app(CONFIG_FILE)
        setup_logging(CONFIG_FILE)

        .... 

Links

A very interesting overview on Python logging and syslog:

• https://hynek.me/articles/taking-some-pain-out-of-python-logging/

Pyramid, MySQL and Windows: the good, the ugly and the bad

Pyramid, MySQL and Windows: the good (Pyramid), the ugly and the bad. This title does not fit perfectly the main characters of this blog post because some of theme are both ugly and bad, but it doesn't matter.

Just in case you are going to set up a Pyramid project based on MySQL and Windows (sometimes you have to)... there are a couple of things useful to know. But let's start with a chronologically order.

Day 0 - morning

You feel like the brave Steeve McQueen:

Evening - day 0

At the end of the day you'll feel also like Steeve McQueen, but a little more proved:

What happened?

Random problems with normal transactions considered too large, thread disconnected, error log entries like:

• InterfaceError
• OperationalError
• MySQL Server has gone away
• database connection failure
• TimeoutError: QueuePool limit of size ... overflow ... reached, connection timed out, timeout ...
• pyramid process hangs
• etc

The solution

1 - adjust your my.ini options like that:

[mysqld]
max_allowed_packet = 64MB    # adjust this parameter according to your situation
wait_timeout = 28800
interactive_timeout = 2880 

2 - be sure your production.ini file looks like the following one (with Python 2):

sqlalchemy.url = mysql+mysqldb://USER:PASSWORD@127.0.0.1:3306/YOURDB?charset=utf8&use_unicode=0
# For Mysql "MySQL Connection Timeout and SQLAlchemy Connection Pool Recycling" issues see:
# http://docs.sqlalchemy.org/en/latest/core/pooling.html#setting-pool-recycle
# http://douglatornell.ca/blog/2012/01/08/staying-alive/
sqlalchemy.pool_recycle = 3600

3 - you can schedule a restart of your application once a day.

4 - [OPTIONAL, not only Windows related] adjust your SqlAlchemy configuration parameters according to how many threads your server runs. For example (production.ini):

sqlalchemy.pool_size = 20
sqlalchemy.max_overflow = 10

5 - if you are using CherryPy as a Windows service, be sure your  'engine.autoreload.on' option is set to False.


6 - [UPDATE20150401] check your sql-mode if you want to prevent not handled ProxyError exceptions due to input longer than the max size provided on the field model. See https://github.com/Kotti/Kotti/issues/404

 Results

No more exceptions or odd behaviours!

Links

• http://docs.sqlalchemy.org/en/rel_0_9/dialects/mysql.html#unicode-encoding-decoding
• http://docs.sqlalchemy.org/en/rel_0_9/dialects/mysql.html#module-sqlalchemy.dialects.mysql.mysqldb
• http://docs.sqlalchemy.org/en/latest/core/pooling.html#setting-pool-recycle
• http://douglatornell.ca/blog/2012/01/08/staying-alive/

Kotti CMS events - insert subobjects automatically

Yet another small recipe for Kotti CMS: how to initialize automatically a new object once inserted with events, for example adding a subobject.

Use case? When someone creates a UserProfile object /users/name-surname, an event should create automatically a profile image in /users/name-surname/photo (a Kotti image instance).

It is quite simple. Let's see our your_package/events.py module, where IMAGE_ID is equals to 'photo':

import os
from kotti.events import (
    ObjectInsert,
    subscribe,
    notify,
    )
from kotti.resources import Image
from your_package.resources import UserProfile
from your_package.config import IMAGE_ID


@subscribe(ObjectInsert, UserProfile)
def user_profile_added(event):
    obj = event.object

    if IMAGE_ID not in obj.keys():
        image_path = os.path.join(
            os.path.dirname(__file__),
            'data', 'fallback.png'
            )
        with open(image_path, 'rb') as image_file:
            obj[IMAGE_ID] = image_obj = Image(
                title='Image',
                in_navigation=False,
                data=image_file.read(),
                filename=u'fallback.png',
                mimetype=u'image/png',
                )
            notify(ObjectInsert(image_obj, event.request)) 

Notes:

1. the subscribe decorator will register our handler when a UserProfile resource will be inserted
2. we should check if IMAGE_ID is already instanciated (prevent errors on paste)
3. if you want your code will work both for Unix-like or under Windows, use os.path.join instead of a plain data/fallback.png path (path separator issues)
4. the final b in the open is important if you want to write code that works under Windows, see https://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files. On Unix, it doesn’t hurt to append a 'b' to the mode, so you can use it platform-independently for all binary files
5. notify the image insertion

And your_package/__init__.py (in this example I've used the scan method but you could also register your event handlers imperatively):

...
def includeme(config):
    """ Don't add this to your ``pyramid_includes``, but add the
    ``kotti_configure`` above to your ``kotti.configurators`` instead.

    :param config: Pyramid configurator object.
    :type config: :class:`pyramid.config.Configurator`
    """
    ...
    config.scan(__name__)

And... tests of course (your_package/tests/test_events.py):

from pytest import fixture

from kotti.testing import DummyRequest


@fixture
def user_profile(db_session, config, root):
    """ returns dummy UserProfile.
    """
    config.include('your_package')

    from your_package.resources import UserProfile
    root['userprofile'] = userprofile = UserProfile(title='UserProfile')
    from kotti.events import notify
    from kotti.events import ObjectInsert
    notify(ObjectInsert(course, DummyRequest()))    return userprofile


class TestUserProfileWithEvents:

    def test_assert_userprofile_image(self, user_profile):
        from your_package.config import IMAGE_ID
        assert IMAGE_ID in user_profile.keys()
    ... 

You can test also if everything goes right after a copy/paste action (see the Kotti's tests).

Done!

All posts about Kotti

• Pyramid, MySQL and Windows: the good, the ugly and the bad
• Kotti CMS events - insert subobject automatically
• Kotti CMS - how to turn your Kotti CMS into an intranet
• Kotti CMS - how to store arbitrary data with annotations
• How to install Kotti CMS on Windows
• Kotti CMS - avoid types addable in content root
• Kotti CMS - how to create a new content type with an image
• Kotti CMS - workflow reference

All Kotti posts published by @davidemoro:

• http://davidemoro.blogspot.it/search/label/kotti