Brainstorm's snippets (20/226)


  Management command to dump local db and media
django, management_command
  Send email from command line
mail
  Capture Video from Camera
opencv
  Downgrade pg_dump 10 script to 9.x
postgresql
  Git: apply patches
git
  Sentry Notes
sentry
  Deleting GIT branches
git, branch
  Django Custom Error Views setup
django, errors, views
  Many-to-many example using "through" to augment M2M relationships
django, models, M2M
  New Django project check-list
django
  Json prettified fields in Django admin
admin, json
  Sinewave redis PUBSUB example
redis, pubsub
  Removing the output of Django runserver
django
  Virtualbox - Fix Freezing with Linux Guests on Windows 10 Hosts
virtualbox
  Logging in to Django REST and POSTing forms using HTTPie
httpie
  Control Spotlight indexing activities
mac
  How to Use Django's Built-in Login System
django, login
  Using Redis as a Time Series Database
redis
  One-line redis client
redis
  Ajax form submission with Django and Bootstrap Modals
django, forms, ajax

  Management command to dump local db and media

file management/commands/dump_local_data.py

Usage:

usage: manage.py dump_local_data [-h] [--version] [-v {0,1,2,3}]
                                 [--settings SETTINGS]
                                 [--pythonpath PYTHONPATH] [--traceback]
                                 [--no-color] [--dry-run] [--max-age MAX_AGE]
                                 [--no-gzip] [--legacy]
                                 target

Dump local db and media for backup purposes (and optionally remove old backup
files)

positional arguments:
  target                choices: db, media, all

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -v {0,1,2,3}, --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=verbose output, 3=very verbose output
  --settings SETTINGS   The Python path to a settings module, e.g.
                        "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be
                        used.
  --pythonpath PYTHONPATH
                        A directory to add to the Python path, e.g.
                        "/home/djangoprojects/myproject".
  --traceback           Raise on CommandError exceptions
  --no-color            Don't colorize the command output.
  --dry-run, -d         Dry run (simulation)
  --max-age MAX_AGE, -m MAX_AGE
                        If > 0, remove backup files old "MAX_AGE days" or more
  --no-gzip             Do not compress result
  --legacy              use legacy Postgresql command syntax

Code snippet has been moved to:

https://github.com/morlandi/dump_local_data

  Send email from command line

Using mail

mail someone@gmail.com
    Subject: Hello World!
    This is an email to myself.

    Hope all is well.
    ...

and terminate with ^D.

or

echo "this is a test mail" | mail -s'send mail test' testuser@somewhere.com

Using mailx

mailx -s "subject" -r sender@somewhere.com receiver1@somewhere.com,receiver2@somewhere.com < textfile

Using mailx with attachment

mailx -A attchment_file -s "subject" -r sender@somewhere.com receiver1@somewhere.com,receiver2@somewhere.com < textfile

Test what's happening

tail -f /var/log/mail.log

  Capture Video from Camera

import cv2
import time
import shutil


cap = cv2.VideoCapture(0)
snapshot = "/tmp/snapshot.jpg"
snapshot_tmp = "/tmp/snapshot.tmp.jpg"

print('Press "q" to quit ...')

while(True):

    # Capture frame-by-frame
    ret, frame = cap.read()

    # Our operations on the frame come here
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Display the resulting frame
    #cv2.imshow('frame',gray)
    cv2.imshow('frame', frame)

    # Update snapshot
    cv2.imwrite(snapshot_tmp, frame)
    shutil.move(snapshot_tmp, snapshot)
    print('%s "%s" updated' % (
        time.asctime(),
        snapshot
    ))

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

    time.sleep(0.1)

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

References:

  Downgrade pg_dump 10 script to 9.x

#!/usr/bin/env python3
import sys

#
#  Downgrades pg_dump 10 script to 9.x
#  removing 'AS integer' from 'CREATE SEQUENCE' statement
#
#  Usage:
#       $ python3 pgdump_10_to_9.py < test10.sql > test9.sql
#  or:
#       $ cat test10.sql | ./pgdump_10_to_9.py > test9.sql
#
#  To obtain a compressed 9.x sql script from a compressed 10 sql script:
#
#       $ gunzip -c test10.sql.gz | ./pgdump_10_to_9.py | gzip > test9.sql.gz
#

inside_create_sequence = False
for row in sys.stdin.readlines():

    if inside_create_sequence and row.strip().lower() == 'as integer':
        pass
    else:
        print(row, end='', flush=True)

    inside_create_sequence = row.strip().startswith('CREATE SEQUENCE ')

  Git: apply patches

Purpose

Retrieve changes from a remote "working clone" of a project to your local development machine

On remote "working clone" ...

where you've done some changes to be recovered, procede as follows:

  1. Move changes to be recovered to a work branch, and commit them:
git stash
git checkout -b dirty
git stash apply

git add .
git commit -m "my new nice changes"
  1. Check the id of the new commit:
$ git log --oneline
eddc3c6 (HEAD -> dirty) my new nice changes
...
  1. Build a single patch file referring to the specific commit:
git format-patch -o patches/ -1 eddc3c6

or

git format-patch -o patches/ -1 HEAD
  1. Sent it to the developer:

http://brainstorm.it/snippets/send-email-command-line/

or download from development machine via rsync:

rsync -a --rsync-path="sudo rsync" master@host:/home/project/project/patches .
  1. Later (after the changes have been integrated in the main project), you might want to cleanup your working clone:
git checkout master
git branch -D dirty
rm -fr ./patches

On your development machine ...

Apply the changes:

$ git am patches/0001-retrieve_avatars-fixes.patch

Note that, as a result of applying the patch, the changes have already been committed to your working branch:

$ git log --oneline
de4ccec (HEAD -> master) my new nice changes

  Sentry Notes

Setting up Your Development Environment

https://docs.sentry.io/development/contribute/environment/

  Deleting GIT branches

Deleting a local branch:

git branch -d feature/login

To delete a remote branch, we do not use the "git branch" command, but instead "git push" with the "--delete" flag:

git push origin --delete feature/login

If that fails because the remote branch has already been deleted, update your local list of remote branches:

git remote update origin --prune
Note

Please keep in mind that local and remote branches actually have nothing too do with each other. They are completely separate objects in Git. Even if you've established a tracking connection (which you should for most scenarios), this still does not mean that deleting one would delete the other, too!

If you want any branch item to be deleted, you need to delete it explicitly.

  Django Custom Error Views setup

First, create your custom views for error handling:

file main/views.py:

from django.shortcuts import render


def bad_request(request, exception=None, template_name='400.html'):
    return render(request, 'errors/400.html')


def permission_denied(request, exception=None, template_name='403.html'):
    return render(request, 'errors/403.html')


def not_found(request, exception=None, template_name='404.html'):
    return render(request, 'errors/404.html')


def server_error(request, exception=None, template_name='500.html'):
    return render(request, 'errors/500.html')

and the corresponding templates:

file main/templates/errors/error_base.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <!-- Simple HttpErrorPages | MIT License | https://github.com/AndiDittrich/HttpErrorPages -->
    <meta charset="utf-8" /><meta http-equiv="X-UA-Compatible" content="IE=edge" /><meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>{% block pagetitle %} Override Here {% endblock %}</title>
    <style type="text/css">/*! normalize.css v5.0.0 | MIT License | github.com/necolas/normalize.css */html{font-family:sans-serif;line-height:1.15;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}body{margin:0}article,aside,footer,header,nav,section{display:block}h1{font-size:2em;margin:.67em 0}figcaption,figure,main{display:block}figure{margin:1em 40px}hr{box-sizing:content-box;height:0;overflow:visible}pre{font-family:monospace,monospace;font-size:1em}a{background-color:transparent;-webkit-text-decoration-skip:objects}a:active,a:hover{outline-width:0}abbr[title]{border-bottom:none;text-decoration:underline;text-decoration:underline dotted}b,strong{font-weight:inherit}b,strong{font-weight:bolder}code,kbd,samp{font-family:monospace,monospace;font-size:1em}dfn{font-style:italic}mark{background-color:#ff0;color:#000}small{font-size:80%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sub{bottom:-.25em}sup{top:-.5em}audio,video{display:inline-block}audio:not([controls]){display:none;height:0}img{border-style:none}svg:not(:root){overflow:hidden}button,input,optgroup,select,textarea{font-family:sans-serif;font-size:100%;line-height:1.15;margin:0}button,input{overflow:visible}button,select{text-transform:none}[type=reset],[type=submit],button,html [type=button]{-webkit-appearance:button}[type=button]::-moz-focus-inner,[type=reset]::-moz-focus-inner,[type=submit]::-moz-focus-inner,button::-moz-focus-inner{border-style:none;padding:0}[type=button]:-moz-focusring,[type=reset]:-moz-focusring,[type=submit]:-moz-focusring,button:-moz-focusring{outline:1px dotted ButtonText}fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em}legend{box-sizing:border-box;color:inherit;display:table;max-width:100%;padding:0;white-space:normal}progress{display:inline-block;vertical-align:baseline}textarea{overflow:auto}[type=checkbox],[type=radio]{box-sizing:border-box;padding:0}[type=number]::-webkit-inner-spin-button,[type=number]::-webkit-outer-spin-button{height:auto}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}[type=search]::-webkit-search-cancel-button,[type=search]::-webkit-search-decoration{-webkit-appearance:none}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}details,menu{display:block}summary{display:list-item}canvas{display:inline-block}template{display:none}[hidden]{display:none}/*! Simple HttpErrorPages | MIT X11 License | https://github.com/AndiDittrich/HttpErrorPages */body,html{width:100%;height:100%;background-color:#21232a}body{color:#fff;text-align:center;text-shadow:0 2px 4px rgba(0,0,0,.5);padding:0;min-height:100%;-webkit-box-shadow:inset 0 0 100px rgba(0,0,0,.8);box-shadow:inset 0 0 100px rgba(0,0,0,.8);display:table;font-family:"Open Sans",Arial,sans-serif}h1{font-family:inherit;font-weight:500;line-height:1.1;color:inherit;font-size:36px}h1 small{font-size:68%;font-weight:400;line-height:1;color:#777}a{text-decoration:none;color:#fff;font-size:inherit;border-bottom:dotted 1px #707070}.lead{color:silver;font-size:21px;line-height:1.4}.cover{display:table-cell;vertical-align:middle;padding:0 20px}footer{position:fixed;width:100%;height:40px;left:0;bottom:0;color:#a0a0a0;font-size:14px}</style>
</head>
<body>
    <div class="cover">
      <h1>
          {% block title %}Bad Request{% endblock title %}
          <small>{% block subtitle %}Error 400{% endblock subtitle %}</small>
      </h1>
      <p class="lead">
          {% block explain %}The server cannot process the request due to something that is perceived to be a client error.{% endblock explain %}
      </p>

      <br />
      <a href="/">Click here to reload</a>

    </div>
    <footer>
      {% comment %}
      <p>Technical Contact: <a href="mailto:x@example.com">x@example.com</a></p>
      {% endcomment %}
      <p>{{ SITE_COMPANY }} - {{ SITE_TITLE }} - {{ PROJECT_BUILD }}</p>
    </footer>
</body>
</html>

file main/templates/errors/400.html:

{% extends 'errors/error_base.html' %}

{% block pagetitle %}
Bad Request
{% endblock pagetitle %}

{% block title %}
Bad Request
{% endblock title %}

{% block subtitle %}
Error 400
{% endblock subtitle %}

{% block explain %}
The server cannot process the request due to something that is perceived to be a client error.
{% endblock explain %}

file main/templates/errors/403.html:

{% extends 'errors/error_base.html' %}

{% block pagetitle %}
Access Denied
{% endblock pagetitle %}

{% block title %}
Access Denied
{% endblock title %}

{% block subtitle %}
Error 403
{% endblock subtitle %}

{% block explain %}
The requested resource requires an authentication.
{% endblock explain %

file main/templates/errors/404.html:

{% extends 'errors/error_base.html' %}

{% block pagetitle %}
Resource Not Found
{% endblock pagetitle %}

{% block title %}
Resource Not Found
{% endblock title %}

{% block subtitle %}
Error 404
{% endblock subtitle %}

{% block explain %}
The requested resource could not be found but may be available again in the future.
{% endblock explain %}

file main/templates/errors/500.html:

{% extends 'errors/error_base.html' %}

{% block pagetitle %}
Webservice currently unavailable
{% endblock pagetitle %}

{% block title %}
Webservice currently unavailable
{% endblock title %}

{% block subtitle %}
Error 500
{% endblock subtitle %}

{% block explain %}
An unexpected condition was encountered.<br />
Our service team has been dispatched to bring it back online.
{% endblock explain %

Finally, tell Django to use our custom views:

file main/urls.py:

...
handler400 = 'main.views.bad_request'
handler403 = 'main.views.permission_denied'
handler404 = 'main.views.not_found'
handler500 = 'main.views.server_error'
...

Test with DEBUG=False in settings

References:

  Many-to-many example using "through" to augment M2M relationships

On one side of the relation, we do the following:

file models.py

class Device(models.Model):

    ...
    users = models.ManyToManyField(
        User,
        through='DeviceUsers',
        related_name='devices'
    )


class DeviceUsers(models.Model):

    class Meta:
        verbose_name = _("Device/User")
        verbose_name_plural = _("Device/Users")

    device = models.ForeignKey(Device, verbose_name=_('Device'), on_delete=models.CASCADE)
    user = models.ForeignKey(User, verbose_name=_('User'), on_delete=models.CASCADE)
    enabled = models.BooleanField(_('Enabled'), null=False, default=True, blank=True)
    expiration_data = models.DateField(_('Expiration date'), null=True, blank=True)

    def __str__(self):
        return ''

while nothing is required in the related Model.

In the admin, we can:

  • add inlines to edit the relation (in both ModelAdmins),
  • (optionally) list the related models in the listing
  • note that we also add a search_fields attribute to have autocomplete_fields working (Django 2.x)

file admin.py

class UserForDeviceTabularInline(admin.TabularInline):
    model = Device.users.through
    extra = 0
    autocomplete_fields = ['user', ]


@admin.register(Device)
class DeviceAdmin(BaseModelAdmin):

    list_display = [..., 'list_users', ]
    search_fields = ['=id', 'description', 'code', ]
    inlines = [UserForDeviceTabularInline, ]

    def list_users(self, obj):
        return mark_safe(', '.join([
            '<a href="%s">%s</a>' % (user.get_admin_url(), str(user))
            for user in obj.users.all()  # .order_by('panelregisters__position')
        ]))
    list_users.short_description = _('users')


class DeviceTabularInline(admin.TabularInline):
    from backend.models import Device
    model = Device.users.through
    #exclude = ['position', ]
    extra = 0
    autocomplete_fields = ['device', ]


@admin.register(User)
class UserAdmin(AuthUserAdmin, HijackUserAdminMixin):

    list_display = [..., 'list_devices', ]
    inlines = [DeviceTabularInline, ]

    def list_devices(self, obj):
        return mark_safe(', '.join([
            '<a href="%s">%s</a>' % (device.get_admin_url(), str(device))
            for device in obj.devices.all()  # .order_by('panelregisters__position')
        ]))
    list_devices.short_description = _('devices')

TODO: add select_related() as required

  New Django project check-list

A check list of what I normally do when creating a brand new Django project.

Initial project skeleton

  • create a local virtual environment for development, and install lastest version of django in it

  • create a folder container for the project as follows:

    |- customerName
         |- myNewProjectName
               |- logs
               |- dumps
               |- public
               |     |- media
               |     |- static
               |- myNewProjectName  <-- project sources go here
    
  • create the project skeleton with python django-admin startproject and name it main

  • replace settings.py file with a "settings" module, and adjust it as needed; the final project layout is:

    |- customerName
         |- myNewProjectName
               |- logs
               |- dumps
               |- public
               |     |- media
               |     |- static
               |- myNewProjectName
                     |- manage.py
                     |- main
                           |- __init__.py
                           |- debug.py
                           |- local.py
                           |- settings.py
                           |- test_settings.py
                           |- test_settings_no_migrations.py
    

    where:

    main/settings/__init__.py:
    
        from main.settings.local import *
    
    main/settings/local.py:
    
        from main.settings.settings import *
        ...
    
  • main/settings/local.py will contain db settings and the secret_key, and will be excluded from GIT repo

  • add a requirements file to keep track of installed modules and versions

Custom User model

  • create a "users" app
  • use it to override User model (derived from django.contrib.auth.models.AbstractUser)
  • add and configure Hijack app

Versioning and other initializations

  • create a git repo for the project
  • create an "admin_ex" app for a customized (possibly empty) AdminSiteEx(admin.AdminSite)
  • add a main.apps.MainConfig to install admin_ex.admin_site.AdminSiteEx as project admin
  • define admin.site.site_header and admin.site.site_title variables
  • configure bumpversion to manage versioning

Deployment procedure

  • add "django-channels" to have daphne installed
  • setup a provisioning procedure for initial host setup
  • setup a deployment procedure for project installation and update
  • run it over a staging server asap

Email setting

Add helper apps

  • django-constance
  • django-smuggler
  • django-debug-toolbar

Review settings for security

  • python manage.py check --deploy

Sample README.rst file

Project "XXXXX"
===============

Manual provisioning on local development machine
------------------------------------------------

Create environment for new project::

    $ cd /Projects/
    $ mkdir XXXXX
    $ mkdir XXXXX/public
    $ mkdir XXXXX/public/static
    $ mkdir XXXXX/public/media

Create and activate a virtualenv for the project::

    $ mkvirtualenv XXXXX
    $ workon XXXXX

Clone source repository::

    $ cd /Projects/XXXXX
    $ git clone git@gitlab.brainstorm.it:YYYYY/XXXXX.git

The final project layout is::

    .
    ├── XXXXX
    │   ├── README.rst
    │   ├── ...
    │   ├── manage.py
    │   ├── main          <---- this is the Django project
    │   ├── requirements
    │   └── ...
    └── public
        ├── media
        └── static

Update the virtualenv (that is: Install required python packages)::

    $ cd /Project/XXXXX/XXXXX
    $ pip install -r requirements/development.txt

Create a local settings file;
for example, copy and adapt "local_example.py"::

    $ cd /Project/XXXXX/XXXXX
    $ cp main/settings/local_example.py main/settings/local.py

Create database (use database name and password specified in local.py)::

    $ psql
    $ create user XXXXX with encrypted password '<PASSWORD>';
    $ create database XXXXX owner XXXXX;

Populate database struct::

    $ python manage.py migrate

Update front-end assets (if any)::

    $ npm install

Create a supersuser account::

    $ python manage.py createsuperuser

Run the development web server::

    $ python manage.py runserver 0.0.0.0:8000


Development workflow
--------------------

::

    $ cd /Project/XXXXX/XXXXX
    $ workon XXXXX
    $ git pull
    $ git submodule update
    $ pip install -r requirements/development.txt
    $ python manage.py migrate
    $ npm install
    $ python manage.py runserver

Monitoring

  Json prettified fields in Django admin

To render a JSONfield as colored and formatted text in a Django change-view, add a readonly field as follows:

@admin.register(DeviceTestTask)
class DeviceTestTaskAdmin(TaskAdmin):

    exclude = ('program_steps', )

    readonly_fields = [..., 'program_steps_prettified', ]

    def program_steps_prettified(self, obj):
        return json_prettify(obj.program_steps)

where json_prettify() is a helper function responsible to format JSON data using Pygments:

file admin_ex.json_prettify.py

import json
from django.utils.safestring import mark_safe
from pygments import highlight
from pygments.lexers import JsonLexer
from pygments.formatters import HtmlFormatter


def json_prettify_styles():
    """
    Used to generate Pygment styles (to be included in a .CSS file) as follows:
        print(json_prettify_styles())
    """
    formatter = HtmlFormatter(style='colorful')
    return formatter.get_style_defs()


def json_prettify(json_data):
    """
    Adapted from:
    https://www.pydanny.com/pretty-formatting-json-django-admin.html
    """

    # Get the Pygments formatter
    formatter = HtmlFormatter(style='colorful')

    # Highlight the data
    json_text = highlight(
        json.dumps(json_data, sort_keys=True, indent=2),
        JsonLexer(),
        formatter
    )

    # # remove leading and trailing brances
    # json_text = json_text \
    #     .replace('<span class="p">{</span>\n', '') \
    #     .replace('<span class="p">}</span>\n', '')

    # Get the stylesheet
    #style = "<style>" + formatter.get_style_defs() + "</style>"
    style = ''

    # Safe the output
    return mark_safe(style + json_text)

Suitable styles can be produced with:

print(json_prettify_styles())

and included in a CSS file; for example:

/*
 *  Pygment styles
 *
 *  Generated by 'gui.utils.json_prettify_styles()'
 */

.highlight .hll { background-color: #ffffcc }
.highlight .c { color: #888888 } /* Comment */
.highlight .err { color: #FF0000; background-color: #FFAAAA } /* Error */
.highlight .k { color: #008800; font-weight: bold } /* Keyword */

References:

  Sinewave redis PUBSUB example

sinewave publisher

import math, time
import redis

r = redis.StrictRedis(host='localhost', port=6379, db=0)

for n in range(0, 100000):
    row = (62 * 'X')[0:32 + int(30*math.sin(n/8.0))]
    r.publish('sinewave', row)
    #print(row)
    print('\x1b[1;36;40m' + row + '\x1b[0m')
    time.sleep(0.01)

sinewave listener

import redis

r = redis.StrictRedis(host='localhost', port=6379, db=0, decode_responses=True)

pubsub = r.pubsub()
pubsub.subscribe('sinewave')

for item in pubsub.listen():
    if item['type'] == 'message':
        print(item['data'])

  Removing the output of Django runserver

In settings:

#LOGGING_CONFIG = None

LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'handlers': {
        'null': {
            'class': 'logging.NullHandler',
        },
    },
    'loggers': {
        'django.server': {
            'handlers': ['null'],
            'level': 'INFO',
            'propagate': False,
        },
        ...
    },
}

# import logging.config
# logging.config.dictConfig(LOGGING)

If using channels, use 'django.channels.server' logger instead:

'loggers': {
    'django.server': {
        'handlers': ['null'],
        'level': 'INFO',
        'propagate': False,
    },
    'django.channels.server': {
        'handlers': ['null'],
        'level': 'INFO',
        'propagate': False,
    },
},

Filtering:

To print everything except 200 responses, you could use

./manage.py runserver 2>&1 | grep -v " 200 "

  Virtualbox - Fix Freezing with Linux Guests on Windows 10 Hosts

In Settings/Storage/Controller: SATA:

make sure "Use Host I/O Cache" is checked

References:

https://www.youtube.com/watch?v=Xd1VKQnofIs

  Logging in to Django REST and POSTing forms using HTTPie

Logging in to Django REST using HTTPie

Logging in to Django using HTTPie:

http localhost:8000/users/ -h
HTTP/1.0 403 Forbidden
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Date: Wed, 13 Apr 2016 12:01:24 GMT
Server: WSGIServer/0.1 Python/2.7.11
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

First we want to get the CSRF Token by visiting a page:

http localhost:8000/api-auth/login/ --session=session1 -h
HTTP/1.0 200 OK
Cache-Control: no-cache, no-store, must-revalidate, max-age=0
Content-Type: text/html; charset=utf-8
Date: Wed, 13 Apr 2016 12:01:51 GMT
Expires: Wed, 13 Apr 2016 12:01:51 GMT
Last-Modified: Wed, 13 Apr 2016 12:01:51 GMT
Server: WSGIServer/0.1 Python/2.7.11
Set-Cookie:  csrftoken=Z6BWtOS9hmVtZ0QoVkYegMo8Ckduyv7y; expires=Wed, 12-Apr-2017 12:01:51 GMT; Max-Age=31449600; Path=/
Vary: Cookie
X-Frame-Options: SAMEORIGIN

It is now stored in a cookie:

cat session1.json
{
    "__meta__": {
        "about": "HTTPie session file",
        "help": "https://github.com/jkbrzt/httpie#sessions",
        "httpie": "0.9.3"
    },
    "auth": {
        "password": null,
        "type": null,
        "username": null
    },
    "cookies": {
        "csrftoken": {
            "expires": 1491998511,
            "path": "/",
            "secure": false,
            "value": "Z6BWtOS9hmVtZ0QoVkYegMo8Ckduyv7y"
        }
    },
    "headers": {}
}

Then we log in with the X-CSRFToken header set:

http -f POST localhost:8000/api-auth/login/ username=admin password=password123 X-CSRFToken:Z6BWtOS9hmVtZ0QoVkYegMo8Ckduyv7y --session=session1 -h
HTTP/1.0 302 Found
Cache-Control: no-cache, no-store, must-revalidate, max-age=0
Content-Type: text/html; charset=utf-8
Date: Wed, 13 Apr 2016 12:02:46 GMT
Expires: Wed, 13 Apr 2016 12:02:46 GMT
Last-Modified: Wed, 13 Apr 2016 12:02:46 GMT
Location: /accounts/profile/
Server: WSGIServer/0.1 Python/2.7.11
Set-Cookie:  csrftoken=xhZLDurJlYrTKu6JGTwaF0BQinQfV8l4; expires=Wed, 12-Apr-2017 12:02:46 GMT; Max-Age=31449600; Path=/
Set-Cookie:  sessionid=skjcmbsm4a4jtn43sl0npybpdopttk5y; expires=Wed, 27-Apr-2016 12:02:46 GMT; httponly; Max-Age=120960
0; Path=/
Vary: Cookie
X-Frame-Options: SAMEORIGIN

References:

Altra sessione per eseguire un POST complesso via linea comandi

In tutte le chiamate faremo riferimento al file di sessione "./session.json" per garantire la persistenza degli headers custom in successive richieste allo stesso host.

Eseguimo un post sulla pagina di login, che fallira' con errore CSRF_FAILURE_VIEW:

http POST 127.0.0.1:8000/login/ -h
HTTP/1.0 403 FORBIDDEN
Content-Language: it
Content-Type: text/html
Date: Tue, 28 Aug 2018 15:19:51 GMT
Server: WSGIServer/0.1 Python/2.7.13
Set-Cookie: sessionid=70b12537dfdf4fec75fbce86d0d6f6e7; expires=Tue, 11-Sep-2018 15:19:51 GMT; Max-Age=1209600; Path=/
Vary: Accept-Language, Cookie

Per ottenere il csrftoken, consultiamo nuovamente la pagina (metodo GET):

http 127.0.0.1:8000/login/ --session=./session.json -h
HTTP/1.0 200 OK
Cache-Control: max-age=0
Content-Language: it
Content-Type: text/html; charset=utf-8
Date: Tue, 28 Aug 2018 15:23:52 GMT
ETag: "7892c1deb1c4eeca304cada025e5db84"
Expires: Tue, 28 Aug 2018 15:23:52 GMT
Last-Modified: Tue, 28 Aug 2018 15:23:52 GMT
Server: WSGIServer/0.1 Python/2.7.13
Set-Cookie: csrftoken=87035d38eee0ce9c3728c3caf53e21e3; Max-Age=31449600; Path=/
Set-Cookie: sessionid=eb0cc1b2a6c930fb12b76857446a19ed; expires=Tue, 11-Sep-2018 15:23:52 GMT; Max-Age=1209600; Path=/
Vary: Cookie, Accept-Language

Eseguiamo il submit della form di login includendo il csrftoken appena ricevuto:

http -f POST localhost:8000/login/ username=USERNAME password=PASSWORD X-CSRFToken:87035d38eee0ce9c3728c3caf53e21e3 --session=./session.json -h

A questo punto possiamo eseguire il post di arbitrarie form; nel seguente esempio abbiamo utilizzato i dati di una form precedentemente compilata interattivamente, ricavati dall'inspector.

Notare che e' stato aggiunto un "&" al termine per evitare che l'ultimo parametro terminasse con un "n" indesiderato:

 echo "target_user=677&elearning_hours=45&section_3=23&section_3=24&section_4=26&promo_code=&company_vat_id=02409140361&company_name=Brainstorm&company_street_type=Viale&company_address=Francesco+Crispi%2C+2&company_zipcode=41121&company_city=MODENA&company_province=108&company_email=aluppi%40brainstorm.it&company_phone_number=059-216138&accept_conditions=on&accept_clauses=on&privacy_agreement=on&marketing_agreement=accetto&"
| http POST 127.0.0.1:8000/e-commerce/acquisti/ --session=./session.json -h
HTTP/1.0 302 FOUND
Content-Language: it
Content-Type: text/html; charset=utf-8
Date: Tue, 28 Aug 2018 15:38:20 GMT
Location: http://127.0.0.1:8000/e-commerce/acquisto-completato/1543/
Server: WSGIServer/0.1 Python/2.7.13
Set-Cookie: sessionid=f25876fa079fd975c435e7dbc75dba95; expires=Tue, 11-Sep-2018 15:38:20 GMT; Max-Age=1209600; Path=/
Vary: Accept-Language, Cookie

  Control Spotlight indexing activities

Turns off indexing of files:

sudo mdutil -a -i off

To turn indexing back on, run:

sudo mdutil -a -i on

Complete re-indexing of your hard drive (be aware this could be an over night job): it will delete your Spotlight data base forcing it to start over:

sudo rm -rf /.Spotlight-V100/*

Hint:

add others to your (do not scan), privacy settings.

  How to Use Django's Built-in Login System

Adapted to Django 2.1 from:

https://simpleisbetterthancomplex.com/tutorial/2016/06/27/how-to-use-djangos-built-in-login-system.html

Pre-requisites

INSTALLED_APPS = [
    ...
    'django.contrib.auth',
    ...
]

MIDDLEWARE = [
    ...
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    ...
]

Login

Configure the urls used by Django for login and redirection after successfull login:

file "settings.py":

LOGIN_URL = '/login/'
LOGIN_REDIRECT_URL = '/'
LOGOUT_URL = '/logout/'

Configure the URL routes

file "urls.py":

from django.urls import path
from django.conf import settings
from django.urls import include, path, re_path
from django.contrib.auth import views as auth_views


urlpatterns = [
    ...
    path('login/', auth_views.LoginView.as_view(template_name="accounts/login.html"), name="login"),
    ...
]

Create a login template

file "accounts/login.html":

{% extends 'base.html' %}

{% block title %}Login{% endblock %}
{% block body-class %}login{% endblock %}

{% block content %}

    <form method="post">
        <h2>Please sign in</h2>
        {% csrf_token %}
        {{ form.as_p }}
        <div class="submit-row">
            <button class="btn btn-lg btn-primary" type="submit">Login</button>
        </div>
    </form>

{% endblock %}

where (Bootstrap):

/*
 * Login form
 */

body.login form {
    width: 100%;
    max-width: 330px;
    padding: 15px;
    margin: 80px auto;
}

body.login form p {
    padding: 0;
    margin: 0;
}

body.login form input {
    height: auto;
    padding: 10px;
    font-size: 16px;
}

body.login form label {
    padding: 10px;
    width: 30%;
}

body.login h2,
body.login .submit-row {
    margin: 0 auto;
    text-align: center;
    margin: 30px 0;
}

Logout

In the URL routes, you can either have the logout view render a specific template:

urlpatterns = [
    ...
    path('logout/', auth_views.LogoutView.as_view(template_name="accounts/logged_out.html"), name='logout'),
    ...
]

or use the next_page parameter to redirect either to a specific target:

urlpatterns = [
    ...
    path('logout/', auth_views.LogoutView.as_view(next_page=settings.LOGIN_REDIRECT_URL), name='logout'),
    ...
]

  Using Redis as a Time Series Database

Code

#!/usr/bin/env python3
import time
import redis

redis_url = 'redis://localhost:6379/0'


#https://www.infoq.com/articles/redis-time-series
def record_event_with_pipeline(connection, id, event):
    event['id'] = id
    event_key = 'event:{id}'.format(id=id)
    pipe = connection.pipeline(True)
    pipe.hmset(event_key, event)
    pipe.zadd('events', **{id: event['timestamp']})
    pipe.execute()


def record_event(connection, id, event):
    """
    Example:
        "HMSET" "event:1" "timestamp" "1533445741.871906" "temperature" "100" "pressure" "200" "id" "1"
        "ZADD" "events" "1533446149.594696" "1"
    """
    event['id'] = id
    event_key = 'event:{id}'.format(id=id)
    connection.hmset(event_key, event)
    connection.zadd('events', **{id: event['timestamp']})


def main():
    redis_connection = redis.Redis.from_url(redis_url)

    for i in range(1, 11):
        id = 'n_{id}'.format(id=i)
        record_event_with_pipeline(redis_connection, id, {
            'timestamp': time.time(),
            'temperature': 100 + i,
            'pressure': 200 + i
        })

    #record_event_with_pipeline(redis_connection, 'n_1', {'timestamp': time.time(), 'temperature': 100, 'pressure': 200})

    # Now yout can do:
    #
    # > HGET event:n_1 timestamp
    # "1533445741.871906"
    # > HGET event:n_1 id
    # "n_1"
    # > HGET event:n_1 temperature
    # "100"
    # > HGET event:n_1 pressure
    # "200"
    #
    # > ZSCORE 'events' n_1
    # "1533446149.594696"
    # > ZRANGE events 0 -1
    #  1) "n_1"
    #  2) "n_2"
    #  3) "n_3"
    #  4) "n_4"
    #  5) "n_5"
    #  6) "n_6"
    #  7) "n_7"
    #  8) "n_8"
    #  9) "n_9"
    # 10) "n_10"

if __name__ == "__main__":
    main()

  One-line redis client

A stupid simple redis client with no dependencies

#!/usr/bin/env python3
import socket

ip = "127.0.0.1"
port = 6379

s = socket.socket()
s.connect((ip, port))

#
# > SET foo 1000
#

key = "foo"
value = "1000"
print('> set %s %s' % (key, value))
s.send(
    ("*3\r\n$3\r\nSET\r\n$%d\r\n%s\r\n$%d\r\n%s\r\n" % (
        len(key),
        key,
        len(value),
        value
    )).encode('utf-8'))
print(s.recv(256))

#
# > RPUSH mylist uno two three
#

listname = "mylist"
items = ["one", "two", "three", ]
print('> rpush %s %s' % (list, ' '.join(items)))
command = "*%d\r\n$5\r\nRPUSH\r\n$%d\r\n%s\r\n" % (
    2 + len(items),
    len(listname),
    listname,
)
for item in items:
    command += '$%d\r\n%s\r\n' % (
        len(item),
        item
    )
s.send(command.encode('utf-8'))
print(s.recv(256))

#
# > PUBLISH mychannel mymessage
#

channel = "mychannel"
message = "mymessage"
print('> publish %s %s' % (channel, message))
s.send(
    ("*3\r\n$7\r\nPUBLISH\r\n$%d\r\n%s\r\n$%d\r\n%s\r\n" % (
        len(channel),
        channel,
        len(message),
        message
    )).encode('utf-8')
)
print(s.recv(256))

s.close()

  Ajax form submission with Django and Bootstrap Modals

The purpose of this snippet is to render a Django form either as modal window or standalone page, with the following requirements:

  • minimize code repetition
  • update the modal with any form errors, then close the modal on successful submission
  • process the form normally in the standalone page

Prepare the modal template to contain the form

In this example, the modal is styled for Bootstrap.

Note that the modal body is initially empty; it will be loaded dynamically just before showing it.

file 'templates/frontend/modals/register_edit.html'

{% load i18n %}

<div class="modal inmodal" id="modal_register_edit" tabindex="-1" role="dialog" aria-hidden="true">
    <div class="modal-dialog modal-lg">
    <div class="modal-content animated bounceInRight">
            <div class="modal-header">
                <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">&times;</span>
                    <span class="sr-only">{% trans 'Close' %}</span>
                </button>
                <h4 class="modal-title"><i class="fa fa-laptop modal-icon"></i> {% trans 'Register edit' %}</h4>
                <small class="font-bold">{% trans 'Insert new values to modify Regiser attributes as required' %}</small>
            </div>
            <div class="modal-body">
            </div>
            <div class="modal-footer">
                <button type="button" class="btn btn-white" data-dismiss="modal">{% trans 'Close' %}</button>
                <button id="btn-save" type="button" class="btn btn-primary">{% trans 'Save changes' %}</button>
            </div>
        </div>
    </div>
</div>

Display the modal

The modal can be invoked via javascript, for example from a link, passing the event and any other required parameters to the handler:

<a href="{% url 'frontend:register-edit' register.id %}"
   onclick="openRegisterEditModal(event, '{{register.id}}'); return false;">
    {% trans 'Edit' %}
</a>

Note that we also assigned to the "href" attribute the end-point in charge for the form rendering; we'll retrieve it later in the handler.

In the handler, we first update the modal's body with the HTML loaded asynchronously from the server, and then (after the ajax call) display it.

file: 'static/frontend/js/register_edit.js'

function openRegisterEditModal(event, register_id) {
    var modal = $('#modal_register_edit');
    var url = $(event.target).closest('a').attr('href');
    modal.find('.modal-body').html('').load(url, function() {
        modal.modal('show');
        formAjaxSubmit(popup, url);
    });
}

Please note that we clear the modal body - html('') - before calling load(), to prevent showing an obsolete content in case of load failure.

The role of the formAjaxSubmit() helper is explained below.

Eventually, you might desire more control over load failure, for example if the server refuses to provide the form for inconsistent user permissions:

@login_required
def register_edit(request, pk):

    if not request.user.has_perm('backend.change_register'):
        raise PermissionDenied

    ...

If this is the case, use $.ajax() instead of load:

function openRegisterEditModal(event, register_id) {
    var modal = prepare_generic_modal(element);
    var url = $(event.target).closest('a').attr('href');
    $.ajax({
        type: "GET",
        url: url
    }).done(function(data, textStatus, jqXHR) {
        modal.find('.modal-body').html(data);
        modal.modal('show');
        formAjaxSubmit(modal, url, updateRegisterCalibrationChart);
    }).fail(function(jqXHR, textStatus, errorThrown) {
        alert(errorThrown);
    });
}

Form validation in the modal

If the modal body contains a form, we need to prevent it from performing its default submit action.

We do this binding to the form's submit event, where we serialize the form's content and sent it via an Ajax call using the form’s defined method.

If the form specifies an action, we use it as end-point of the ajax call; if not, we're using the same view for both rendering and form processing, so we can reuse the original url instead.

Upon a successufull response from the server, we need to further investigate the HTML received: if it contains any field errors, the form did not validate successfully, so we update the modal body with the new form and its error; otherwise, the modal will be closed.

file: 'static/frontent/js/utils.js'

'use strict';

function formAjaxSubmit(modal, action) {
    var form = modal.find('.modal-body form');
    var footer = $(modal).find('.modal-footer');

    // bind to the form’s submit event
    $(form).on('submit', function(event) {

        // prevent the form from performing its default submit action
        event.preventDefault();
        footer.addClass('loading');

        // either use the action supplied by the form, or the original rendering url
        var url = $(this).attr('action') || action;

        // serialize the form’s content and sent via an AJAX call
        // using the form’s defined method
        $.ajax({
            type: $(this).attr('method'),
            url: url,
            data: $(this).serialize(),
            success: function(xhr, ajaxOptions, thrownError) {

                // If the server sends back a successful response,
                // we need to further check the HTML received

                // If xhr contains any field errors, the form did not
                // validate successfully, so we update the modal body
                // with the new form and its error
                if ($(xhr).find('.has-error').length > 0) {
                    $(modal).find('.modal-body').html(xhr);
                    formAjaxSubmit(modal, url);
                } else {
                    // otherwise, we've done and can close the modal
                    $(modal).modal('hide');
                }
            },
            error: function(xhr, ajaxOptions, thrownError) {
            },
            complete: function() {
                footer.removeClass('loading');
            }
        });
    });
}
/*
    Add spinner while new image gets loaded completely:
      https://stackoverflow.com/questions/21258195/add-spinner-while-new-image-gets-loaded-completely#21262404

    Set the CSS background-image property of the images to a loading spinner graphic;
    once the actual image is downloaded, it covers up the "loading" animated GIF background image.
*/

.loading {
    background: transparent url(/static/frontend/images/spinner.gif) no-repeat scroll center center;
}

Server side processing

The view responsible to process the form server-side has no special requirements, except that we want to use it for both modals and as a full standalone page.

We obtain this by splitting the rendering template as follows, and using the "inner" version whenever we detect an ajax request.

file: 'templates/frontend/includes/register_edit_form.html'

{% extends "base.html" %}
{% load static staticfiles i18n %}

{% block content %}
{% include 'frontend/includes/register_edit_form_inner.html' %}
{% endblock content %}

file: 'templates/frontend/includes/register_edit_form_inner.html'

{% load i18n bootstrap3 %}

<div class="row">
    <div class="col-sm-4">
        <form action="{% url 'frontend:register-edit' register.id %}" method="post" class="form">
            {% csrf_token %}
            {% bootstrap_form form %}
            {% buttons %}
                <div class="form-submit-row">
                    <button type="submit" class="btn btn-primary">
                        {% bootstrap_icon "star" %} {% trans 'Send' %}
                    </button>
                </div>
            {% endbuttons %}
        </form>
    </div>
</div>

As explained above, the form action is optional in this context.

And finally, here's the view code:

file: 'urls.py'

urlpatterns = [
    ...
    path('register/<uuid:pk>/edit', views.register_edit, name="register-edit"),

file: 'views.py'

from django.shortcuts import render
from .forms import RegisterEditForm

def register_edit(request, pk):

    # Either render only the modal content, or a full standalone page
    if request.is_ajax():
        template_name = 'frontend/includes/register_edit_form_inner.html'
    else:
        template_name = 'frontend/includes/register_edit_form.html'

    register = get_object_by_uuid_or_404(Register, pk)

    if request.method == 'POST':
        form = RegisterEditForm(instance=register, data=request.POST)
        if form.is_valid():
            form.save()
    else:
        form = RegisterEditForm(instance=register)

    return render(request, template_name, {
        'register': register,
        'form': form,
    })

Use a single modal for different purposes

Provided you have a suitable template for a generic modal:

{% load i18n %}

<div class="modal inmodal" id="modal_generic" tabindex="-1" role="dialog" aria-hidden="true">
    <div class="modal-dialog modal-lg">
    <div class="modal-content animated bounceInRight">
            <div class="modal-header">
                <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">&times;</span>
                    <span class="sr-only">{% trans 'Close' %}</span>
                </button>
                <h4><i class="fa fa-laptop modal-icon"></i> <span class="modal-title"></span></h4>
                <small class="modal-subtitle font-bold"></small>
            </div>
            <div class="modal-body">
            </div>
            <div class="modal-footer">
                <button type="button" class="btn btn-white" data-dismiss="modal">{% trans 'Close' %}</button>
                <button id="btn-save" type="button" class="btn btn-primary">{% trans 'Save changes' %}</button>
            </div>
        </div>
    </div>
</div>

you might want to specify a custom title and subtitle in the calling template (where strings can be easily localized):

<a href="{% url 'frontend:register-set_value' register.id %}"
   data-title="{% trans 'Register set value' %}"
   data-subtitle="{% trans 'Insert the new value to be assigned to the Register' %}"
   onclick="openRegisterSetValueModal(event, '{{register.id}}'); return false;">
    <i class="fa fa-keyboard-o"></i> {{ register.description }}
</a>

and set the appropriate values from javascript before showing the modal:

function openRegisterSetValueModal(event, register_id) {
    var modal = $('#modal_generic');
    var element = $(event.target).closest('a');
    modal.find('.modal-title').text(element.data('title'));
    modal.find('.modal-subtitle').text(element.data('subtitle'));
    ...
}

Take extra action after successful submit

In this use case, we want to update the main HTML page after a modal form has been successfully submitted, to reflect any possible change.

We start by adding a second cbAfterSuccess callback to our formAjaxSubmit() helper:

function formAjaxSubmit(modal, action, cbAfterLoad, cbAfterSuccess) {
    var form = modal.find('.modal-body form');
    var header = $(modal).find('.modal-header');

    // use footer save button, if available
    var btn_save = modal.find('.modal-footer #btn-save');
    if (btn_save) {
        modal.find('.modal-body form .form-submit-row').hide();
        btn_save.off().on('click', function(event) {
            modal.find('.modal-body form').submit();
        });
    }
    if (cbAfterLoad) { cbAfterLoad(modal); }

    setTimeout(function() {
        modal.find('form input:visible').first().focus();
    }, 1000);

    // bind to the form’s submit event
    $(form).on('submit', function(event) {

        // prevent the form from performing its default submit action
        event.preventDefault();
        header.addClass('loading-left');

        var url = $(this).attr('action') || action;

        // serialize the form’s content and sent via an AJAX call
        // using the form’s defined action and method
        $.ajax({
            type: $(this).attr('method'),
            url: url,
            data: $(this).serialize(),
            success: function(xhr, ajaxOptions, thrownError) {

                // If the server sends back a successful response,
                // we need to further check the HTML received

                // update the modal body with the new form
                $(modal).find('.modal-body').html(xhr);

                // If xhr contains any field errors,
                // the form did not validate successfully,
                // so we keep it open for further editing
                if ($(xhr).find('.has-error').length > 0) {
                    formAjaxSubmit(modal, url, cbAfterLoad, cbAfterSuccess);
                } else {
                    // otherwise, we've done and can close the modal
                    $(modal).modal('hide');
                    if (cbAfterSuccess) { cbAfterSuccess(modal); }
                }
            },
            error: function(xhr, ajaxOptions, thrownError) {
            },
            complete: function() {
                header.removeClass('loading-left');
            }
        });
    });
}

When opening the modal, we supply the address of a suitable handler to be invoked after successfull submit; note that we also store some extra data into the modal object, for later reference in afterRegisterEditModalSuccess():

function openRegisterEditModal(event, register_id) {
    var modal = $('#modal_register_edit');
    modal.data('register-id', register_id);
    modal.data('target', event.target);
    var element = $(event.target).closest('a');
    var url = element.attr('href');
    modal.find('.modal-body').load(url, function() {
        modal.modal('show');
        formAjaxSubmit(modal, url, null, afterRegisterEditModalSuccess);
    });
}

function afterRegisterEditModalSuccess(modal) {
    var url = sprintf('/register/%s/render', modal.data('register-id'));
    var element = $(modal.data('target').closest('.register.widget'));
    var container = element.parent();
    container.load(url, function() {
        var new_element = container.find('.register.widget');
        init_register_widgets(new_element);
    });
}