Brainstorm's snippets (20/233)


  Django delete FileField
django
  PUB/SUB in javascript
javascript, pubsub
  Prometheus Guide
monitoring
  pyclean alias
python
  How to Install Node.js and NPM on a Mac
mac, npm, nodejs
  Monit guide (updated)
monitoring
  Flat list out of list of lists in Python
python
  Raspberry PI setup guide
setup, raspberrypi
  Run ssh-agent and ssh-add on login via SSH
ssh
  Management command to dump local db and media
django, management_command
  Send email from command line
mail
  Capture Video from Camera
opencv
  Downgrade pg_dump 10 script to 9.x
postgresql
  Git: apply patches
git
  Sentry Notes
sentry
  Deleting GIT branches
git, branch
  Django Custom Error Views setup
django, errors, views
  Many-to-many example using "through" to augment M2M relationships
django, models, M2M
  New Django project check-list
django
  Json prettified fields in Django admin
admin, json

  Django delete FileField

import os
from django.db import models
from django.dispatch import receiver


def remove_file_and_cleanup(filepath):
    """
    Helper to remove a media file;
    also removes the containing folder, if left empty
    """
    folder = os.path.dirname(filepath)
    # remove file
    if os.path.isfile(filepath):
        os.remove(filepath)
    # finally, remove folder if empty
    if os.path.isdir(folder) and len(os.listdir(folder)) <= 0:
        os.rmdir(folder)


@receiver(models.signals.post_delete, sender=MyModel)
def auto_delete_file_on_delete(sender, instance, **kwargs):
    """ Deletes file from filesystem when corresponding `MediaFile` object is deleted.
        Adapted from: http://stackoverflow.com/questions/16041232/django-delete-filefield
    """

    # Collect names of FileFields
    fieldnames = [f.name for f in instance._meta.get_fields() if isinstance(f, models.FileField)]
    for fieldname in fieldnames:
        field = getattr(instance, fieldname)
        if bool(field):
            remove_file_and_cleanup(field.path)


@receiver(models.signals.pre_save, sender=MyModel)
def auto_delete_file_on_change(sender, instance, **kwargs):
    """ Deletes file from filesystem when corresponding object is changed or removed.
        Adapted from: http://stackoverflow.com/questions/16041232/django-delete-filefield
    """

    if not instance.pk or sender.objects.filter(pk=instance.pk).count() <= 0:
        return False

    fieldnames = [f.name for f in instance._meta.get_fields() if isinstance(f, models.FileField)]
    for fieldname in fieldnames:
        try:
            old_field = getattr(sender.objects.get(pk=instance.pk), fieldname)
            old_filepath = old_field.path

            new_field = getattr(instance, fieldname)
            new_filepath = new_field.path if new_field else None

            # if ready to save a new file, delete the old one
            if old_filepath != new_filepath:
                remove_file_and_cleanup(old_filepath)
        except:
            pass

  PUB/SUB in javascript

file publisher.js:

function initialize_datatable(element, url) {

    $.ajax({
        type: 'GET',
        url: url + '?action=initialize',
        dataType: 'json'
    }).done(function(data, textStatus, jqXHR) {

        ...
        var table = ...

        // Notify subscribers
        $('.subscribe-datatable-initialized').trigger(
            'datatableInitialized', [table]
        );

    });

}

file sample_subscriber.html:

<div class="panel panel-danger subscribe-datatable-initialized" id="stale-acquisition-panel" style="display: none;">

    <p>This panel is to be shown only after datatable has been initialized</p>

</div>

<script language="javascript">

    $(document).ready(function() {
        $('#stale-acquisition-panel').on('datatableInitialized', function(e, eventInfo) {
            $(e.target).show();
        });
    });

</script>

  Prometheus Guide

  pyclean alias

alias pyclean="find . \( -name \*.pyc -o -name \*.pyo -o -name __pycache__ \) -prune -exec rm -rf {} +"

  How to Install Node.js and NPM on a Mac

Install node with homebrew:

brew install node

To see if Node is installed:

node -v
    v10.12.0

To see if NPM is installed:

npm -v
    6.4.1

How to Update Node and NPM

brew update
brew upgrade node

How to Uninstall Node and NPM

brew uninstall node

Hot to use

Then in your project you can finally populate node_modules from package.json as follows:

npm install

How to Install Sass compiler on a Mac

Check if ruby is available:

ruby -v

If not, install it:

brew install ruby

Install Sass:

sudo gem install sass

In case of SSL error "SSL Error When installing rubygems, Unable to pull data from 'https://rubygems.org/", try this:

sudo gem sources -r https://rubygems.org
sudo gem sources -a http://rubygems.org

the again:

sudo gem install sass

Eventuale aggiornamento dei requirements:

pip install -r requirements/development.txt

  Monit guide (updated)

Install Monit System Monitor On Ubuntu 18.04

sudo apt update
sudo apt install monit

After installing Monit, the commands below can be used to stop, start and enable Monit service:

sudo systemctl stop monit.service
sudo systemctl start monit.service
sudo systemctl enable monit.service

Configure Monit service

By default all files located on /etc/monit/conf.d/ and /etc/monit/conf-enabled/ are read by monit when the service is started.

Use the /etc/monit/conf.d/ directory to put all your monitoring configuration files in it.

Example: file /etc/monit/conf.d/disk:

check device root with path /
    if SPACE usage > 80% then alert

check device backup_disk with path /mnt/backup
    if SPACE usage > 80% then alert

then:

sudo monit check -t
sudo monit reload
sudo monit start all

Check Monit status

sudo monit status

Access Monit Web Portal

Add this to file /etc/monit/monitrc:

set httpd port 2812
    allow admin:password

then access the web page from anywhere with:

http://host:2812

or:

http://host:2812/_status?format=xml

  Flat list out of list of lists in Python

Given superlist as a list of lists,

flat_list = [item for sublist in superlist for item in sublist]

which means:

flat_list = []
for sublist in superlist:
    for item in sublist:
        flat_list.append(item)

References:

  Raspberry PI setup guide

1   Initial setup

Eventually expand file system if part of the SD is unused:

raspi-config --expand-rootfs

2   Enable SSH connections

Connect with a keyboard, then issue these commands from terminal:

systemctl enable ssh.service
systemctl start ssh.service

3   Set locale

sudo -s
export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
locale-gen en_US.UTF-8
dpkg-reconfigure locales

5   Expand swap partition ...

100 Mb (the default) is too small, and if you are doing memory intensive stuff (e.g., web surfing), you can easily max it out.

The recommended swap size is 2*physical RAM - in the case of the Pi (modern/current versions of), this is 2G.

You can increase the swap size by changing the CONF_SWAPSIZE or the CONF_SWAPFACTOR parameter in /etc/dphys-swapfile followed by a reboot:

cat /etc/dphys-swapfile

[...]
CONF_SWAPFACTOR=2
[...]

Test after reboot:

# free -h
              total        used        free      shared  buff/cache   available
Mem:           927M        259M        425M         30M        242M        585M
Swap:          1.8G          0B        1.8G

6   ... or remove it !

On the other hand, you might want to turn off swap entirely in order to reduce the amount of write operations on the SD card – because SD cards have their life limited to the amount of write operations:

sudo systemctl disable dphys-swapfile

Test after reboot:

# free -h
              total        used        free      shared  buff/cache   available
Mem:           927M        257M        437M         24M        232M        593M
Swap:            0B          0B          0B

Another option is to move swap to an external device (for example a USB key).

References:

7   Setup SMTP

TODO; vedere:

Installazione e Configurazione di Postfix su Raspberry usando come Smarthost GMAIL:

https://www.raffaelechiatto.com/installazione-configurazione-postfix-raspberry-usando-smarthost-gmail/

NULLMAILER – IL POSTINO MINIMALISTA:

https://hamradio.fe.linux.it/nullmailer-il-postino-minimalista/

8   Start chromium in Kiosk mode on raspbian jessie

file ~/Desktop/runChromium.desktop:

[Desktop Entry]
Type=Application
Exec=/usr/bin/chromium-browser --noerrdialogs --disable-session-crashed-bubble --disable-infobars --kiosk http://127.0.0.1
Hidden=false
X-GNOME-Autostart-enabled=true
Name[en_US]=RunChromium
Name=RunChromium
Comment=Start Chromium in kiosk mode; copy int ~/.config/autostart to have it run automatically

References:

9   Pi display

How to hide the cursor in the kiosk mode automatically:

sudo apt-get install unclutter

then add this to file /etc/xdg/lxsession/LXDE/autostart:

@unclutter -idle 0.1 -root

10   Display rotation

file "/boot/config.txt":

# LCD Rotation
lcd_rotate=2

# Display Rotate (HDMI)
#display_rotate=2

11   Disable screen sleep

file "/etc/lightdm/lightdm.conf":

[Seat:*]
...
# don't sleep the screen
xserver-command=X -s 0 dpms

or:

sudo apt-get install xscreensaver

then configure the screensaver application under the Preferences option on the main desktop menu.

12   How to up a Static IP on Your Ethernet or Wireless Network Connection

file /etc/dhcpcd.conf:

# setup ethernet static ip
interface eth0
inform 192.168.1.18
static routers=255.255.255.0

# setup wireless static ip
interface wlan0
inform 192.168.1.19
static routers=255.255.255.0

or:

# setup ethernet static ip
interface eth0
static ip_address=172.26.11.85/24
static routers=172.26.11.100
static domain_name_servers=8.8.8.8 8.8.4.4

then:

#sudo /etc/init.d/networking restart
sudo reboot

https://projects.raspberrypi.org/en/projects/getting-started-with-picamera

13   Setting up a Raspberry Pi as an access point in a standalone network (NAT)

Brief summary:

Install required software:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install dnsmasq hostapd

then stop the services and reboot:

sudo systemctl stop dnsmasq
sudo systemctl stop hostapd
sudo reboot

Configuring a static IP for wlan0:

add this to /etc/dhcpcd.conf:

interface wlan0
    static ip_address=192.168.4.1/24
    nohook wpa_supplicant

then:

sudo service dhcpcd restart

Configuring the DHCP server (dnsmasq):

sudo mv /etc/dnsmasq.conf /etc/dnsmasq.conf.orig
sudo vim /etc/dnsmasq.conf

and add:

interface=wlan0      # Use the require wireless interface - usually wlan0
    dhcp-range=192.168.4.2,192.168.4.20,255.255.255.0,24h

Configuring the access point host software (hostapd):

add this to /etc/hostapd/hostapd.conf:

interface=wlan0
driver=nl80211
ssid=NETWORK_NAME
hw_mode=g
channel=7
wmm_enabled=0
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_passphrase=NETWORK_PASSPHRASE
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

where:

  • NETWORK_PASSPHRASE: between 8 and 64 characters

  • hw_mode:

    • a = IEEE 802.11a (5 GHz)
    • b = IEEE 802.11b (2.4 GHz)
    • g = IEEE 802.11g (2.4 GHz)
    • ad = IEEE 802.11ad (60 GHz)

Add this to /etc/default/hostapd:

DAEMON_CONF="/etc/hostapd/hostapd.conf"

Start the access point host software:

sudo systemctl start hostapd
sudo systemctl start dnsmasq

if necessary:

sudo systemctl unmask hostapd
sudo systemctl enable hostapd
sudo systemctl start hostapd

Add routing and masquerade:

Uncomment this line in /etc/sysctl.conf:

net.ipv4.ip_forward=1

Add a masquerade for outbound traffic on eth0:

sudo iptables -t nat -A  POSTROUTING -o eth0 -j MASQUERADE

Save the iptables rule:

sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"

Edit /etc/rc.local and add this just above "exit 0" to install these rules on boot:

iptables-restore < /etc/iptables.ipv4.nat

Reboot and done !

References:

Setting up a Raspberry Pi as an access point in a standalone network (NAT)

15   Installing Python 3.7.x on Raspbian

sudo su
cd
mkdir downloads
cd downloads

apt-get update -y
apt-get install build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev libffi-dev -y
wget https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tar.xz
tar xf Python-3.7.2.tar.xz
cd Python-3.7.2
./configure
make -j 4
make altinstall
cd ..
rm -r Python-3.7.2
rm Python-3.7.2.tar.xz
apt-get --purge remove build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev libffi-dev -y
apt-get autoremove -y
apt-get clean

References:

16   todo

All my Pi's (including Pi Zero's) have these two lines added to /etc/fstab:

tmpfs /tmp tmpfs defaults,noatime,nosuid 0 0
tmpfs /var/log tmpfs defaults,noatime,nosuid,size=16m 0 0

The default maximum size is half the memory, but of course tmpfs only takes as much memory as the files need.

While you are in /etc/fstab, if you want to reduce writes to the SD then the other simple change is the flush rate.

For the ext4 / mount make sure the options include "commit=600" That is, for example:

PARTUUID=e96d960e-02 / ext4 defaults,noatime,commit=600,errors=remount-ro 0 1

Obviously do not do this if your site is prone to power cuts or other unexpected outages. (dirty pages in the disk cache are written out every ten minutes instead of every five seconds which is the default). This improves performance as well.

  Run ssh-agent and ssh-add on login via SSH

You can try adding this:

eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa

This way the ssh-agent does not start a new shell, it just launches itself in the background and spits out the shell commands to set the appropriate environment variables.

As said in the comment, maybe you do not want to run the agent at all on the remote host, but rather on the box you are working from, and use:

ssh -A remote-host

to forward the services of your local ssh agent to the remote-host.

For security reasons you should only use agent forwarding with hosts run by trustworthy people, but it is better than running a complete agent remotely any time.

  Management command to dump local db and media

file management/commands/dump_local_data.py

Usage:

usage: manage.py dump_local_data [-h] [--version] [-v {0,1,2,3}]
                                 [--settings SETTINGS]
                                 [--pythonpath PYTHONPATH] [--traceback]
                                 [--no-color] [--dry-run] [--max-age MAX_AGE]
                                 [--no-gzip] [--legacy]
                                 target

Dump local db and media for backup purposes (and optionally remove old backup
files)

positional arguments:
  target                choices: db, media, all

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -v {0,1,2,3}, --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=verbose output, 3=very verbose output
  --settings SETTINGS   The Python path to a settings module, e.g.
                        "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be
                        used.
  --pythonpath PYTHONPATH
                        A directory to add to the Python path, e.g.
                        "/home/djangoprojects/myproject".
  --traceback           Raise on CommandError exceptions
  --no-color            Don't colorize the command output.
  --dry-run, -d         Dry run (simulation)
  --max-age MAX_AGE, -m MAX_AGE
                        If > 0, remove backup files old "MAX_AGE days" or more
  --no-gzip             Do not compress result
  --legacy              use legacy Postgresql command syntax

Code snippet has been moved to:

https://github.com/morlandi/dump_local_data

  Send email from command line

Using mail

mail someone@gmail.com
    Subject: Hello World!
    This is an email to myself.

    Hope all is well.
    ...

and terminate with ^D.

or

echo "this is a test mail" | mail -s'send mail test' testuser@somewhere.com

Using mailx

mailx -s "subject" -r sender@somewhere.com receiver1@somewhere.com,receiver2@somewhere.com < textfile

Using mailx with attachment

mailx -A attchment_file -s "subject" -r sender@somewhere.com receiver1@somewhere.com,receiver2@somewhere.com < textfile

Test what's happening

tail -f /var/log/mail.log

  Capture Video from Camera

import cv2
import time
import shutil


cap = cv2.VideoCapture(0)
snapshot = "/tmp/snapshot.jpg"
snapshot_tmp = "/tmp/snapshot.tmp.jpg"

print('Press "q" to quit ...')

while(True):

    # Capture frame-by-frame
    ret, frame = cap.read()

    # Our operations on the frame come here
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Display the resulting frame
    #cv2.imshow('frame',gray)
    cv2.imshow('frame', frame)

    # Update snapshot
    cv2.imwrite(snapshot_tmp, frame)
    shutil.move(snapshot_tmp, snapshot)
    print('%s "%s" updated' % (
        time.asctime(),
        snapshot
    ))

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

    time.sleep(0.1)

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

References:

  Downgrade pg_dump 10 script to 9.x

#!/usr/bin/env python3
import sys

#
#  Downgrades pg_dump 10 script to 9.x
#  removing 'AS integer' from 'CREATE SEQUENCE' statement
#
#  Usage:
#       $ python3 pgdump_10_to_9.py < test10.sql > test9.sql
#  or:
#       $ cat test10.sql | ./pgdump_10_to_9.py > test9.sql
#
#  To obtain a compressed 9.x sql script from a compressed 10 sql script:
#
#       $ gunzip -c test10.sql.gz | ./pgdump_10_to_9.py | gzip > test9.sql.gz
#

inside_create_sequence = False
for row in sys.stdin.readlines():

    if inside_create_sequence and row.strip().lower() == 'as integer':
        pass
    else:
        print(row, end='', flush=True)

    inside_create_sequence = row.strip().startswith('CREATE SEQUENCE ')

  Git: apply patches

Purpose

Retrieve changes from a remote "working clone" of a project to your local development machine

On remote "working clone" ...

where you've done some changes to be recovered, procede as follows:

  1. Move changes to be recovered to a work branch, and commit them:
git stash
git checkout -b dirty
git stash apply

git add .
git commit -m "my new nice changes"
  1. Check the id of the new commit:
$ git log --oneline
eddc3c6 (HEAD -> dirty) my new nice changes
...
  1. Build a single patch file referring to the specific commit:
git format-patch -o patches/ -1 eddc3c6

or

git format-patch -o patches/ -1 HEAD
  1. Sent it to the developer:

http://brainstorm.it/snippets/send-email-command-line/

or download from development machine via rsync:

rsync -a --rsync-path="sudo rsync" master@host:/home/project/project/patches .
  1. Later (after the changes have been integrated in the main project), you might want to cleanup your working clone:
git checkout master
git branch -D dirty
rm -fr ./patches

On your development machine ...

Apply the changes:

$ git am patches/0001-retrieve_avatars-fixes.patch

Note that, as a result of applying the patch, the changes have already been committed to your working branch:

$ git log --oneline
de4ccec (HEAD -> master) my new nice changes

  Sentry Notes

Setting up Your Development Environment

https://docs.sentry.io/development/contribute/environment/

  Deleting GIT branches

Deleting a local branch:

git branch -d feature/login

To delete a remote branch, we do not use the "git branch" command, but instead "git push" with the "--delete" flag:

git push origin --delete feature/login

If that fails because the remote branch has already been deleted, update your local list of remote branches:

git remote update origin --prune
Note

Please keep in mind that local and remote branches actually have nothing too do with each other. They are completely separate objects in Git. Even if you've established a tracking connection (which you should for most scenarios), this still does not mean that deleting one would delete the other, too!

If you want any branch item to be deleted, you need to delete it explicitly.

  Django Custom Error Views setup

First, create your custom views for error handling:

file main/views.py:

from django.shortcuts import render


def bad_request(request, exception=None, template_name='400.html'):
    return render(request, 'errors/400.html')


def permission_denied(request, exception=None, template_name='403.html'):
    return render(request, 'errors/403.html')


def not_found(request, exception=None, template_name='404.html'):
    return render(request, 'errors/404.html')


def server_error(request, exception=None, template_name='500.html'):
    return render(request, 'errors/500.html')

and the corresponding templates:

file main/templates/errors/error_base.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <!-- Simple HttpErrorPages | MIT License | https://github.com/AndiDittrich/HttpErrorPages -->
    <meta charset="utf-8" /><meta http-equiv="X-UA-Compatible" content="IE=edge" /><meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>{% block pagetitle %} Override Here {% endblock %}</title>
    <style type="text/css">/*! normalize.css v5.0.0 | MIT License | github.com/necolas/normalize.css */html{font-family:sans-serif;line-height:1.15;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}body{margin:0}article,aside,footer,header,nav,section{display:block}h1{font-size:2em;margin:.67em 0}figcaption,figure,main{display:block}figure{margin:1em 40px}hr{box-sizing:content-box;height:0;overflow:visible}pre{font-family:monospace,monospace;font-size:1em}a{background-color:transparent;-webkit-text-decoration-skip:objects}a:active,a:hover{outline-width:0}abbr[title]{border-bottom:none;text-decoration:underline;text-decoration:underline dotted}b,strong{font-weight:inherit}b,strong{font-weight:bolder}code,kbd,samp{font-family:monospace,monospace;font-size:1em}dfn{font-style:italic}mark{background-color:#ff0;color:#000}small{font-size:80%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sub{bottom:-.25em}sup{top:-.5em}audio,video{display:inline-block}audio:not([controls]){display:none;height:0}img{border-style:none}svg:not(:root){overflow:hidden}button,input,optgroup,select,textarea{font-family:sans-serif;font-size:100%;line-height:1.15;margin:0}button,input{overflow:visible}button,select{text-transform:none}[type=reset],[type=submit],button,html [type=button]{-webkit-appearance:button}[type=button]::-moz-focus-inner,[type=reset]::-moz-focus-inner,[type=submit]::-moz-focus-inner,button::-moz-focus-inner{border-style:none;padding:0}[type=button]:-moz-focusring,[type=reset]:-moz-focusring,[type=submit]:-moz-focusring,button:-moz-focusring{outline:1px dotted ButtonText}fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em}legend{box-sizing:border-box;color:inherit;display:table;max-width:100%;padding:0;white-space:normal}progress{display:inline-block;vertical-align:baseline}textarea{overflow:auto}[type=checkbox],[type=radio]{box-sizing:border-box;padding:0}[type=number]::-webkit-inner-spin-button,[type=number]::-webkit-outer-spin-button{height:auto}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}[type=search]::-webkit-search-cancel-button,[type=search]::-webkit-search-decoration{-webkit-appearance:none}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}details,menu{display:block}summary{display:list-item}canvas{display:inline-block}template{display:none}[hidden]{display:none}/*! Simple HttpErrorPages | MIT X11 License | https://github.com/AndiDittrich/HttpErrorPages */body,html{width:100%;height:100%;background-color:#21232a}body{color:#fff;text-align:center;text-shadow:0 2px 4px rgba(0,0,0,.5);padding:0;min-height:100%;-webkit-box-shadow:inset 0 0 100px rgba(0,0,0,.8);box-shadow:inset 0 0 100px rgba(0,0,0,.8);display:table;font-family:"Open Sans",Arial,sans-serif}h1{font-family:inherit;font-weight:500;line-height:1.1;color:inherit;font-size:36px}h1 small{font-size:68%;font-weight:400;line-height:1;color:#777}a{text-decoration:none;color:#fff;font-size:inherit;border-bottom:dotted 1px #707070}.lead{color:silver;font-size:21px;line-height:1.4}.cover{display:table-cell;vertical-align:middle;padding:0 20px}footer{position:fixed;width:100%;height:40px;left:0;bottom:0;color:#a0a0a0;font-size:14px}</style>
</head>
<body>
    <div class="cover">
      <h1>
          {% block title %}Bad Request{% endblock title %}
          <small>{% block subtitle %}Error 400{% endblock subtitle %}</small>
      </h1>
      <p class="lead">
          {% block explain %}The server cannot process the request due to something that is perceived to be a client error.{% endblock explain %}
      </p>

      <br />
      <a href="/">Click here to reload</a>

    </div>
    <footer>
      {% comment %}
      <p>Technical Contact: <a href="mailto:x@example.com">x@example.com</a></p>
      {% endcomment %}
      <p>{{ SITE_COMPANY }} - {{ SITE_TITLE }} - {{ PROJECT_BUILD }}</p>
    </footer>
</body>
</html>

file main/templates/errors/400.html:

{% extends 'errors/error_base.html' %}

{% block pagetitle %}
Bad Request
{% endblock pagetitle %}

{% block title %}
Bad Request
{% endblock title %}

{% block subtitle %}
Error 400
{% endblock subtitle %}

{% block explain %}
The server cannot process the request due to something that is perceived to be a client error.
{% endblock explain %}

file main/templates/errors/403.html:

{% extends 'errors/error_base.html' %}

{% block pagetitle %}
Access Denied
{% endblock pagetitle %}

{% block title %}
Access Denied
{% endblock title %}

{% block subtitle %}
Error 403
{% endblock subtitle %}

{% block explain %}
The requested resource requires an authentication.
{% endblock explain %}

file main/templates/errors/404.html:

{% extends 'errors/error_base.html' %}

{% block pagetitle %}
Resource Not Found
{% endblock pagetitle %}

{% block title %}
Resource Not Found
{% endblock title %}

{% block subtitle %}
Error 404
{% endblock subtitle %}

{% block explain %}
The requested resource could not be found but may be available again in the future.
{% endblock explain %}

file main/templates/errors/500.html:

{% extends 'errors/error_base.html' %}

{% block pagetitle %}
Webservice currently unavailable
{% endblock pagetitle %}

{% block title %}
Webservice currently unavailable
{% endblock title %}

{% block subtitle %}
Error 500
{% endblock subtitle %}

{% block explain %}
An unexpected condition was encountered.<br />
Our service team has been dispatched to bring it back online.
{% endblock explain %}

Finally, tell Django to use our custom views:

file main/urls.py:

...
handler400 = 'main.views.bad_request'
handler403 = 'main.views.permission_denied'
handler404 = 'main.views.not_found'
handler500 = 'main.views.server_error'
...

Test with DEBUG=False in settings

References:

  Many-to-many example using "through" to augment M2M relationships

On one side of the relation, we do the following:

file models.py

class Device(models.Model):

    ...
    users = models.ManyToManyField(
        User,
        through='DeviceUsers',
        related_name='devices'
    )


class DeviceUsers(models.Model):

    class Meta:
        verbose_name = _("Device/User")
        verbose_name_plural = _("Device/Users")

    device = models.ForeignKey(Device, verbose_name=_('Device'), on_delete=models.CASCADE)
    user = models.ForeignKey(User, verbose_name=_('User'), on_delete=models.CASCADE)
    enabled = models.BooleanField(_('Enabled'), null=False, default=True, blank=True)
    expiration_data = models.DateField(_('Expiration date'), null=True, blank=True)

    def __str__(self):
        return ''

while nothing is required in the related Model.

In the admin, we can:

  • add inlines to edit the relation (in both ModelAdmins),
  • (optionally) list the related models in the listing
  • note that we also add a search_fields attribute to have autocomplete_fields working (Django 2.x)

file admin.py

class UserForDeviceTabularInline(admin.TabularInline):
    model = Device.users.through
    extra = 0
    autocomplete_fields = ['user', ]


@admin.register(Device)
class DeviceAdmin(BaseModelAdmin):

    list_display = [..., 'list_users', ]
    search_fields = ['=id', 'description', 'code', ]
    inlines = [UserForDeviceTabularInline, ]

    def list_users(self, obj):
        return mark_safe(', '.join([
            '<a href="%s">%s</a>' % (user.get_admin_url(), str(user))
            for user in obj.users.all()  # .order_by('panelregisters__position')
        ]))
    list_users.short_description = _('users')


class DeviceTabularInline(admin.TabularInline):
    from backend.models import Device
    model = Device.users.through
    #exclude = ['position', ]
    extra = 0
    autocomplete_fields = ['device', ]


@admin.register(User)
class UserAdmin(AuthUserAdmin, HijackUserAdminMixin):

    list_display = [..., 'list_devices', ]
    inlines = [DeviceTabularInline, ]

    def list_devices(self, obj):
        return mark_safe(', '.join([
            '<a href="%s">%s</a>' % (device.get_admin_url(), str(device))
            for device in obj.devices.all()  # .order_by('panelregisters__position')
        ]))
    list_devices.short_description = _('devices')

TODO: add select_related() as required

  New Django project check-list

A check list of what I normally do when creating a brand new Django project.

1   Initial project skeleton

  • create a local virtual environment for development, and install lastest version of django in it

  • create a folder container for the project as follows:

    |- customerName
         |- myNewProjectName
               |- logs
               |- dumps
               |- public
               |     |- media
               |     |- static
               |- myNewProjectName  <-- project sources go here
    
  • create the project skeleton with python django-admin startproject and name it main

  • replace settings.py file with a "settings" module, and adjust it as needed; the final project layout is:

    |- customerName
         |- myNewProjectName
               |- logs
               |- dumps
               |- public
               |     |- media
               |     |- static
               |- myNewProjectName
                     |- manage.py
                     |- main
                           |- settings
                               |- __init__.py
                               |- debug.py
                               |- local.py
                               |- settings.py
                               |- test_settings.py
                               |- test_settings_no_migrations.py
    

    where:

    main/settings/__init__.py:
    
        from main.settings.local import *
    
    main/settings/local.py:
    
        from main.settings.settings import *
        ...
    
    main/settings/debug.py:
    
        from main.settings.local import *
        DEBUG = True
    
  • main/settings/local.py will contain db settings and the secret_key, and will be excluded from GIT repo

  • add a requirements file to keep track of installed modules and versions

3   Custom User model

  • create a "users" app
  • use it to override User model (derived from django.contrib.auth.models.AbstractUser)
  • add and configure Hijack app

4   Versioning and other initializations

  • create a git repo for the project
  • create an "admin_ex" app for a customized (possibly empty) AdminSiteEx(admin.AdminSite)
  • add a main.apps.MainConfig to install admin_ex.admin_site.AdminSiteEx as project admin
  • define admin.site.site_header and admin.site.site_title variables
  • configure bumpversion to manage versioning

5   Deployment procedure

  • add "django-channels" to have daphne installed
  • setup a provisioning procedure for initial host setup
  • setup a deployment procedure for project installation and update
  • run it over a staging server asap

6   Email setting

8   Add helper apps

  • django-constance
  • django-smuggler
  • django-debug-toolbar
  • django-compressor

9   Review settings for security

  • python manage.py check --deploy

10   Sample README.rst file

Project "XXXXX"
===============

Manual provisioning on local development machine
------------------------------------------------

Create environment for new project::

    $ cd /Projects/
    $ mkdir XXXXX
    $ mkdir XXXXX/public
    $ mkdir XXXXX/public/static
    $ mkdir XXXXX/public/media

Create and activate a virtualenv for the project::

    $ mkvirtualenv XXXXX
    $ workon XXXXX

Clone source repository::

    $ cd /Projects/XXXXX
    $ git clone git@gitlab.brainstorm.it:YYYYY/XXXXX.git

The final project layout is::

    .
    ├── XXXXX
    │   ├── README.rst
    │   ├── ...
    │   ├── manage.py
    │   ├── main          <---- this is the Django project
    │   ├── requirements
    │   └── ...
    └── public
        ├── media
        └── static

Update the virtualenv (that is: Install required python packages)::

    $ cd /Project/XXXXX/XXXXX
    $ pip install -r requirements/development.txt

Create a local settings file;
for example, copy and adapt "local_example.py"::

    $ cd /Project/XXXXX/XXXXX
    $ cp main/settings/local_example.py main/settings/local.py

Create database (use database name and password specified in local.py)::

    $ psql
    $ create user XXXXX with encrypted password '<PASSWORD>';
    $ create database XXXXX owner XXXXX;

Populate database struct::

    $ python manage.py migrate

Update front-end assets (if any)::

    $ npm install

Create a supersuser account::

    $ python manage.py createsuperuser

Run the development web server::

    $ python manage.py runserver 0.0.0.0:8000


Development workflow
--------------------

::

    $ cd /Project/XXXXX/XXXXX
    $ workon XXXXX
    $ git pull
    $ git submodule update
    $ pip install -r requirements/development.txt
    $ python manage.py migrate
    $ npm install
    $ python manage.py runserver

11   Monitoring

  Json prettified fields in Django admin

To render a JSONfield as colored and formatted text in a Django change-view, add a readonly field as follows:

@admin.register(DeviceTestTask)
class DeviceTestTaskAdmin(TaskAdmin):

    exclude = ('program_steps', )

    readonly_fields = [..., 'program_steps_prettified', ]

    def program_steps_prettified(self, obj):
        return json_prettify(obj.program_steps)

where json_prettify() is a helper function responsible to format JSON data using Pygments:

file admin_ex.json_prettify.py

import json
from django.utils.safestring import mark_safe
from pygments import highlight
from pygments.lexers import JsonLexer
from pygments.formatters import HtmlFormatter


def json_prettify_styles():
    """
    Used to generate Pygment styles (to be included in a .CSS file) as follows:
        print(json_prettify_styles())
    """
    formatter = HtmlFormatter(style='colorful')
    return formatter.get_style_defs()


def json_prettify(json_data):
    """
    Adapted from:
    https://www.pydanny.com/pretty-formatting-json-django-admin.html
    """

    # Get the Pygments formatter
    formatter = HtmlFormatter(style='colorful')

    # Highlight the data
    json_text = highlight(
        json.dumps(json_data, sort_keys=True, indent=2),
        JsonLexer(),
        formatter
    )

    # # remove leading and trailing brances
    # json_text = json_text \
    #     .replace('<span class="p">{</span>\n', '') \
    #     .replace('<span class="p">}</span>\n', '')

    # Get the stylesheet
    #style = "<style>" + formatter.get_style_defs() + "</style>"
    style = ''

    # Safe the output
    return mark_safe(style + json_text)

Suitable styles can be produced with:

print(json_prettify_styles())

and included in a CSS file; for example:

/*
 *  Pygment styles
 *
 *  Generated by 'gui.utils.json_prettify_styles()'
 */

.highlight .hll { background-color: #ffffcc }
.highlight .c { color: #888888 } /* Comment */
.highlight .err { color: #FF0000; background-color: #FFAAAA } /* Error */
.highlight .k { color: #008800; font-weight: bold } /* Keyword */

References: