Brainstorm's snippets (20/239)


  Automatically zip a FileField upon saving in Django
filefield, zip
  Cherry-picking a GIT commit from another git repository
git
  Sortable column in ModelAdmin.list_display on a calculated property
admin
  CSS: How To Center an Object Exactly In The Center
css
  Git partial cherry-picking
git
  How to float a <label> to the left of an <input>?
css
  Django delete FileField
django
  PUB/SUB in javascript
javascript, pubsub
  Prometheus Guide
monitoring
  pyclean alias
python
  How to Install Node.js and NPM on a Mac
mac, npm, nodejs
  Monit guide (updated)
monitoring
  Flat list out of list of lists in Python
python
  Raspberry PI setup guide
setup, raspberrypi
  Run ssh-agent and ssh-add on login via SSH
ssh
  Management command to dump local db and media
django, management_command
  Send email from command line
mail
  Capture Video from Camera
opencv
  Downgrade pg_dump 10 script to 9.x
postgresql
  Git: apply patches
git

  Automatically zip a FileField upon saving in Django

"""
(c) 2019 Mario Orlandi, Brainstorm S.n.c.

Automatically zip a FileField upon saving in Django

__author__    = "Mario Orlandi"
__copyright__ = "Copyright (c) 2019, Brainstorm S.n.c."
__license__   = "GPL"
"""

import os
import zipfile
from django.core.files.storage import FileSystemStorage


class ZipFileStorage(FileSystemStorage):
    """
    A specialized FileSystemStorage which, upon saving the file,
    transparently compresses it into a zip archive (if not already zipped)

    Sample usage:

        source_data = models.FileField(..., storage=ZipFileStorage())
    """

    def save(self, name, content, max_length=None):

        # Let the base storage class save the file as usual
        name = super().save(name, content, max_length)

        # Check if already zipped
        if not name.lower().endswith('.zip'):

            # If not, create a new zip archive and write the file into it
            path = self.path(name)
            zf = zipfile.ZipFile(path + '.zip', mode='w', compression=zipfile.ZIP_DEFLATED)
            try:
                zf.write(path, os.path.basename(path))
            finally:
                zf.close()

            # Cleanup: the orginal unzipped file can now be removed
            os.remove(path)

            # The new return value will be the name of the zip archive just created
            name += '.zip'

        return name

  Cherry-picking a GIT commit from another git repository

This script applies locally the diff of a commit from another repo; "apply" doesn't commits the patch, so you have to supply a commit message yourself

git --git-dir=../[some_other_repo]/.git show [commit_SHA] | git apply

The following script also commits the patch:

git --git-dir=../[some_other_repo]/.git format-patch -k -1 --stdout [commit_SHA] | git am -3 -k --ignore-whitespace

Explanation:

  • The git format-patch command creates a patch
  • from [some_other_repo]'s commit
  • specified by its [commit_SHA]
  • "-1" for one single commit alone
  • "-k" keeps the subject (don't add [PATCH])
  • This patch is piped to git am, which applies the patch locally
  • "-3" means trying the three-way merge if the patch fails to apply cleanly
  • "-k" keeps the subject (don't strip [PATCH])

Explanation:

  Sortable column in ModelAdmin.list_display on a calculated property

class DeAdmin(admin.ModelAdmin):
    list_display = ("[...]", "s_d", "gd", "na", "de", "fr" )

    def get_queryset(self, request):
        queryset = super().get_queryset(request)
        queryset = queryset.annotate(
            _s_d=Case(
                When(fr=True, then='s_d'),
                When(fr=False, then=F('gd') + F('na')),
                default=Value(0),
                output_field=IntegerField(),
            )
        )
        return queryset

    def s_d(self, obj):
        return obj._s_d
    s_d.admin_order_field = '_s_d'

References:

https://stackoverflow.com/questions/58366953/how-to-implement-sorting-in-django-admin-for-calculated-model-properties-without

  CSS: How To Center an Object Exactly In The Center

.centered {
    position: absolute;
    top: 50%;
    left: 50%;
    transform: translate(-50%, -50%);
}

To center text in a div, se line-height equal to height:

.center {
    height: 400px;
    line-height: 400px; /* same as height! */
}

References:

  Git partial cherry-picking

git cherry-pick -n <commit> # get your patch, but don't commit (-n = --no-commit)
git reset                   # unstage the changes from the cherry-picked commit
git add -p                  # make all your choices (add the changes you do want)
git commit                  # make the commit!

It looks like there's a new way to do this with interactively checking out:

git checkout -p bc66559

References:

https://stackoverflow.com/questions/1526044/partly-cherry-picking-a-commit-with-git#1526093

https://stackoverflow.com/questions/1526044/partly-cherry-picking-a-commit-with-git#17376789

  How to float a <label> to the left of an <input>?

<div class="field">
    <label>Test</label>
    <input type="text">
</div>
.field {
    display: table;
    width: 100%;
}

.field > label,
.field > input {
    display: table-cell;
}

.field > input {
    width: 80%;
}

or

.field {
    display: table;
    width: 100%;
}

.field > label,
.field > input {
    display: table-cell;
}

.field > label {
    text-align: right;
    font-weight: bold;
    padding-right: 8px;
}

.field > label:after {
    content: ':';
}

.field > input {
    width: 80%;
}

Credits:

https://stackoverflow.com/questions/32217349/how-to-float-a-label-to-the-left-of-an-input#32217404

  Django delete FileField

import os
from django.db import models
from django.dispatch import receiver


def remove_file_and_cleanup(filepath):
    """
    Helper to remove a media file;
    also removes the containing folder, if left empty
    """
    folder = os.path.dirname(filepath)
    # remove file
    if os.path.isfile(filepath):
        os.remove(filepath)
    # finally, remove folder if empty
    if os.path.isdir(folder) and len(os.listdir(folder)) <= 0:
        os.rmdir(folder)


@receiver(models.signals.post_delete, sender=MyModel)
def auto_delete_file_on_delete(sender, instance, **kwargs):
    """ Deletes file from filesystem when corresponding `MediaFile` object is deleted.
        Adapted from: http://stackoverflow.com/questions/16041232/django-delete-filefield
    """

    # Collect names of FileFields
    fieldnames = [f.name for f in instance._meta.get_fields() if isinstance(f, models.FileField)]
    for fieldname in fieldnames:
        field = getattr(instance, fieldname)
        if bool(field):
            remove_file_and_cleanup(field.path)


@receiver(models.signals.pre_save, sender=MyModel)
def auto_delete_file_on_change(sender, instance, **kwargs):
    """ Deletes file from filesystem when corresponding object is changed or removed.
        Adapted from: http://stackoverflow.com/questions/16041232/django-delete-filefield
    """

    if not instance.pk or sender.objects.filter(pk=instance.pk).count() <= 0:
        return False

    fieldnames = [f.name for f in instance._meta.get_fields() if isinstance(f, models.FileField)]
    for fieldname in fieldnames:
        try:
            old_field = getattr(sender.objects.get(pk=instance.pk), fieldname)
            old_filepath = old_field.path

            new_field = getattr(instance, fieldname)
            new_filepath = new_field.path if new_field else None

            # if ready to save a new file, delete the old one
            if old_filepath != new_filepath:
                remove_file_and_cleanup(old_filepath)
        except:
            pass

  PUB/SUB in javascript

file publisher.js:

function initialize_datatable(element, url) {

    $.ajax({
        type: 'GET',
        url: url + '?action=initialize',
        dataType: 'json'
    }).done(function(data, textStatus, jqXHR) {

        ...
        var table = ...

        // Notify subscribers
        $('.subscribe-datatable-initialized').trigger(
            'datatableInitialized', [table]
        );

    });

}

file sample_subscriber.html:

<div class="panel panel-danger subscribe-datatable-initialized" id="stale-acquisition-panel" style="display: none;">

    <p>This panel is to be shown only after datatable has been initialized</p>

</div>

<script language="javascript">

    $(document).ready(function() {
        $('#stale-acquisition-panel').on('datatableInitialized', function(e, eventInfo) {
            $(e.target).show();
        });
    });

</script>

  Prometheus Guide

  pyclean alias

alias pyclean="find . \( -name \*.pyc -o -name \*.pyo -o -name __pycache__ \) -prune -exec rm -rf {} +"

  How to Install Node.js and NPM on a Mac

Install node with homebrew:

brew install node

To see if Node is installed:

node -v
    v10.12.0

To see if NPM is installed:

npm -v
    6.4.1

How to Update Node and NPM

brew update
brew upgrade node

How to Uninstall Node and NPM

brew uninstall node

Hot to use

Then in your project you can finally populate node_modules from package.json as follows:

npm install

How to Install Sass compiler on a Mac

Check if ruby is available:

ruby -v

If not, install it:

brew install ruby

Install Sass:

sudo gem install sass

In case of SSL error "SSL Error When installing rubygems, Unable to pull data from 'https://rubygems.org/", try this:

sudo gem sources -r https://rubygems.org
sudo gem sources -a http://rubygems.org

the again:

sudo gem install sass

Eventuale aggiornamento dei requirements:

pip install -r requirements/development.txt

  Monit guide (updated)

Install Monit System Monitor On Ubuntu 18.04

sudo apt update
sudo apt install monit

After installing Monit, the commands below can be used to stop, start and enable Monit service:

sudo systemctl stop monit.service
sudo systemctl start monit.service
sudo systemctl enable monit.service

Configure Monit service

By default all files located on /etc/monit/conf.d/ and /etc/monit/conf-enabled/ are read by monit when the service is started.

Use the /etc/monit/conf.d/ directory to put all your monitoring configuration files in it.

Example: file /etc/monit/conf.d/disk:

check device root with path /
    if SPACE usage > 80% then alert

check device backup_disk with path /mnt/backup
    if SPACE usage > 80% then alert

then:

sudo monit check -t
sudo monit reload
sudo monit start all

Check Monit status

sudo monit status

Access Monit Web Portal

Add this to file /etc/monit/monitrc:

set httpd port 2812
    allow admin:password

then access the web page from anywhere with:

http://host:2812

or:

http://host:2812/_status?format=xml

  Flat list out of list of lists in Python

Given superlist as a list of lists,

flat_list = [item for sublist in superlist for item in sublist]

which means:

flat_list = []
for sublist in superlist:
    for item in sublist:
        flat_list.append(item)

References:

  Raspberry PI setup guide

1   Initial setup

Eventually expand file system if part of the SD is unused:

raspi-config --expand-rootfs

2   Enable SSH connections

Connect with a keyboard, then issue these commands from terminal:

systemctl enable ssh.service
systemctl start ssh.service

3   Set locale

sudo -s
export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
locale-gen en_US.UTF-8
dpkg-reconfigure locales

5   Expand swap partition ...

100 Mb (the default) is too small, and if you are doing memory intensive stuff (e.g., web surfing), you can easily max it out.

The recommended swap size is 2*physical RAM - in the case of the Pi (modern/current versions of), this is 2G.

You can increase the swap size by changing the CONF_SWAPSIZE or the CONF_SWAPFACTOR parameter in /etc/dphys-swapfile followed by a reboot:

cat /etc/dphys-swapfile

[...]
CONF_SWAPFACTOR=2
[...]

Test after reboot:

# free -h
              total        used        free      shared  buff/cache   available
Mem:           927M        259M        425M         30M        242M        585M
Swap:          1.8G          0B        1.8G

6   ... or remove it !

On the other hand, you might want to turn off swap entirely in order to reduce the amount of write operations on the SD card – because SD cards have their life limited to the amount of write operations:

sudo systemctl disable dphys-swapfile

Test after reboot:

# free -h
              total        used        free      shared  buff/cache   available
Mem:           927M        257M        437M         24M        232M        593M
Swap:            0B          0B          0B

Another option is to move swap to an external device (for example a USB key).

References:

7   Setup SMTP

TODO; vedere:

Installazione e Configurazione di Postfix su Raspberry usando come Smarthost GMAIL:

https://www.raffaelechiatto.com/installazione-configurazione-postfix-raspberry-usando-smarthost-gmail/

NULLMAILER – IL POSTINO MINIMALISTA:

https://hamradio.fe.linux.it/nullmailer-il-postino-minimalista/

8   Start chromium in Kiosk mode on raspbian jessie

file ~/Desktop/runChromium.desktop:

[Desktop Entry]
Type=Application
Exec=/usr/bin/chromium-browser --noerrdialogs --disable-session-crashed-bubble --disable-infobars --kiosk http://127.0.0.1
Hidden=false
X-GNOME-Autostart-enabled=true
Name[en_US]=RunChromium
Name=RunChromium
Comment=Start Chromium in kiosk mode; copy int ~/.config/autostart to have it run automatically

References:

9   Pi display

How to hide the cursor in the kiosk mode automatically:

sudo apt-get install unclutter

then add this to file /etc/xdg/lxsession/LXDE/autostart:

@unclutter -idle 0.1 -root

10   Display rotation

file "/boot/config.txt":

# LCD Rotation
lcd_rotate=2

# Display Rotate (HDMI)
#display_rotate=2

11   Disable screen sleep

file "/etc/lightdm/lightdm.conf":

[Seat:*]
...
# don't sleep the screen
xserver-command=X -s 0 dpms

or:

sudo apt-get install xscreensaver

then configure the screensaver application under the Preferences option on the main desktop menu.

12   How to up a Static IP on Your Ethernet or Wireless Network Connection

file /etc/dhcpcd.conf:

# setup ethernet static ip
interface eth0
inform 192.168.1.18
static routers=255.255.255.0

# setup wireless static ip
interface wlan0
inform 192.168.1.19
static routers=255.255.255.0

or:

# setup ethernet static ip
interface eth0
static ip_address=172.26.11.85/24
static routers=172.26.11.100
static domain_name_servers=8.8.8.8 8.8.4.4

then:

#sudo /etc/init.d/networking restart
sudo reboot

https://projects.raspberrypi.org/en/projects/getting-started-with-picamera

13   Setting up a Raspberry Pi as an access point in a standalone network (NAT)

Brief summary:

Install required software:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install dnsmasq hostapd

then stop the services and reboot:

sudo systemctl stop dnsmasq
sudo systemctl stop hostapd
sudo reboot

Configuring a static IP for wlan0:

add this to /etc/dhcpcd.conf:

interface wlan0
    static ip_address=192.168.4.1/24
    nohook wpa_supplicant

then:

sudo service dhcpcd restart

Configuring the DHCP server (dnsmasq):

sudo mv /etc/dnsmasq.conf /etc/dnsmasq.conf.orig
sudo vim /etc/dnsmasq.conf

and add:

interface=wlan0      # Use the require wireless interface - usually wlan0
    dhcp-range=192.168.4.2,192.168.4.20,255.255.255.0,24h

Configuring the access point host software (hostapd):

add this to /etc/hostapd/hostapd.conf:

interface=wlan0
driver=nl80211
ssid=NETWORK_NAME
hw_mode=g
channel=7
wmm_enabled=0
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_passphrase=NETWORK_PASSPHRASE
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

where:

  • NETWORK_PASSPHRASE: between 8 and 64 characters

  • hw_mode:

    • a = IEEE 802.11a (5 GHz)
    • b = IEEE 802.11b (2.4 GHz)
    • g = IEEE 802.11g (2.4 GHz)
    • ad = IEEE 802.11ad (60 GHz)

Add this to /etc/default/hostapd:

DAEMON_CONF="/etc/hostapd/hostapd.conf"

Start the access point host software:

sudo systemctl start hostapd
sudo systemctl start dnsmasq

if necessary:

sudo systemctl unmask hostapd
sudo systemctl enable hostapd
sudo systemctl start hostapd

Add routing and masquerade:

Uncomment this line in /etc/sysctl.conf:

net.ipv4.ip_forward=1

Add a masquerade for outbound traffic on eth0:

sudo iptables -t nat -A  POSTROUTING -o eth0 -j MASQUERADE

Save the iptables rule:

sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"

Edit /etc/rc.local and add this just above "exit 0" to install these rules on boot:

iptables-restore < /etc/iptables.ipv4.nat

Reboot and done !

References:

Setting up a Raspberry Pi as an access point in a standalone network (NAT)

15   Installing Python 3.7.x on Raspbian

sudo su
cd
mkdir downloads
cd downloads

apt-get update -y
apt-get install build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev libffi-dev -y
wget https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tar.xz
tar xf Python-3.7.2.tar.xz
cd Python-3.7.2
./configure
make -j 4
make altinstall
cd ..
rm -r Python-3.7.2
rm Python-3.7.2.tar.xz
apt-get --purge remove build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev libffi-dev -y
apt-get autoremove -y
apt-get clean

References:

16   todo

All my Pi's (including Pi Zero's) have these two lines added to /etc/fstab:

tmpfs /tmp tmpfs defaults,noatime,nosuid 0 0
tmpfs /var/log tmpfs defaults,noatime,nosuid,size=16m 0 0

The default maximum size is half the memory, but of course tmpfs only takes as much memory as the files need.

While you are in /etc/fstab, if you want to reduce writes to the SD then the other simple change is the flush rate.

For the ext4 / mount make sure the options include "commit=600" That is, for example:

PARTUUID=e96d960e-02 / ext4 defaults,noatime,commit=600,errors=remount-ro 0 1

Obviously do not do this if your site is prone to power cuts or other unexpected outages. (dirty pages in the disk cache are written out every ten minutes instead of every five seconds which is the default). This improves performance as well.

  Run ssh-agent and ssh-add on login via SSH

You can try adding this:

eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa

This way the ssh-agent does not start a new shell, it just launches itself in the background and spits out the shell commands to set the appropriate environment variables.

As said in the comment, maybe you do not want to run the agent at all on the remote host, but rather on the box you are working from, and use:

ssh -A remote-host

to forward the services of your local ssh agent to the remote-host.

For security reasons you should only use agent forwarding with hosts run by trustworthy people, but it is better than running a complete agent remotely any time.

  Management command to dump local db and media

file management/commands/dump_local_data.py

Usage:

usage: manage.py dump_local_data [-h] [--version] [-v {0,1,2,3}]
                                 [--settings SETTINGS]
                                 [--pythonpath PYTHONPATH] [--traceback]
                                 [--no-color] [--dry-run] [--max-age MAX_AGE]
                                 [--no-gzip] [--legacy]
                                 target

Dump local db and media for backup purposes (and optionally remove old backup
files)

positional arguments:
  target                choices: db, media, all

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -v {0,1,2,3}, --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=verbose output, 3=very verbose output
  --settings SETTINGS   The Python path to a settings module, e.g.
                        "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be
                        used.
  --pythonpath PYTHONPATH
                        A directory to add to the Python path, e.g.
                        "/home/djangoprojects/myproject".
  --traceback           Raise on CommandError exceptions
  --no-color            Don't colorize the command output.
  --dry-run, -d         Dry run (simulation)
  --max-age MAX_AGE, -m MAX_AGE
                        If > 0, remove backup files old "MAX_AGE days" or more
  --no-gzip             Do not compress result
  --legacy              use legacy Postgresql command syntax

Code snippet has been moved to:

https://github.com/morlandi/dump_local_data

  Send email from command line

Using mail

mail someone@gmail.com
    Subject: Hello World!
    This is an email to myself.

    Hope all is well.
    ...

and terminate with ^D.

or

echo "this is a test mail" | mail -s'send mail test' testuser@somewhere.com

Using mailx

mailx -s "subject" -r sender@somewhere.com receiver1@somewhere.com,receiver2@somewhere.com < textfile

Using mailx with attachment

mailx -A attchment_file -s "subject" -r sender@somewhere.com receiver1@somewhere.com,receiver2@somewhere.com < textfile

Test what's happening

tail -f /var/log/mail.log

  Capture Video from Camera

import cv2
import time
import shutil


cap = cv2.VideoCapture(0)
snapshot = "/tmp/snapshot.jpg"
snapshot_tmp = "/tmp/snapshot.tmp.jpg"

print('Press "q" to quit ...')

while(True):

    # Capture frame-by-frame
    ret, frame = cap.read()

    # Our operations on the frame come here
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Display the resulting frame
    #cv2.imshow('frame',gray)
    cv2.imshow('frame', frame)

    # Update snapshot
    cv2.imwrite(snapshot_tmp, frame)
    shutil.move(snapshot_tmp, snapshot)
    print('%s "%s" updated' % (
        time.asctime(),
        snapshot
    ))

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

    time.sleep(0.1)

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

References:

  Downgrade pg_dump 10 script to 9.x

#!/usr/bin/env python3
import sys

#
#  Downgrades pg_dump 10 script to 9.x
#  removing 'AS integer' from 'CREATE SEQUENCE' statement
#
#  Usage:
#       $ python3 pgdump_10_to_9.py < test10.sql > test9.sql
#  or:
#       $ cat test10.sql | ./pgdump_10_to_9.py > test9.sql
#
#  To obtain a compressed 9.x sql script from a compressed 10 sql script:
#
#       $ gunzip -c test10.sql.gz | ./pgdump_10_to_9.py | gzip > test9.sql.gz
#

inside_create_sequence = False
for row in sys.stdin.readlines():

    if inside_create_sequence and row.strip().lower() == 'as integer':
        pass
    else:
        print(row, end='', flush=True)

    inside_create_sequence = row.strip().startswith('CREATE SEQUENCE ')

  Git: apply patches

Purpose

Retrieve changes from a remote "working clone" of a project to your local development machine

On remote "working clone" ...

where you've done some changes to be recovered, procede as follows:

  1. Move changes to be recovered to a work branch, and commit them:
git stash
git checkout -b dirty
git stash apply

git add .
git commit -m "my new nice changes"
  1. Check the id of the new commit:
$ git log --oneline
eddc3c6 (HEAD -> dirty) my new nice changes
...
  1. Build a single patch file referring to the specific commit:
git format-patch -o patches/ -1 eddc3c6

or

git format-patch -o patches/ -1 HEAD
  1. Sent it to the developer:

http://brainstorm.it/snippets/send-email-command-line/

or download from development machine via rsync:

rsync -a --rsync-path="sudo rsync" master@host:/home/project/project/patches .
  1. Later (after the changes have been integrated in the main project), you might want to cleanup your working clone:
git checkout master
git branch -D dirty
rm -fr ./patches

On your development machine ...

Apply the changes:

$ git am patches/0001-retrieve_avatars-fixes.patch

Note that, as a result of applying the patch, the changes have already been committed to your working branch:

$ git log --oneline
de4ccec (HEAD -> master) my new nice changes