2016-07-12

randword: Generating memorable random passwords

This started when I decided to learn python by rewriting one of my old perl scripts in python.  randstring is a script to generate a random string of characters.  I use it sometimes to generate passwords, but password strings or random characters usually can't be remembered, at least not easily.  Passwords like that can be useful at times.  You need to store them in an encrypted password safe.

I have another script that generates more memorable passwords.  Some people I know, have found it useful.  There are always some passwords you need to be memorable.  For instance your login password and the password to your password safe.  randword generates a bunch of words from a dictionary.  XKCD style passwords, if you like.  In the process of examining it, I rewrote it in both perl and python, fixed some bugs and added some features.

In general a bunch of words can be much easier to remember and can be just as difficult or far more difficult  to crack.  I like to generate a bunch and choose a few at random.  4 or 5 or more words is OK.  Hint: misspellings are good but not if you can't remember what you did.  Passwords on websites are a bit mad at the moment with complicated rules, like: "there's an illegal character" or "you must have an upper-case letter and a number", or "that password is too short", or "too long" etc. 

New features of randword:
  • There's a couple of new options about output format, like camel case.   
  • randword can use any dictionary or word frequency lists as long as they have a fairly simple format - ie at least a word and an optional number at the start of each line. 
  • randword can also take a bunch of text and create dictionaries of words that it can use to generate random passwords.  
For word lists, I have used various texts, for instance Jane Austen's complete works, Shakespeare, Mark Twain, Chaucer.  There are many works that can be easily got from Project Gutenburg among other places on the net.  Also word lists and text that can be found at COCA or Lancaster University  etc.Since I only want ascii because I can't type non-ascii characters easily, I used unidecode (python or original perl version) to turn them into ascii.  Python unidecode comes with a command line script.  I wrote a very simple perl script to detect non-ascii characters, (not included) although working out what encoding a page is in is a kind of major headache and you need to know the encoding before unidecode will work, grrrr. 

The links below include word lists from Chaucer, Shakespeare, Mark Twain, and the linux word dictionary.

This is my original blog post on the scripts with all the links to the scripts and associated stuff.

Links:
randstring.pl randstring.py
randword.pl randword.py
some word lists
tarred and zipped archive of scripts and wordlists


2016-01-11

In memory of my mother, Marjorie Pizer Holburn, 3rd April 1920 - 4th January 2016

Marjorie Pizer died on the 4th of January 2016. She is sadly missed.

It's impossible in a short time and with my limited time and writing skills to summarise her.  Here is my brief message.  In time I will perhaps add links to other people writing about her.  If you knew her please feel free to add a comment.

My mother was a poet, artist, psychotherapist and a deep thinker. She had many unconventional beliefs, which she held passionately. Politically, she almost always sided with the underdog; and she was troubled by the inequities in our world. She was inspired by the beauty of nature, and loved music, poetry and art of all kinds. She read 2 or 3 books a day, every day, on almost every subject imaginable, until near the end of her life. My mother was a member of all the local libraries; municipal, city and state. Brought up in Melbourne, she lived much of her life in Sydney where she went to the beach and swam nearly every day for forty years of her life.

Marje was one of my best friends for my whole life. I was always much more a scientist and she a poet and writer, our discussions were always wide ranging and broadened both our view points. She was and is a reference point for me for ideas and values. Although in later life I agreed with her ideas less, we always had lively and interesting discussions.

When I think of my mother I picture her surrounded by books and works of art and odd found objects from the beach. Pieces of driftwood, and a vase of fresh cut flowers on the table. We would eat a simple meal together, have a cup of tea and discuss the state of the world. This is how I will remember her.


Marjorie's books can be bought at lulu and apple and amazon.  They are published by Pinchgut Press (me). 

Daniela Torsh's obituary from the Sydney Morning Herald: Poetry Marjorie Pizer's vehicle of optimism and understanding. And her speech.


A memory

When my son comes home late,
He sits on my bed
And tells me about his day.
Someday he will remember this
When I will be no more,
When I have had my say
And gone before.
Then I will not exist
As I am now.
This me will be a memory
Of his when I,
Who now am here alone,
Have gone into oblivion.

2015-11-15

Conception, the twin problems and souls.

People used to believe that women were like a field and men plowed them and planted a seed and those seeds grew into babies.  There are echoes of this belief in the current right to life movement when they say "life begins at the moment of conception."  Apart from the obvious: "so sperms and eggs aren't alive?"  What does this really mean?  That at the "moment" of conception a fertilised egg suddenly gets a soul?  How long does conception actually take?  What part of that process is the "moment"?  Of course it begs the question, does a fertilised egg/fetus get a "soul" suddenly or gradually.  Does that question even have any meaning?  Why should having a soul be an on/off binary thing. 

Then there are the twin problems. 

The first is that identical twins both originate from the same fertilised egg.  Does this mean they share the same soul?  That they only have half a soul each?  That their soul divides into two?  Can souls do that?  That one of them gets another soul when the fetus divides into two?  Which one gets the new one and which the old one? 

The second twin problem is that sometimes with fraternal twins, one of the twins is absorbed by the other.  The "surviving" twin can become a chimera with both sets of genes.  Does this mean they have two souls?

 

2015-07-04

Allow only one program at a time to access a resource in linux

I had several programs I needed to run but I wanted to make sure only one program got access to a resource at the same time.  So I wrote a short and simple perl script that would do this.  This script needs a directory in /var/run that it can write to.  Since the script doesn't run as root, I have to create this directory after each reboot.

The script is invoked like this:

1nstance.pl -d /var/run/once -n com1 -- my-program my-args

It will run my-program if there is no other script accessing the named resource com1.  If there is another script currently running and using com1 then it will just quit.

The script can be downloaded from here.

--------------------------1nstance.pl-------------------



#!/usr/bin/perl -w
#
#(c) copyright 2015 Kim Holburn
# Licensed under GPLv3

use strict;
use Getopt::Long;

my $verbose = 0;
my $help = 0;
my $myname = "";
my $mydir = "/var/run/once";
my $PID = 0;
my $time = 0;
my $timeout = 0;

Getopt::Long::Configure('require_order');
GetOptions ('verbose+' => \$verbose,
            'directory=s' => \$mydir,
            'name=s' => \$myname,
            'timeout=i' => \$timeout,
            'help|?' => \$help);

if ($help) {
  print <<EOM;
one instance - make sure a program only runs one instance.
usage $0:
$0 [options] -- <command> [ARGS]
  options:
    -h|-?|--help - print this screen
    -d|--directory <directory> 
       directory to store run files. 
       Default is /var/run/once
    -t|--timeout <n> 
       time in seconds that a program is allowed to run 
       before being considered hung and will be killed
       -t=0 means no timeout. The running process will 
       be left untouched.
    -v|--verbose - print extra messages
    -n|--name
       name of process or resource to be run once
       The default is the name of the program.    

EOM

  exit;
}

if ($timeout < 0) { die "$0 ERROR: specified timeout less than zero ($timeout)"; }
if (10000 < $timeout) { die "$0 ERROR: ($timeout) too large"; }
if ($mydir !~ m{^/}) { die "$0 ERROR: directory ($mydir) is not an absolute path!"; }
if (! -d $mydir) { die "$0 ERROR: directory ($mydir) does not exist!"; }

# first argument is command to run
my $path = shift;
if (!$path) { die "$0 ERROR: No command "; } 
my $program = $path;
# command must be a full absolute path.  
my $dir = ".";
my $args="";
if ($program =~ m#/#) {
  $program =~ s#^.*/##;
  $dir =~ s#/[^/]*$##;
}
if (!$program) { die "$0 ERROR: program name invalid.  Must be a filename"; } 
if (! -e $path) { die "$0 ERROR: program ($path) does not exist"; }
if (-d $path) { die "$0 ERROR: program ($path) is a directory"; }
if (! -x $path) { die "$0 ERROR: program ($path) not executable"; }
if (!$myname) { $myname=$program; }

my $run1 = "$mydir/$myname";

sub deleterun {
  if ( -e $run1 ) {
    unlink $run1;
    if ( -e $run1 ) { die "$0 ERROR: cannot delete file ($run1)"; }
  }
}

if ($verbose) {
  print "debug timeout=($timeout) dir=($mydir) name=($myname) \n";
  print "debug v=($verbose) 1=($run1) prog=($program) args=(";
  print join (")(",@ARGV);
  print ") \n";
}

if (open(my $fh, '<', "$run1")) {
  while (<$fh>) {
    chomp ;
    if (/^\s*$/) { next; }
    if (/^PID (\d+)$/i) { $PID = $1; }
    elsif (/^TIME (\d+)$/i) { $time = $1; }
  }
  close $fh;
  # no PID or time
  # this shouldn't happen and probably means we have the wrong file
  if (!$PID or !$time) { die "$0 ERROR: run file ($run1) has no PID or time"; }
  chomp (my $proc = `ps hp $PID -o %c`);
  # orphaned run file, delete
  if ($proc eq "") { deleterun; }
  else  {
    # another instance running
    my $timediff = time - $time;
    if (0 == $timeout or $timediff <= $timeout) { die "$0 ERROR: another instance of $myname ($proc) running"; }  
    print STDERR "$0 ERROR: another instance of $myname ($proc) has probably hung\n";
    # another process has probably hung
    # grab its children
    my @PIDS = (`ps h --ppid $PID -o %p`, $PID);
    kill ('TERM', @PIDS);
    chomp ($proc = `ps hp $PID -o %c`);
    # couldn't kill it.  Give up.
    if ($proc) { die "$0 ERROR: can't kill instance ($myname) ($proc) PID ($PID)"; }
    # it died OK.  Delete run file if possible.
    deleterun;
  }
}

if (1 < $verbose) { print STDERR "creating run file ($run1) ... \n"; }
open(my $fh, '>', "$run1") or die "$0 ERROR: opening file ($run1) ($!)" ;
print $fh "PID $$\n";
print $fh "time ", time, "\n";
close $fh;

#If something bad happens delete the run file
sub cleanup {
  if (-e $run1) {
    if (1 < $verbose) { print STDERR "Deleting run file...\n"; }
    unlink $run1;
  }
  exit;
};
$SIG{'INT'}=\&cleanup;
$SIG{'TERM'}=\&cleanup;
$SIG{'QUIT'}=\&cleanup;

if ($verbose) { print STDERR "Starting ....\n"; }

system ($path, @ARGV);

cleanup;

Detecting Rogue DHCP servers 2

So my script detecting rogue DHCP servers worked and worked well but having two or three DHCP servers covering the same scope is somewhat problematical.  So we eventually gave in and changed configuration to having a main DHCP server and failover servers.  I had to change the logic slightly to get the script to cover that.  The new version has a mode where it will only alert if it detects a rogue DHCP server it has not been told about, or gets no response from any of the servers in its valid server list.  One or more valid servers and all is right with the world.

As before, this script works in nagios.  I run nagios on ubuntu.  The scripts in the nagios package reside in /usr/lib/nagios/plugins.   Maybe there's a good place to put your own scripts but I put mine in there too (/usr/lib/nagios/plugins/check_rogue_dhcp.pl).

This script uses the nagios builtin DHCP checker: /usr/lib/nagios/plugins/check_dhcp.

Then you need a plugin command config file.  I edited the dhcp.cfg command file (/etc/nagios/plugins/) and added these lines:

# 'check_rogue_dhcp' command definition
define command{
   command_name check_rogue_dhcp
   command_line /usr/lib/nagios/plugins/check_rogue_dhcp.pl -f '$ARG1$' '$ARG2$' '$ARG3$'
}

Then to actually invoke the check you need to define a nagios object where you give the command the IP addresses of the servers (12.34.12.34 etc).  That may need a hostgroup or some other object depending on how you have nagios set up

#check that no rogue dhcp services are running
define service {
   service_description rogue-dhcp
   check_command check_rogue_dhcp!12.34.12.34!12.34.12.45
   use generic-service
   notification_interval 0 ; set > 0 if you want to be renotified
}

There's various ways to do this and I'm not great at strategic configuration of nagios, so I'll leave that to you.

The script can be downloaded from here.

The script:
------------check_rogue_dhcp.pl-------------------


#!/usr/bin/perl -w
# nagios: -epn
# the above line makes nagios run the script as a separately.
# rather than as part of nagios.
use POSIX;
use lib "/usr/lib/nagios/plugins";
use utils qw(%ERRORS);

sub fail_usage {
  if (scalar @_) {
    print "$0: error: \n";
    map { print "   $_\n"; } @_;
  }
  print "$0: Usage: \n";
  print "$0 [<options>] <server> [<server> [<server>]] \n";
  print "$0 [<options>] [-s <server> [-s <server> [-s <server>]]] \n";
  print "    options:  \n";
  print "      [-v [-v [-v]]] (verbose) \n";
  print "      [-t  <secs>] (wait this number of seconds)   \n";
  print "      [-f] (fuzzy - ok if one or more of the designated servers answer)   \n";
  print "      [-F] (force (default) - all the designated servers must answer)   \n";
  print " \n";
  exit 3 ;
}

my $verbose = 0;
my %servers=();
my $opt = "-t 5";
my $time = 5;
my $force=1;

## for some reason I can't test for empty ARGs in the while loop
@ARGV = grep {!/^\s*$/} @ARGV;

# examine commandline args
while ($ARGV=$ARGV[0]) {
  my $myarg = $ARGV;
  if ($ARGV eq '-s') {
    shift @ARGV;
    if (!($ARGV = $ARGV[0])) { fail_usage ("$myarg needs an argument"); }
    if ($ARGV =~ /^-/) { fail_usage ("$myarg must be followed by an argument"); }
    if (!defined($servers{$ARGV})) { $servers{$ARGV}=1; }
  }
  elsif ($ARGV eq '-t') {
    shift @ARGV;
    if (!($ARGV = $ARGV[0])) { fail_usage ("$myarg needs an argument"); }
    if ($ARGV =~ /^-/) { fail_usage ("$myarg must be followed by an argument"); }
    if ($ARGV !~ /^(\d+)$/) { fail_usage ("$myarg must be followed by an number"); }
    $time = $1;
    $opt = "-t $time";
  }
  elsif ($ARGV eq '-f' ) { $force=0; }
  elsif ($ARGV eq '-F' ) { $force=1; }
  elsif ($ARGV eq '-h' or $ARGV eq '--help' ) { fail_usage ; }
  elsif ($ARGV =~ /^-/ ) { fail_usage " invalid option ($ARGV)"; }
  elsif ($ARGV =~ /^\d+\.\d+\.\d+\.\d+$/)
    # servers should be ip addresses.  I'm not doing detailed checks for this.
    { if (!defined($servers{$ARGV})) { $servers{$ARGV}=1; } }
  else { last; }
  shift @ARGV;
}

if (scalar @ARGV) { fail_usage "didn't understand arguments: (".join (" ",@ARGV).")"; }  
my $serversn = scalar keys %servers;

if ($verbose > 2) {
  print "verbosity=($verbose)\n";
  print "servers = ($serversn)\n";
  if ($serversn) { for my $i (keys %servers) { print "server ($i)\n"; } }
}

if (!$serversn) { fail_usage "no servers"; }
my $responses=0;
my $responders="";
my @check_dhcp = qx{/usr/lib/nagios/plugins/check_dhcp -v $opt};
foreach my $value (@check_dhcp) {
  if ($value =~ /Added offer from server \@ /i){
    $value =~ m/(\d+\.\d+\.\d+\.\d+)/i;
    my $host = $1;
    # we find a server in our list
    if (defined($servers{$host})) { $responses++; $responders.="$host "; }
    else {
      # we find a rogue DHCP server.  Danger Will Robinson!
      print "SERVICE STATUS:CRITICAL: Rogue DHCP service running on $host";
      exit $ERRORS{'CRITICAL'}
    }
  }
}
if ($responses == $serversn) {
  # we saw all the servers in our list.  All is good.
  print "SERVICE STATUS:OK: $responses of $serversn Expected Responses to DHCP Broadcast";
  exit $ERRORS{'OK'};
}

if ($responses == 0) {
  # we found no DHCP responses.
  print "SERVICE STATUS:CRITICAL: no DHCP service responded";
  exit $ERRORS{'CRITICAL'}
}

# we found less DHCP servers than we should have. Oh Nos!
$responders =~ s/ $//;
if ($force == 1) {
  print "SERVICE STATUS:WARNING: $responses of $serversn Responses to DHCP Broadcast. Only ($responders) responded. ";
  exit $ERRORS{'WARNING'};
}
else {
  print "SERVICE STATUS:OK: $responses of $serversn Responses to DHCP Broadcast. Only ($responders) responded. ";
  exit $ERRORS{'OK'};
}

2014-10-29

The myth of competition and the free market.

Mathematically the free market is not a stable environment.  It quickly becomes an oligopoly and from there it's only a step or two to a monopoly.

The point is that whenever and to the extent that there is a free market, companies must become efficient to compete.

But free markets only exist when companies compete, when they agree to compete, when they cooperate to compete.  The bulk of any company's business is done on a cooperative basis and that cooperation underlies everything we do as a society.

Free markets cannot exist if companies cooperate.  This is why some forms of cooperation have to be made illegal in order to create a "free market" but companies in a competitive environment can still cooperate without an agreement, without even communicating with each other. To state that more plainly: there is no "free market" without government intervention.

As soon as a market has reached the oligopoly stage, there are entrenched players and at this point new players find it very difficult to impossible to enter the market.  An oligopoly company gets far more benefit from cooperating in the market than competing.  Even if there is no overt agreement to cooperate, oligopoly players will benefit more from cooperation than competition.  They will especially cooperate to lock new and potential competitors out of their markets.  They may compete in a desultory way with other oligopoly members but ultimately the oligopoly state benefits them more than competition.

It is the oligopoly companies that are always extolling the virtues of the free market.

The primary enemy, the primary competitor of a company is competition itself.  Without competition companies don't have to be efficient, they can live off the fat of a monopoly or oligarchy.  There is no transparency.  Companies will always fight hardest to not have to compete.   Not having to compete is a massive benefit for them.

Companies in a free market, when it occurs and to the extent it occurs, will compete and will try to end that competition.  Competition will gradually go away for various reasons, because companies go bankrupt, give up in that market,  get bought out or merge and as the major players in a market get bigger, the market becomes an oligopoly and finally a monopoly and competition ceases.  Legislation can lengthen this process, that is can help maintain competition for longer, but cannot really stop it.  As the companies get large enough and rich enough they will lobby governments to change the rules to their benefit or they will go transnational and be out of the reach of governments or apply pressure in other ways.

When companies lobby for changes in law they will always site "the free market" and this is nearly always an indicator of an oligopoly.

2014-05-07

ngraph: another random text generator.

In my series of scripts to generate random text for fun and for helping create secure passwords this is my latest.  This new set of scripts generate text based on frequency weighted random choice of ngraphs.  I use it sometimes to generate word-like things  to create passwords.  There may be other uses, for instance creating random text with English characteristics.

ngraph:  Generating text from frequency weighted letter combinations

An ngraph is a group of n consecutive letters occurring in a language.  (This is my definition, there may be another word for it but I couldn't find it.) A set of ngraph frequencies is a set of the number of times each ngraph is used in a group of texts.  So for instance for n=1 we have the frequencies of the letters.  For n=2 we have the set of frequencies of the digraphs, for n=3, the trigraphs etc.  Because it is easy to do on a computer, I include some punctuation (space return - ' , . ; ! ? &).  I use these files among others:
The complete works of Jane Austen and the complete works of William Shakespeare.
Any texts would do.  Gutenburg texts have a few oddities that the script is designed to work with.

There are two main scripts and a subsidiary script.  (ngraph.pl, dbfill.sh and ngraph-db.pl)

The first script is "ngraph.pl": this script has two separate functions.
Firstly it reads a series of files with presumably text in a human language and generates a set of ngraphs for an "n" you specify.
Then ngraph.pl will generate a random set of text based on the ngraphs and their frequency.

Alternatively it can just output the set of ngraph frequencies as text or sql.  The reason for this is that reading the files and creating ngraph frequency tables is a resource intensive process so I decided to create a database of ngraphs and to generate text from that database.  I found a database with n = 1 to 5 to be most useful and above 5 the amount of data gets massive and more actual words are generated. 

The second script "dbfill.sh" is a subsidiary script.  It creates the database and populates it with ngraphs using the first script.

The third script is "ngraph-db.pl"
This uses the database and generates text based on the ngraphs in the database.  Because it has access to a database with ngraphs of say n = 1 to 5 it can generate text from random sized ngraphs as well as a single ngraph.

The generated text can include words but it mostly has word-like things that are a bit memorable but not actual words.  I never use the generated text directly to create passwords but pick and choose bits and let parts of the text inspire a password.

The scripts are available under the GPL here.

Here is a sample output:
$ngraph-db.pl -W -c 1000 -2 -g 1-4