Government control of ideas

From before 1AD to around 1400AD (I believe there is no zero AD because we didn't have zero till much later - an Indian invention!) there were three main centers of civilization - Europe, China and India.  They were of similar population and developed culturally and technically quite similarly.  Then around 1400, Europe started an unprecedented scientific and technological development that continued until recently.  Even though China invented the basis of many of our modern technologies, it never developed them or put them together.  It is interesting to contemplate why this is so. 

My theory is that to really develop technically and scientifically we need two important things in our social environment.  The first is a reasonably stable society and the second is freedom of ideas from government intervention. 

The first part: stable society.  Now I've not read Indian history but I believe that India didn't really provide a stable social environment.  I'm happy to be corrected in this and one day I hope to get time to read Indian history.  China did  have a stable culture, although with interregnums where it dissolved into chaos, but for much of the time it had a stable society.  Europe had a stable society although it was fluid as wars and changes swept through it but overall there was stability.  Europe had geographic divisions which allowed it to maintain diverse political boundaries but with countries remaining connected in a way China and India didn't have.

The second part: freedom of ideas from government intervention shows Europe to be quite different to the others.  For most of history when China was stable, it was also monolithic and uniform with considerable government intervention in ideas and trade.  The monolithic nature of China meant government intervention in ideas was final, there was nowhere to go.

Europe on the other hand was politically and geographically very diverse.  While any one part may not have been stable there were always parts that were stable.  To some extent people could even move to stable parts.  Europe was always more diverse and divided than China. 

The critical point in Europe seems to have been around the time of the reformation.  In the first days of the printing press, the Catholic church had an early and very harsh copyright law.  Allegedly this was to save the "jobs" of monks copying books by hand in scriptoriums but mainly to control and censor books.  The penalty for breaking the copyright law was death.  The law applied more or less in catholic countries, which was to say, much of western Europe.  But from the time of the reformation it was possible for people to by-pass the copyright law by printing in a non-catholic country.  This was how Gallileo's books were published.  His books were banned by the church, and therefore in catholic countries, from around 1633 to 1835.  In fact from the time of the reformation Europe experienced a period of scientific development unprecedented in world history. 

It seems clear to me that the geographic and political diversity and contention in Europe provided a key ingredient here, a way of by-passing government intervention in ideas. 

As Europe consolidates and creates uniform intellectual property laws: ie government intervention in ideas, it will cut off the critical driver that has moved it into the forefront of scientific development.  The US is in the process of doing this. 

Some background:  China and Europe 1500 - 2000

Update: I was reading this article in wired:
Why Black Market Entrepreneurs Matter to the World Economy and it gives an idea of the size, scale, energy and innovation of unregulated economies.

There seems to be a very fine balance between the government control you need to get a stable society and the over-control that stifles and censors ideas.  It seems to me that no single government can get it right.

A Million Monkeys: who owns the copyright?

The random sentence program that I wrote about in my last blog entry can output stories.   It's my version of the million monkeys.

As before, it can be downloaded as a simple perl text file from here or in the zip file with the other programs and the source frequency list from here.  

I will add instructions for creating the database soon.   The database is not necessary for using the program, it just makes it faster.

Each story on this page is created once and is unique, the one you are seeing is a story just for you.   Consider these stories copyrighted!   If you copy and use my program, and I encourage you to do so, it is free but I retain copyrights for all stories created by the program!!   All stories will be cc-sa.

This brings up a question for me: who owns the copyright on words generated by a machine?

Edited:  There was a copyright case recently, a photographer vs. wikipedia in which the court decided that a a photographer couldn't claim copyright of a selfie taken by a monkey because it was the monkey's copyright and monkeys can't own copyright.  So I guess the same goes for a machine.

$randsent -t 5 -p 1000 -g

More random passwords, entropy and words

Following on from my previous blog post I have written a new perl program, this time to create kind of sentence like structures.  It is very simple at the moment but I will continue working on it on and off.  As with the previous programs it is written in perl and like randword it uses Adam Kilgarriff's BNC list from here http://www.kilgarriff.co.uk/BNClists/all.al.gz albeit in a slightly different way - it uses the parts of speech encoded therein.

In looking for the structure of English I found among other things some very interesting pages from this site: http://papyr.com/hypertextbooks/grammar/, specifically the pages on the phrases: the noun phrase, the verb phrase, etc.  What I haven't found yet is anyone who has enumerated the probabilities of the structure: what is the probability, for instance, that the subject phrase of a sentence is a simple pronoun? What is the probability that the main adjective in an adjectival phrase is a verb participle? etc.  I'll probably be working on the way the structure is generated but at the moment the recursivity in the structure seems to have gotten away from me slightly.

This new program called "randsent" creates English-like sentences with a kind of rough grammar so the sentences often seem like English although they may not make a lot of sense.  Of course they're not meant to make sense but to be memorable.  I find it's rather fun to run it and enjoy some of the sentences.  Some of the words from Kilgarriff's list are a bit strange at times.

I've read that this might be done with Markov chains but I'm not familiar with how to do that.  Any interesting ideas, suggestions or rewrites accepted, or encouraged.

As before, it can be downloaded as a simple perl text file from here or in the zip file with the other programs and the source frequency list from here.  


Edit: If you have iframes here is an updating example:

$randsent -t 10 -p 1000

Random passwords, entropy and words

The concepts of passwords and entropy have generated a lot of discussion especially since the xkcd comic.  One of the basic results is that a small additional password length can add as much entropy to a password as adding a larger range of characters and could be much easier to remember.  Think of it like this if you like: each character in a password might be one of say 26 characters or if you add numbers and uppercase and punctuation maybe around 100 possibilities for each letter.  But the number of words in just  the English language gives you a much larger number of possibilities.

A long while ago I wrote a perl program for generating random passwords.  It and the new ones I talk about here are available under the GPL for you to download and use at the end of this article if you're interested.  To use them you will need a computer that runs perl.

My perl random password generator generates a password of random letters.  The command has lots of options to change the make-up of characters and the way it chooses them

$ randstring
 y5tr0nshc_mf
$ randstring -t 5
 yx02w_zlem79
 38rn91schrud
 2vnr2ujty1tt
 oijmmhpjokrg
 lr2v2vknlwws

For me, what started the idea for extending my random password generator to add random words was when someone on the link list sent me these first two links below.

My first try was a variation on the original which generates a series of wordlike things.

$ randstring -w -t 5
 rie hiuv disu coayu
 esxyi sc aim kyw
 kionuj lujyc oni aoahy
 fii ausnzg gad puku
 zna vymiq as mam

Then I decided to write a program that chooses random words from a dictionary.

$ randword -t 5
 yeta fit fot brach
 casave tid oleous pram
 lawyer Ro coto testa
 drawly ras Trapa Ao
 cosec crappo ay hi

Lastly I added a word generator that uses a weighted random choice based on a word frequency list so it has a tendency to choose more often used words.

$ randword -w -t 5
 look and april mr
 home not grass larger
 child that charts lucky
 the the very these
 andrew very she the

This command has many options including an interesting option to limit the word selection to more or less popular words.  In this next example all the words have a frequency of more than 100 in the list.  This cuts the word list in this case from 236660 entries to 26310.

$ randword -w -t 5 -p 100
 upon quest jokes to
 loved scale 's awards
 do in soft pounds
 for and it the
 us most your of

I'm not going to make a web page to do this at the moment, partly because the word-based password generators use quite a lot of memory and CPU resources and partly because someone has already done it (see below) but mainly because it's a silly idea to use a password from a website on the internet.  So for your benefit I am offering these scripts as a GPL program for those interested in using them.

I should note that the word frequency list is based on a 1989 one from Adam Kilgarriff's BNC lists from and thanks to him for permission to use it.
http://www.kilgarriff.co.uk/BNClists/all.al.gz

Links:
http://www.archonmagnus.com/articles/security/passwordEntropy.php
https://secure.wikimedia.org/wikipedia/en/wiki/Password_strength
http://www.baekdal.com/tips/password-security-usability
https://www.grc.com/haystack.htm
https://www.xkcd.com/936/

I've just discovered that someone wrote a webpage which generated "xkcd" style passwords:
http://preshing.com/20110811/xkcd-password-generator
http://passphra.se/

Here are links to download my programs:
randstring
randwords
tgz archive (includes randstring, randword and the word frequency list).

They are, of course, works in progress and any suggestions, bug reports, etc will be gratefully accepted. 

The dictionary version looks for a standard linux, BSD or MacOS dictionary at "/usr/shar/dict/words" with one entry (word) per line of probably ASCII text.  If you don't have that, you may have to get a copy of it.  I have not internationalized the programs.

Random musings on Peter F. Hamilton's Commonwealth Saga

Peter F. Hamilton Commonwealth Saga:
Pandora's Star, Judas Unchained

This is really a single book published as two volumes but as each is around 1200 pages, it would be way too big as a single book. It's a pretty good read, complex with many threads.

I have some issues with the "technology". Given of course that in reality advanced technology is not always completely understood by most people using it but it is somewhat of a trope in novels that many things are explained to us the readers that the characters in the book don't get to see.

Longevity:
As a platform for his exploration of longevity with rejuvenation and recording and recloning life the saga is an interesting but distracting platform. As with the wormhole/teleportation technology he doesn't really explore much the possibility of duplication. Perhaps wormholes create teleportation without the issue of duplication but the longevity technology doesn't have the same limitation at all.

Wormholes:
The commonwealth is based on wormhole technology which allows people to move between worlds not too far away (in light years), instantaneously. Generators can create a wormhole between the gateway and a point up to some light years distance. I think of the maths involved in tracking a wormhole between a spot on the rotating and revolving and vibrating Earth and the rotating and revolving and vibrating surface of Mars. Not a simple thing. It is never explained how wormholes are limited by distance. There is some limit. How many light years away is too far? Is the limit energy? Surely wormholes could allow you to skim energy off a sun more directly and open the availability of vast amounts of energy? That is never explored. Is the energy required to set up the wormhole or to run it or to transfer stuff or all three?

I like his use of trains as the main form of transport through the wormholes. A very British idea and well handled. Kind of quantum steam-punk.

One thing that is never explored is the use of wormholes themselves as a weapon. I can't see why you couldn't slice off part of a ship or missile. OK force fields would apparently stop that. Or place a wormhole in front of a missile or ship and transfer it into the sun or far away. Or shoot missiles back into hostile wormholes. None of the ships have their own wormholes as transporters or weapons even though their FTL drive is based on wormholes. A ship equipped with wormhole missile launchers? Transporters to the planet? Why not?

This also brings up the whole problem of what happens at the edges of the wormholes. Especially a floating one, generated by a single gateway at a distance. Also what would happen if a wormhole somehow turned off when you were half way through is not discussed.