Government control of ideas

From before 1AD to around 1400AD (I believe there is no zero AD because we didn't have zero till much later - an Indian invention!) there were three main centers of civilization - Europe, China and India.  They were of similar population and developed culturally and technically quite similarly.  Then around 1400, Europe started an unprecedented scientific and technological development that continued until recently.  Even though China invented the basis of many of our modern technologies, it never developed them or put them together.  It is interesting to contemplate why this is so. 

My theory is that to really develop technically and scientifically we need two important things in our social environment.  The first is a reasonably stable society and the second is freedom of ideas from government intervention. 

The first part: stable society.  Now I've not read Indian history but I believe that India didn't really provide a stable social environment.  I'm happy to be corrected in this and one day I hope to get time to read Indian history.  China did  have a stable culture, although with interregnums where it dissolved into chaos, but for much of the time it had a stable society.  Europe had a stable society although it was fluid as wars and changes swept through it but overall there was stability.  Europe had geographic divisions which allowed it to maintain diverse political boundaries but with countries remaining connected in a way China and India didn't have.

The second part: freedom of ideas from government intervention shows Europe to be quite different to the others.  For most of history when China was stable, it was also monolithic and uniform with considerable government intervention in ideas and trade.  The monolithic nature of China meant government intervention in ideas was final, there was nowhere to go.

Europe on the other hand was politically and geographically very diverse.  While any one part may not have been stable there were always parts that were stable.  To some extent people could even move to stable parts.  Europe was always more diverse and divided than China. 

The critical point in Europe seems to have been around the time of the reformation.  In the first days of the printing press, the Catholic church had an early and very harsh copyright law.  Allegedly this was to save the "jobs" of monks copying books by hand in scriptoriums but mainly to control and censor books.  The penalty for breaking the copyright law was death.  The law applied more or less in catholic countries, which was to say, much of western Europe.  But from the time of the reformation it was possible for people to by-pass the copyright law by printing in a non-catholic country.  This was how Gallileo's books were published.  His books were banned by the church, and therefore in catholic countries, from around 1633 to 1835.  In fact from the time of the reformation Europe experienced a period of scientific development unprecedented in world history. 

It seems clear to me that the geographic and political diversity and contention in Europe provided a key ingredient here, a way of by-passing government intervention in ideas. 

As Europe consolidates and creates uniform intellectual property laws: ie government intervention in ideas, it will cut off the critical driver that has moved it into the forefront of scientific development.  The US is in the process of doing this. 

Some background:  China and Europe 1500 - 2000

Update: I was reading this article in wired:
Why Black Market Entrepreneurs Matter to the World Economy and it gives an idea of the size, scale, energy and innovation of unregulated economies.

There seems to be a very fine balance between the government control you need to get a stable society and the over-control that stifles and censors ideas.  It seems to me that no single government can get it right.

A Million Monkeys: who owns the copyright?

The random sentence program that I wrote about in my last blog entry can output stories.   It's my version of the million monkeys.

As before, it can be downloaded as a simple perl text file from here or in the zip file with the other programs and the source frequency list from here.  

I will add instructions for creating the database soon.   The database is not necessary for using the program, it just makes it faster.

Each story on this page is created once and is unique, the one you are seeing is a story just for you.   Consider these stories copyrighted!   If you copy and use my program, and I encourage you to do so, it is free but I retain copyrights for all stories created by the program!!   All stories will be cc-sa.

This brings up a question for me: who owns the copyright on words generated by a machine?

Edited:  There was a copyright case recently, a photographer vs. wikipedia in which the court decided that a a photographer couldn't claim copyright of a selfie taken by a monkey because it was the monkey's copyright and monkeys can't own copyright.  So I guess the same goes for a machine.

$randsent -t 5 -p 1000 -g

More random passwords, entropy and words

Following on from my previous blog post I have written a new perl program, this time to create kind of sentence like structures.  It is very simple at the moment but I will continue working on it on and off.  As with the previous programs it is written in perl and like randword it uses Adam Kilgarriff's BNC list from here http://www.kilgarriff.co.uk/BNClists/all.al.gz albeit in a slightly different way - it uses the parts of speech encoded therein.

In looking for the structure of English I found among other things some very interesting pages from this site: http://papyr.com/hypertextbooks/grammar/, specifically the pages on the phrases: the noun phrase, the verb phrase, etc.  What I haven't found yet is anyone who has enumerated the probabilities of the structure: what is the probability, for instance, that the subject phrase of a sentence is a simple pronoun? What is the probability that the main adjective in an adjectival phrase is a verb participle? etc.  I'll probably be working on the way the structure is generated but at the moment the recursivity in the structure seems to have gotten away from me slightly.

This new program called "randsent" creates English-like sentences with a kind of rough grammar so the sentences often seem like English although they may not make a lot of sense.  Of course they're not meant to make sense but to be memorable.  I find it's rather fun to run it and enjoy some of the sentences.  Some of the words from Kilgarriff's list are a bit strange at times.

I've read that this might be done with Markov chains but I'm not familiar with how to do that.  Any interesting ideas, suggestions or rewrites accepted, or encouraged.

As before, it can be downloaded as a simple perl text file from here or in the zip file with the other programs and the source frequency list from here.  


Edit: If you have iframes here is an updating example:

$randsent -t 10 -p 1000

Random passwords, entropy and words

The concepts of passwords and entropy have generated a lot of discussion especially since the xkcd comic.  One of the basic results is that a small additional password length can add as much entropy to a password as adding a larger range of characters and could be much easier to remember.  Think of it like this if you like: each character in a password might be one of say 26 characters or if you add numbers and uppercase and punctuation maybe around 100 possibilities for each letter.  But the number of words in just  the English language gives you a much larger number of possibilities.

A long while ago I wrote a perl program for generating random passwords.  It and the new ones I talk about here are available under the GPL for you to download and use at the end of this article if you're interested.  To use them you will need a computer that runs perl.

My perl random password generator generates a password of random letters.  The command has lots of options to change the make-up of characters and the way it chooses them

$ randstring
 y5tr0nshc_mf
$ randstring -t 5
 yx02w_zlem79
 38rn91schrud
 2vnr2ujty1tt
 oijmmhpjokrg
 lr2v2vknlwws

For me, what started the idea for extending my random password generator to add random words was when someone on the link list sent me these first two links below.

My first try was a variation on the original which generates a series of wordlike things.

$ randstring -w -t 5
 rie hiuv disu coayu
 esxyi sc aim kyw
 kionuj lujyc oni aoahy
 fii ausnzg gad puku
 zna vymiq as mam

Then I decided to write a program that chooses random words from a dictionary.

$ randword -t 5
 yeta fit fot brach
 casave tid oleous pram
 lawyer Ro coto testa
 drawly ras Trapa Ao
 cosec crappo ay hi

Lastly I added a word generator that uses a weighted random choice based on a word frequency list so it has a tendency to choose more often used words.

$ randword -w -t 5
 look and april mr
 home not grass larger
 child that charts lucky
 the the very these
 andrew very she the

This command has many options including an interesting option to limit the word selection to more or less popular words.  In this next example all the words have a frequency of more than 100 in the list.  This cuts the word list in this case from 236660 entries to 26310.

$ randword -w -t 5 -p 100
 upon quest jokes to
 loved scale 's awards
 do in soft pounds
 for and it the
 us most your of

I'm not going to make a web page to do this at the moment, partly because the word-based password generators use quite a lot of memory and CPU resources and partly because someone has already done it (see below) but mainly because it's a silly idea to use a password from a website on the internet.  So for your benefit I am offering these scripts as a GPL program for those interested in using them.

I should note that the word frequency list is based on a 1989 one from Adam Kilgarriff's BNC lists from and thanks to him for permission to use it.
http://www.kilgarriff.co.uk/BNClists/all.al.gz

Links:
http://www.archonmagnus.com/articles/security/passwordEntropy.php
https://secure.wikimedia.org/wikipedia/en/wiki/Password_strength
http://www.baekdal.com/tips/password-security-usability
https://www.grc.com/haystack.htm
https://www.xkcd.com/936/

I've just discovered that someone wrote a webpage which generated "xkcd" style passwords:
http://preshing.com/20110811/xkcd-password-generator
http://passphra.se/

Here are links to download my programs:
randstring
randwords
tgz archive (includes randstring, randword and the word frequency list).

They are, of course, works in progress and any suggestions, bug reports, etc will be gratefully accepted. 

The dictionary version looks for a standard linux, BSD or MacOS dictionary at "/usr/shar/dict/words" with one entry (word) per line of probably ASCII text.  If you don't have that, you may have to get a copy of it.  I have not internationalized the programs.

Commons, the environment and the public domain

One of the problems with the previous age - the industrial age was (and still is) pollution. The environment which we all depend on to survive has not come into economics which is what business focuses on. So essentially a company dumping toxic or climate changing waste products into our environment was getting a free ride. This works until you add the environment into the economics. The problem with the environment is the problem of commonly owned things. They don't really fit into a private-ownership based economic system. They are a commons. It is complicated to value the damage to the environment in order to make companies pay for that damage. Still, people are working on the problem and good or bad, paying for CO2 is on the horizon just as paying to (or in some cases paying to not) pollute has been there for some time.

In the world of science, knowledge and art the environment is the public domain. Well, it's actually more than the pure public domain. It includes scraps of things which have been created recently. This is the environment or knowledge-base upon which all works or inventions are based. The life, the air from which science and art arise. One of the key unaddressed issues with copyright is what a monopoly right of a work of art or an invention takes out of the intellectual environment, the intellectual commons. i.e. what we lose as a result of those monopoly rights.

For instance, it has mostly always been a right of users and of learning institutions to copy parts of works to show points, to illustrate arguments etc. This is called "fair-use" in the US but it might have other names in other countries, it might be an educational exemption for instance. If companies publish works in which it is not possible to do this (because of DRM say) then they are taking, stealing if you like, from the intellectual commons.

It has almost everywhere been the right of users to lend a work to a friend, to sell it when they don't want it anymore. This in the US is called the "first sale" doctrine. Digital works with DRM do not have this right at all in most cases.

I'm not an economist but I think it's vital that we try and find ways to evaluate in an economic or monetary sense the value taken from the intellectual environment of creative works and including taking rights like fair-use and first sale. ie what do we lose in order to create copyright on a work. What do we lose, or pay if copyright is extended for another say 20 years.

How much does it cost us if DRM takes away fair use and first sale?


Here is a small part of an FSF article which talks about some of these concepts:

In the case of copyright, it’s the public’s freedom that the government is spending, to obtain in return for the public scientific and cultural goods. Right now, governments are squandering this freedom. They are spending far too much and getting far too little in return. Plenty of authors and artists are telling the government that works can and will be made without such expenditure. The international free software movement has been proving this for many years now, having successfully produced a fully functional operating system in GNU/Linux that can be freely used, shared and improved upon by anyone who wants to do so; and more recently there have been people doing similar things in encyclopedias, textbooks, and the world of the arts (including music).

Citizens are tired of watching their governments squander their freedom to enrich this handful of corporations,

Again, we need to find a way of quantifying our losses to make it clear what we are losing.

UPDATE:
I don't think it is really possible for most people to make sense of the idea without some quantifiable grasp of what the value is. While I and many others clearly feel we are losing something important, without numbers it is meaningless to many.

I do have one idea but I'm not sure how useful it is. Perhaps the amount lost by a DVD with DRM could be some particular amount. I'm not sure what that amount should be but I think it would probably be the same no matter how popular or unpopular the DVD is. No matter how many were sold.

Copyright Musings

Here are some of my musings about copyright. They're not really complete or definite, just thoughts.

This arises out of the awareness that computers, the internet, digital technology, have changed the nature of copying and therefore copyright. The fact is that with modern computers copying is not only incredibly simple, trivial even, but it is an essential part of the process of reading, listening or watching a work. To read or listen to or watch something you must at least copy it into memory and then onto the screen or output device. The ability to copy and communicate work will only become easier and simpler and will not go away. The old concepts of copyright will inevitably fall away and take business models based on it with it.

Just a short note about the use of encryption to control copying:
DRM (Digital Rights Management) and various encryption techniques have been tried in order to try and control the ability of users to copy digital files but they end up being much more difficult to use than simple unrestricted files and much more restrictive than plain copies are. For example, you can't "lend" a restricted file to a friend like you can a book, CD, DVD, you can't copy a paragraph of text etc. Encumbered files can cause people great difficulty and even make them unable to use data they have paid for (for instance when users upgrade their computers or a key server goes away). Given a choice people will choose not to use such things. They will opt for unencumbered data. Ultimately in order to use such things people must be able to decrypt the data and view it in the clear and this will always be the downfall of encrypting data for public viewing. In order to be successful corporations must control the viewing platform. You end up with a situation where you need different devices to view different companies files.

With the internet these days, what is on the internet stays on the internet. One of the things that has changed is that many people can "publish" their own works directly on the internet with no middle-men, no publishers, distributors or physical shop-fronts. Actually for most people that's not entirely true, but the middle-men have changed to web-hosting companies, secure-online payment companies, web designers, ISPs etc. Mainly people who really have very little or nothing to do with the content at all.

I think that copyright on books, music and movies are very different things and affect different people and traditional businesses in different ways.

Copyright on books, on the written word, has quite a long tradition, it has been around since sometime after the invention of the printing press. Say about 200 or 300 years. There are many people who earn their living one way or another by the use of copyright on words. You may have noticed that books published in the last 70 years or so are much less likely to be on the net. Publishing companies gained leverage and earn money on the basis of the scarcity and expense of copying equipment. The old publishing industry has really not come to grips with the net at all so there are many books that have never or will not at least for the near future be openly available on the net even though the ability to manipulate, to copy, transmit over the internet and view books has been around for more than a decade now. Newspapers and magazines are trying various business models to cope with the internet but finding restricting readership is not too successful. Not going online is not really an option for them either though.

Compare copyright of books with music. The technology that allowed us to reproduce and copy sounds, pictures and moving pictures arose much later and the ideas of copyright of the written word were extended to cover music and movies. Until near the beginning of the 20th century musicians earned their money by performance and most still do. Then people began to be able to reproduce music and sell copies. Very few musicians ever earned really big bucks from this. What has changed is that there are some, let's call them "publishing" companies, that have made an enormous amount of money out of music but this really only started in a big way half way through the 20th century. The window where recording equipment was possible and expensive and before digitisation allowed copying to become trivial was much smaller for sound than for the written word and while the profits involved are staggering and those companies don't want to give up their golden goose, the window for this has really passed quite quickly.

If you think of movies as an extension of the playhouse the situation with movies is really rather similar to music. The differences are that people can spend a lot more money to make movies and movie making can require a large team of people. Until fairly recently copying movies and even just watching movies required special equipment. Technology to distribute and watch movies in the home is not that old, starting with TV and extending to videos and finally digital but the quality is only now reaching cinema quality. Communication and storage bandwidth have only recently got to a point where watching, storing and transmitting reasonable quality movies is in the normal range of ordinary household computer activity.

In other words if you consider the window between the time which copying was first possible but was expensive and involved scarce resources and the time when it became trivial, the written word had a much longer window compared to music or movies and the written word has a much more entrenched tradition of making money through copyright. Ultimately though, copying being a trivial operation now and into the future the whole idea of copyright and business models that depend on it will inevitably have to change.

Post Muse (2009-01-16) : I was talking about this with a friend the other day and he said something which surprised me. He said: "I don't read actual books much these days. I do read a lot but I don't read much that isn't published online any more." I wonder if this will be the fate of books, to be left behind as publishing moves to the web. Certainly technical writing seems to be moving that way. Things just move so fast that a book is often old as soon as it's printed.