Author Archives: jeremytarling

About jeremytarling

Data Architect for BBC News online

a tarot card reader in Python

I’ve recently been helping some work colleagues get their heads around Python’s Flash web toolkit, and it seemed like a good opportunity to update my old Ruby tarot card reader to Python.

https://python-tarot.herokuapp.com/

The app draws one or more randomly selected cards from a dictionary (hash) of 72 Tarot card description and image references.

The card images used are Arthur Waite’s deck, by artist Pamela Colman Smith. According to Wikipedia they are out of copyright in the US now and will be in the UK in 2022.

The source code to this app is here – please feel free to fork or re-use, there are no license restrictions.

extracts from Towards Democracy

Edward Carpenter’s seminal work Towards Democracy has been a constant companion to me since discovering it in the work of Denning and Phillips.  Here’s a passage with magical or alchemical meaning:

“Is there one in all of the world who does not desire to be divinely beautiful?
To have the most perfect body – unerring skill, strength, limpid clearness of mind, as of the sunlight over the hills,
To radiate love wherever he goes, to move in and out, accepted?
The secret lies close to you, so close.

You are that person; it lies close to you, so close – deep down within –
But in Time it shall come forth and be revealed.

Not by accumulating riches, but by giving away what you have,
Shall you become beautiful;
You must undo the wrappings, not case yourself in fresh ones;
Not by multiplying clothes shall you make your body sound and healthy, but rather by discarding them;
Not by multiplying knowledge shall you beautify your mind;
It is not the food that you eat that has to vivify you, but you that have to vivify the food.

Always emergence, and the parting of veils for the hidden to appear;
The child emerges from its mother’s body, and out of that body again in time another child.
When the body which thou now hast falls away, another body shall be already prepared beneath,
And beneath that again another.
Always that which appears last in time is first, and the cause of all – and not that which appears first.

===

Freedom has to be won afresh every morning,
Every morning thou must put forth thy strength afresh upon the world, to create out of chaos the garden in which thou walkest.
(Behold! I love thee – I wait for thee in thine own garden, lingering till eventide among the bushes;
I tune the lute for thee; I prepare my body for thee, bathing unseen in the limpid waters.)

===

Wondrous is Man – the human body: to understand and possess this, to create it every day afresh, is to possess all things.
The tongue and all that proceeds from it; spoken and written words, languages, commands, controls, the electric telegraph girdling the earth;
The eyes ordaining, directing; the feet and all that they indicate – the path they travel for years and years;
The passions of the body, the belly and the cry for food, the heaving breasts of love, the phallus, the fleshy thighs,
The erect proud head and neck, the sturdy back, and knees well-knit or wavering; All the interminable attitudes and what they indicate;
Every relation of one man to another, every cringing, bullying, lustful, obscene, pure, honourable, chaste, just and merciful;
The fingers differently shaped according as they handle money for gain or gift;
All the different ramifications and institutions of society which proceed from such one difference in the crook of a finger;
All that proceed from an arrogant or slavish contour of the neck;
All the evil that goes forth from any part of a man’s body which is not possessed by himself, all the devils let loose – from a twist of the tongue or a leer of the eye, or the unmanly act of any member – and swirling into society; all the good which gathers round a man who is clean and strong – the threads drawing from afar to the tips of his fingers, the interpretations in his eyes, all the love which passes through his limbs into heaven;
What it is to command and be Master of this wondrous body with all its passions and powers, to truly possess it – that is to command and possess all things, that is to create.

===

The art of creation, like every other art, has to be learnt:
Slowly, slowly, through many years, thou buildest up thy body,
And the power that thou now hast (such as it is) to build up this present body, thou hast acquired in the past in other bodies;
So in the future shalt thou use again the power that thou now acquirest.
But the power to build up the body includes all powers.

Do not be dismayed because thou art yet a child of chance, and at the mercy greatly both of Nature and fate;
Because if thou wert not subject to chance, then wouldst thou be Master of thyself; but since thou art not yet Master of thine own passions and powers, in that degree must thou needs be at the mercy of some other power.
And if thou choosest to call that power “Chance”, well and good. It is the angel with whom thou hast to wrestle.

===

Beware how thou seekest this for thyself and that for thyself. I do not say Seek not, but Beware how thou seekest.
For a soldier who is going on a campaign does not seek what fresh furniture he can carry on his back, but rather what he can leave behind;
Knowing well that every additional thing which he cannot freely use and handle is an impediment to him.
So if thou seekest fame or ease or pleasure or aught for thyself, the image of that thing which thou seekest willcome and cling to thee – and thou wilt have to carry it about;
And the images and powers which thou hast thus evoked will gather round and form for thee a new body – clamouring for sustenance and satisfaction;
And if thou art not able to discard this image now, thou wilt not be able to discard that body then: but will have to carry it about.
Beware then lest it become thy grave and thy prison – instead of thy winged abode, and palace of joy.

For (over and over again) there is nothing that is evil except because a man has not mastery over it; and there is no good thing that is not evil if it have mastery over a man;
And there is no passion or power, or pleasure or pain, or created thing whatsoever, which is not ultimately for man and for his use – or which he need be afraid of, or ashamed at.
The aesthetics and the self-indulgent divide things into good and evil – as it were to throw away the evil;
But things cannot be divided into good and evil, but all are good so soon as they are brought into subjection.
And seest thou not that except for Death thou couldst never overcome Death –
For since by being a slave to things of sense thou hast clothed thyself with a body which thou art not master of, thou wert condemned to a living tomb were that body not to be destroyed.
But now through pain and suffering out of this tomb shalt thou come; and through the experience thou hast acquired shalt build thyself a new and better body;
And so on many times, till thou spreadest thy wings and hast all powers diabolic and angelic concentrated in thy flesh.

===

And so at last I saw Satan appear before me – magnificent, fully formed.
Feet first, with shining limbs, he glanced down from above among the bushes,
And stood there erect, dark-skinned, with nostrils dilated with passion;
(In the burning intolerable sunlight he stood, and I in the shade of the bushes);

Fierce and scathing the effluence of his eyes, and scornful of dreams and dreamers (he touched a rock hard by and it split with a sound like thunder);
Fierce the magnetic effluence of his dusky flesh; his great foot, well-formed, was planted firm in the sand – with spreading toes;
“Come out,” he said with a taunt, “Art thou afraid to meet me?”

And I answered not, but sprang upon him and smote him; And he smote me a thousand times, and brashed and scorched and slew me as with hands of flame;

And I was glad, for my body lay there dead; and I sprang upon him again with another body: And he turned upon me, and smote me a thousand times and slew that body; And I was glad and sprang upon him again with another body-And with another and another and again another;

And the bodies which I took on yielded before him, and were like cinctures of flame upon me, but I flung them aside; And the pains which I endured in one body were powers which I wielded in the next, and I grew in strength, till at last I stood before him complete, with a body like his own and equal in might – exultant in pride and joy.

Then he ceased, and said “I love thee.” And lo! his form changed, and he leaned backwards and drew me upon him, And bore me up into the air, and floated me over the topmost trees and the ocean, and round the curve of the earth under the moon – till we stood again in Paradise.”

calculating Term Frequency in Ruby

I’ve recently been doing an online course in Data Science, part of which involves practical exercises in data mining and sentiment evaluation. The course required me to do them in Python which was a useful learning experience, but since Ruby is my preferred hacking language I’ll re-do some of them in Ruby here over the course of a few posts. This one is about calculating Term Frequency, a common requirement for text mining and the first step in a TF-IDF analysis.

1. read your text into an array of words

using a simple regex here to ignore anything that doesn’t start with a letter:

words = File.read("my_doc.txt").text.split(/\W+/)

(for big documents you’ll want to split this e.g. using each_line)

2. make a hash of word frequency

word_freq = Hash.new(0)
words.each { |word| word_freq[word] += 1 }

as always with Ruby there’s more than one way to do it so here are a couple of alternatives; first with reduce (inject):

word_freq = words.reduce(Hash.new(0)) {|word, freq| word[freq] +=1; word}

and here’s another way using Enumerable’s each_with_object method:

word_freq = words.each_with_object(Hash.new(0)) {|freq, word| word[freq] +=1}

(note the switching of the order of key/value params).

I’ve not tested which of these is most performant against large documents yet but from a purely stylistic point of view I find each_with_object the clearest way

3. calculate the TF by dividing word frequency by total words in the text

term_freq = Hash[word_freq.map { |k,v| [k, (v.to_f / words.length).round(3) ] } ]

or if you’re on Ruby 2.1 the rather tidier .to_h Enumerable method:

term_freq = word_freq.map { |k,v| [k, (v.to_f / words.length).round(3) ] }.to_h

4. sort, reverse, enjoy!

Hash[term_freq.sort_by { |k,v| v}.reverse]

(or again, in 2.1)

term_freq.sort_by { |k,v| -v }.to_h

That’s about it, nothing too taxing. Next up I’ll do Inverse Document Frequency to complete the TF-IDF algorithm.