I have discovered a small problem in a well deployed Unix utility. It is (just) possible that this problem has some security related issues.
This code, by the way, is probably in every single deployed Unix system.
What should I do now?
BBC: don’t know their GDP from their elbow
2008-10-03Why do I subject myself to the horror that is BBC science reporting?
In this article about funding Kopernikus (you know, earth observation) says:
Then, the UK contributed just £11m for the entire first segment, or phase, of Kopernikus (the new name for GMES). This amounted to 4.5% of UK gross domestic product (GDP); industry had expected Britain to participate at around 17%.
Anyone with a gram of common sense could instantly see that this 11 million being 4.5% of GDP is just total nonsense. I’m sure there’s something going on that involves those numbers, but it’s not that. Probably Kopernikus has some sort of funding scale set according to a country’s GDP, but I couldn’t find out what it is, and this article certainly doesn’t make it clear.
Fucking arts graduates.
[2008-10-03: Corrected "countries" to "country's". Fucking maths graduates.]
Python and bragging about C89
2008-10-02People who brag about how cool and modern they are to require a C89 compiler (that’s you Python 2.6) should not be using // for comments. Maybe they meant they require a C99 compiler.
And yes, this really matters. On AIX 6.1 the Python 2.6 configure script selects a C89 compiler («cc_r -qlanglvl=extc89» if you must know). Which barfs on their warthog creating code.
Meh. Maybe I’ll even file a bug report.
C: return and parentheses
2008-10-02Please don’t write «return(0);». The parentheses are unnecessary. This request generalises. So the NSIDC should not be writing (on line 90 of cproj.c; some sort of Google codesearch link):
return((cosphi / (sqrt (1.0 - con * con))));
Oh yeah. Gotta love those parentheses; they give a nice warm cuddly feeling and make my expressions feel extra protected.
Program as if you know the language. At least a bit:
return cosphi / sqrt (1.0 - con * con);
There. Isn’t that clearer? More parentheses doesn’t always mean clearer code.
700,000,000,000 USD
2008-09-29How can we visualise such a large amount?
Well, it’s about a dime for every square meter of land in the US (9.1 cents per square meter of the contiguous 48).
Instead of buying Wall Street, the US taxpayers could’ve bought over 77 Large Hadron Colliders!
Or… enough gold to cover Lake Superior in a gold sheet (16 nanometers thick!).
700 billion dollars, man!
My Battle with Metroids
2008-09-29«Metroid II: Return of Samus» (Game Boy): Got to final boss, haven’t killed it.
«Super Metroid» (SNES): Got to final boss, haven’t killed it.
«Metroid Prime» (GameCube): Got to final boss (*), haven’t killed it.
«Metroid Fusion» (Game Boy Advance): Got to final boss, killed it! Yay!
So, what are my chances with:
«Metroid Prime Hunters» (DS)
?
(*) I have a sneaking suspicion that the “final boss” I am currently fighting in Metroid Prime may not in fact be the final boss. How can I tell?
I learn Python
2008-09-25It takes a long time to get to the point where you stop learning a language, perhaps you never do stop.
Some useful Python tidbits I’ve recently picked up that I feel I ought to have known much earlier:
Using the dict constructor instead of the literal syntax:
# awkward
{'cht': 'lc', 'chs': '500x600'}
# nice
dict(cht='lc', chs='500x600')
This is mentioned in the tutorial. Bad code monk. (Ah, this is that new fangled Python 2.3; that explains why I previously didn’t know it. Basically I learnt the vast majority of my working Python when lambda was new and «from __future__ import nested_scopes» was necessary)
When I met Raymond Hettinger at PyCon UK (I was doing a “Code Clinic” with him) one of the first questions I asked him was “when are we going to get unzip“; when he replied “zip is also unzip, you can just use zip(*l)” I had one of those forehead slapping moments (quite literally I think). I course I knew that “zip(*l)” transposed a matrix (see Norvig’s IAQ), but I wasn’t used to thinking of a list of pairs, say, as being a matrix that I could transpose to get a pair of lists. Kick! Punch! It’s all in the mind.
Using split instead of literal lists:
# awkward ['F', 'PD', 'AD', 'D', 'TD', 'ED', 'ABO'] # nice 'F PD AD D TD ED ABO'.split()
This isn’t so much a new trick, but I was always a bit embarrassed about it. But now I’ve seen other Python programmers do it too, so I know it’s more socially acceptable.
Now one that I don’t actually use so much, nicked from Thomas Guest’s blog:
"chd=s:%(xs)s,%(ys)s" % locals()
Can you see what’s going on? The variables xs and ys are local variables whose values are spread into the string using «% locals()» as a sort of “interpolate from local variables” operator. I’m not a great fan, but I can see that it is useful. I’m not such a fan because it uses locals which is very cool, but could interfere with a compiler’s optimisations. Though in this case, with a constant string on the left of the % operator, the compiler has enough information to do a good analysis.
Speaking of Python string formatting, I was disappointed to learn that the optimisation of compiling format strings, which is routine in the Lisp world, is not generally done in Python. The optimisation I am talking about is one where «”some constant string” % stuff» gets converted into «some_function(stuff)» where some_function is a compiled function that does the formatting. It’s just one of the signs of how immature whole Python thing is.
And one specially for Pythonistas doing Google charts:
>>> d = dict(cht='lc', chs='500x600') # The dict from above
>>> '&'.join(map('='.join, d.items()))
'chs=500x600&cht=lc'
An earlier version of this example involved a lambda: «’&’.join(map(lambda item: ‘=’.join(item), d.items()))», but then just as I was pasting it into this article I realised I could drop the lambda altogether. Bound methods rule!
What Python have you learnt recently?
Putting the Heat on Wheat
2008-09-24Wherein I play with the lovely Google Charts API and expose my total incompetence in statistics, economics, agriculture, and geography. And quite possibly other things too.
So I was reading the Open Knowledge Foundation blog and came across this article featuring US wheat production, which points to this dataset of wheaty goodness. My recent work on Clear Climate Code had made me already aware of the availability of GISTEMP’s summary data products.
So it occurred to me that this could be used to answer the question “when the weather is warmer, does more wheat grow?”.
So the wheat data is US wheat production, including yields in bushels/acre, sigh. GISTEMP even do a dataset that shows the temperature anomaly for the US. I think this is incredibly parochial, but it happens to be just what I want.
So the wheat yield (volume of wheat per harvested unit area) has a general upward trend. At least from the mid 1930’s or so. Because I’m only interested in the local variation I have detrended the wheat data:
My hypothesis is that any deviation of the temperature from the long term average will lower wheat yields. I think this because I would expect that over the thousands of years of selection humans will have cultivated a variety of wheat that is optimised to grow at the average temperatures and it will do less well when temperatures deviate.
So what do we see? Here’s wheat yields and temperatures together:
Well, there’s no obvious correlation to eyeball. Scattergram:
(which is almost just changing ‘cht=lc’ to ‘cht=s’ in the above chart URL)
Bit of a blurry mess. If anything a slight negative trend, which would mean that colder temperatures gave a higher wheat yield. And indeed Pearson’s correlation is about -0.3 (assuming my calculations are correct) indicating a weak negative correlation.
There are problems. One problem is that I have no p-value. That’s partly because I haven’t read that far on the Wikipedia page (I’m not using some fancy stats package for my analysis; everything is hand-coded in Python), and partly because I have a degrees of freedom problem. Temperature is autocorrelated, so whilst I have 128 samples, that’s fewer than 128 degrees of freedom, so the standard assumption of independent variables is incorrect.
The other problem is that it looks like the detrending might have introduced a bit of an alarming feature into the wheat anomalies. There’s a gentle hump from 1866 to about 1940 and a similar one from about 1940 to 2000. This is almost certainly because I’ve used a cubic polynomial to fit to the data to detrend it. It looks like a two-leg linear fit would be better (with a kink around 1942), but I haven’t found how to do that. I have a sneaking suspicion I have some FORTRAN code lying around here to do it, but I’m too scared to look.
Final tiny problem almost too small to be worth mentioning: the wheat data is for the entire US, whereas the temperature data is for the contiguous 48. I’m guessing that Alaska and Hawaii make so little wheat contribution that it doesn’t matter.
In any case it doesn’t really look like fixing these problems would ever indicate a strong positive trend between temperature anomalies and wheat yields. So we can reject the notion that warmer weather means higher wheat yields. Of course warmer weather might mean we can grow more of something else (possibly just a different variety of wheat); it also might mean that the available belt of land for growing wheat is larger (but this is unlikely since it probably means the available belt of land for growing wheat has moved North).
Ofsted: satisfactory doublethink
2008-09-22Maybe you’ve read the BBC article “how maths teaching is not good enough”? Perhaps you should read the Oftsted report. Perhaps I should.
41% of the maths teaching (in secondary schools) is satisfactory. The tone of the news article is that this is not good enough.
This is characterised by the section headline in the Ofsted report (section 26): «What is not good enough about ‘satisfactory’ teaching?»
I have news for Ofsted. “satisfactory” means almost the same thing as “good enough”. If you’re not satisfied with “satisfactory” teaching, then you set your assessment criteria incorrectly. How unsatisfactory.
My PyCon UK talks
2008-09-16My «Embedded Programming with Python» talk was the first Saturday morning slot, and really it was about manipulating hex-files and ’scope dumps with Python:
On Sunday Nick B gave a presentation of Clear Climate Code. I didn’t do much during the presentation; I was the OmniGraffle monkey. Here’s the presentation at the Clear Climate Code site.
