Planet ALUG

January 23, 2019

Mick Morgan

congratulations to BT

I have been running my own mail server now for well over a decade. Whilst the actual physical hardware (or actually VPS system) may have changed once or twice during that time, the underlying software (postfix and dovecot on debian) has not really changed all that much. However, what has changed over the last decade or so, is the expectation that mail systems will be much more robust, better managed, less insecure (no more “open relays”) and harder on spam than had been the case in the early days of wide takeup of email by the public. Ignoring the “free” offerings from the likes of google, microsoft and others, it would arguably be cheaper, and certainly easier, for me to simply pay for an external mail service by one of the many providers out there. It is pretty easy to find companies offering to host personal email for about a tenner or at most twenty pounds a year. Those “solutions” (as providers seem to love to call their products) usually give you decent anti-spam, A/V scanning, POP3S/IMAPS connectivity (or if you really must, a webmail interface) and can usually alias mail to your preferred domain – particularly if you buy a domain name with your email service. But they always have limitations that I don’t like. The most obvious ones are: restrictions on the number of actual email addresses (as opposed to aliases), limited storage (though that is becoming less of a problem these days), and artificial restrictions on attachment sizes. And I’m bloody minded. I like to control my own email. I run my own email service for the same reason I manage my own DNS, run my own webservers, manage my own wordpress installation, run my own XMPP server and VPNs and manage my own domestic local network with assorted servers hanging off it. I like control and I dislike the opportunity outsourced services have for providing third parties access to my data. My personal data.

Besides, a boy needs a hobby.

However, I do occasionally get one or two problems in mail delivery – though usually /to/ my system rather than /from/ my system. For example I still get the occasional spam or cruddy email which gets past my protection mechanisms. Indeed I recently received one of those ridiculous extortion scam emails purporting to come from my own email adddress – more of which later – but this post is about an outbound mail failure from me to a friend of mine with a btinternet.com account.

I routinely correspond by email with a bunch of long standing friends who once lived relatively close together but are now more widely geographically dispersed. The group (or sub groups in some cases) get together on occasion for holidays, outings and meals. For some odd reason, many of those friends of mine have AOL accounts (I know, I know, but try telling them that). In a list of about two dozen regular correspondents, about a quarter of those people use AOL. The majority of the rest use BT, hotmail and gmail with one or two minor providers or work based accounts. On occasion in the past I have had mail to those AOL based accounts refused by AOL on the spurious grounds that my mail looked like spam because it was aimed at about half a dozen separate AOL accounts all at once. Well, that’s what happens when you “reply-all” to a mail list. Sadly AOL never could figure this out. After a while I gave up emailing their postmaster explaining the problem (and it was /their/ problem, identical email to the individual accounts always got through) because I never, ever, received a reply.

But this is about BT, not AOL.

Members of the mail list are shortly to meet for the group’s annual Christmas meal (it is always late, but hey) and one member “volunteered” to arrange the gathering, find a venue, sort menus etc. Said member has a btinternet email account (@btinternet.com) and he circulated a menu seeking choices for the meal. My reply was refused by BT with a “hard” 554 message which was reported to me by my mail system as below:

The mail system

forename.surname@btinternet.com: host mx.bt.lon5.cpcloud.co.uk[65.20.0.49] said:
554 Message rejected on 2019/01/15 15:00:24 GMT, policy (3.2.2.1) – Your
message looks like SPAM or has been reported as SPAM please read
www.bt.com/bulksender (in reply to end of DATA command)

Now this was decidedly odd, because not 10 days beforehand I had happily sent earlier mails to the same address when our volunteer was initially talking about venue and proposed dates for the gathering. Just to be certain I wasn’t at fault, I checked the advice given by BT on their mail site referred to by the bounce message. Now the only thing I do not have set up for my mail server is DKIM signing. Everything else is hunky-dory – Proper “From” address? check. SPF? check. Proper MX records? check. Fixed IP address? check. PTR record? check. Good reputation? check. Not blacklisted? check (mxtoolbox says I’m fine). Furthermore, I never send HTML email (which I abhor as an abominable bastardisation of proper email standards) so did not have any embedded images or other bloody silly links in my mail). So after trying once or twice more later in the day (and failing) I emailed the BT postmaster saying I was having a problem and pointing out that whilst I might not use Domain keys, there seemed to me to be little else wrong with my email. I didn’t expect an answer, but you have to try,

BT responded – and they responded quickly. I sent my notification, with the failure message, to the BT postmaster address timed at 17.16. At 17.23 I received a reply saying:

“Hi,
Can you please send an example of the failing email to [investigation-address]@btinternet.com.
Please do not forward the email as an attachment but resend it.
Please let “postmaster” know when this has been sent so we can check the email’s content and possible reason for thinking it is spam.

Thank You,”

Slightly stunned, I did as requested and a short time later (at half past midnight when I was asleep) I received another email from BT saying:

“Hi,
That email is scoring high as spam so I have reported it to our spam engine provider, I will email you again when I have some news.
Thank you,”

Sure enough, that same morning at 02.50, I received the following good news:

“Hello,

We have made a change that should stop the emails being scored as spam, this change is being rolled-out now so please try again later.

Thanks”

On reading this when I got up that day I resent my email and, sure enough, it got through. Way to go BT! I have never, ever received that kind of rapid response from any ISP anywhere in the world – and I quite often email “abuse@” network addresses when some toerag or particularly persistent ‘bot shows up in my logs trying to do things I don’t like.

However, as much as I would like to believe that BT fixed a problem simply to accomodate my mail system, I actually think that unlikely. Given that mail from my system to @btinternet.com addresses had been working fine up until a few days ago, I think it much more likely that BT mail administrators had made some recent change, perhaps in one of their spam filters, which caused sigificant volumes of inbound mail to be rejected. My email had then simply been caught up in that wider problem and they were receiving queries or complaints from other mail administrators and not just me. Be that as it may, they still responded correctly, and efficiently as they moved to rectify whatever was causing the problem. So, my congratulations, and heartfelt thanks to the BT postmaster team for actually doing the sort of job that postmasters are supposed to, but rarely do properly.

by Mick at January 23, 2019 04:14 PM

January 16, 2019

Daniel Silverstone (Kinnison)

Plans for 2019

At the end of last year I made eight statements about what I wanted to do throughout 2019. I tried to split them semi-evenly between being a better adult human and being a better software community contributor. I have had a few weeks now to settle my thoughts around what they mean and I'd like to take some time to go through the eight and discuss them a little more.

I've been told that doing this reduces the chance of me sticking to the points because simply announcing the points and receiving any kind of positive feedback may stunt my desire to actually achieve the goals. I'm not sure about that though, and I really want my wider friends community to help keep me honest about them all. I've set a reminder for April 7th to review the situation and hopefully be able to report back positively on my progress.


My list of goals was stated in a pair of tweets:

  1. Continue to lose weight and get fit. I'd like to reach 80kg during the year if I can
  2. Begin a couch to 5k and give it my very best
  3. Focus my software work on finishing projects I have already started
  4. Where I join in other projects be a net benefit
  5. Give back to the @rustlang community because I've gained so much from them already
  6. Be better at tidying up
  7. Save up lots of money for renovations
  8. Go on a proper holiday

Weight and fitness

Some of you may be aware already, others may not, that I have been making an effort to shed some of my excess weight over the past six or seven months. I "started" in May of 2018 weighing approximately 141kg and I am, as of this morning, weighing approximately 101kg. Essentially that's a semi-steady rate of 5kg per month, though it has, obviously, been slowing down of late.

In theory, given my height of roughly 178cm I should aim for a weight of around 70kg. I am trying to improve my fitness and to build some muscle and as such I'm aiming long-term for roughly 75kg. My goal for this year is to continue my improvement and to reach and maintain 80kg or better. I think this will make a significant difference to my health and my general wellbeing. I'm already sleeping better on average, and I feel like I have more energy over all. I bought a Garmin Vivoactive 3 and have been using that to track my general health and activity. My resting heart rate has gone down a few BPM over the past six months, and I can see my general improvement in sleep etc over that time too. I bought a Garmin Index Scale to track my weight and body composition, and that is also showing me good values as well as encouraging me to weigh myself every day and to learn how to interpret the results.

I've been managing my weight loss partly by means of a 16:8 intermittent fasting protocol, combined with a steady calorie deficit of around 1000kcal/day. While this sounds pretty drastic, I was horrendously overweight and this was critical to getting my weight to shift quickly. I expect I'll reduce that deficit over the course of the year, hence I'm only aiming for a 20kg drop over a year rather than trying to maintain what could in theory be a drop of 30kg or more.

In addition to the IF/deficit, I have been more active. I bought an e-bike and slowly got going on that over the summer, along with learning to enjoy walks around my local parks and scrubland. Since the weather got bad enough that I didn't want to be out of doors I joined a gym where I have been going regularly since September. Since the end of October I have been doing a very basic strength training routine and my shoulders do seem to be improving for it. I can still barely do a pushup but it's less embarassingly awful than it was.

Given my efforts toward my fitness, my intention this year is to extend that to include a Couch to 5k type effort. Amusingly, Garmin offer a self adjusting "coach" called Garmin Coach which I will likely use to guide me through the process. While I'm not committing to any, maybe I'll get involved in some parkruns this year too. I'm not committing to reach an ability to run 5k because, quite simply, my bad leg may not let me, but I am committing to give it my best. My promise to myself was to start some level of jogging once I hit 100kg, so that's looking likely by the end of this month. Maybe February is when I'll start the c25k stuff in earnest.

Adulting

I have put three items down in this category to get better at this year. One is a big thing for our house. I am, quite simply put, awful at tidying up. I leave all sorts of things lying around and I am messy and lazy. I need to fix this. My short-term goal in this respect is to pick one room of the house where the mess is mostly mine, and learn to keep it tidy before my checkpoint in April. I think I'm likely to choose the Study because it's where others of my activities for this year will centre and it's definitely almost entirely my mess in there. I'm not yet certain how I'll learn to do this, but it has been a long time coming and I really do need to. It's not fair to my husband for me to be this awful all the time.

The second of these points is to explicitly save money for renovations. Last year we had a new bathroom installed and I've been seriously happy about that. We will need to pay that off this year (we have the money, we're just waiting as long as we can to earn the best interest on it first) and then I'll want to be saving up for another spot of renovations. I'd like to have the kitchen and dining room done - new floor, new units and sink in the kitchen, fix up the messy wall in the dining room, have them decorated, etc. I imagine this will take quite a bit of 2019 to save for, but hopefully this time next year I'll be saying that we managed that and it's time for the next part of the house.

Finally I want to take a proper holiday this year. It has been a couple of years since Rob and I went to Seoul for a month, and while that was excellent, it was partly "work from home" and so I'd like to take a holiday which isn't also a conference, or working from home, or anything other than relaxation and seeing of interesting things. This will also require saving for, so I imagine we won't get to do it until mid to late 2019, but I feel like this is part of a general effort I've been making to take care of myself more. The fitness stuff above being physical, but a proper holiday being part of taking better care of my mental health.

Software, Hardware, and all the squishy humans in between

2018 was not a great year for me in terms of getting projects done. I have failed to do almost anything with Gitano and I did not doing well with Debian or other projects I am part of. As such, I'm committing to do better by my projects in 2019.

First, and foremost, I'm pledging to focus my efforts on finishing projects which I've already started. I am very good at thinking "Oh, that sounds fun" and starting something new, leaving old projects by the wayside and not getting them to any state of completion. While software is never entirely "done", I do feel like I should get in-progress projects to a point that others can use them and maybe contribute too.

As such, I'll be making an effort to sort out issues which others have raised in Gitano (though I doubt I'll do much more feature development for it) so that it can be used by NetSurf and so that it doesn't drop out of Debian. Since the next release of Debian is due soon, I will have to pull my finger out and get this done pretty soon.

I have been working, on and off, with Rob on a new point-of-sale for our local pub Ye Olde Vic and I am committing to get it done to a point that we can experiment with using it in the pub by the summer. Also I was working on a way to measure fluid flow through a pipe so that we can correlate the pulled beer with the sales and determine wastage etc. I expect I'll get back to the "beer'o'meter" once the point-of-sale work is in place and usable. I am not going to commit to getting it done this year, but I'd like to make a dent in the remaining work for it.

I have an on-again off-again relationship with some code I wrote quite a while ago when learning Rust. I am speaking of my Yarn implementation called (imaginatively) rsyarn. I'd like to have that project reworked into something which can be used with Cargo and associated tooling nicely so that running cargo test in a Rust project can result in running yarns as well.

There may be other projects which jump into this category over the year, but those listed above are the ones I'm committing to make a difference to my previous lackadaisical approach.

On a more community-minded note, one of my goals is to ensure that I'm always a net benefit to any project I join or work on in 2019. I am very aware that in a lot of cases, I provide short drive-by contributions to projects which can end up costing that project more than I gave them in benefit. I want to stop that behaviour and instead invest more effort into fewer projects so that I always end up a net benefit to the project in question. This may mean spending longer to ensure that an issue I file has enough in it that I may not need to interact with it again until verification of a correct fix is required. It may mean spending time fixing someone elses' issues so that there is the engineering bandwidth for someone else to fix mine. I can't say for sure how this will manifest, beyond being up-front and requesting of any community I decide to take part in, that they tell me if I end up costing more than I'm bringing in benefit.

Rust and the Rust community

I've mentioned Rust above, and this is perhaps the most overlappy of my promises for 2019. I want to give back to the Rust community because over the past few years as I've learned Rust and learned more and more about the community, I've seen how much of a positive effect they've had on my life. Not just because they made learning a new programming langauge so enjoyable, but because of the community's focus on programmers as human beings. The fantastic documentation ethics, and the wonderfully inclusive atmosphere in the community meant that I managed to get going with Rust so much more effectively than with almost any other language I've ever tried to learn since Lua.

I have, since Christmas, been slowly involving myself in the Rust community more and more. I joined one of the various Discord servers and have been learning about how crates.io is managed and I have been contributing to rustup.rs which is the initial software interface most Rust users encounter and forms such an integral part of the experience of the ecosystem that I feel it's somewhere I can make a useful impact.

While I can't say a significant amount more right now, I hope I'll be able to blog more in the future on what I'm up to in the Rust community and how I hope that will benefit others already in, and interested in joining, the fun that is programming in Rust.


In summary, I hope at least some of you will help to keep me honest about my intentions for 2019, and if, in return, I can help you too, please feel free to let me know.

by Daniel Silverstone at January 16, 2019 11:29 AM

January 08, 2019

Jonathan McDowell

MP130 Maple Leaf and Debian

Like any conference one of the nice things about DebConf is the random interesting people you meet. At the DebConf18 conference dinner I had the pleasure of talking to Jia-Bin Huang, from Centrum Embedded Systems. He told me about how he was using Debian as the basis for his MapleBoard (mostly in Chinese, unfortunately) project, which is based on the Allwinner H3 and has thousands of shipped units. The next day I went to take a look at the website (Google Translate proved helpful), which has details of the board including full schematics of the board. I reached out to Jia-Bin to see if he was still at the conference and if he had any boards with him, but he’d already returned to Taipei. He generously offered to send me one and a few weeks later (mostly due to UK customs delays) I had an “Economy Edition” board in my hands.

MapleBoard MP130

Getting up and running was simple; the board came with a pl2303 based USB serial cable, but I found it a little unreliable so ended up using my own cable. The supplied documentation was in Chinese, but the login details were clearly visible - username mpl1, password 1234. After logging in I found it was running a lightly customized Debian Stretch variant, with the following packages not from the main Debian repository:

base-files              9.9+mp4     Maple base system miscellaneous files
debian-archive-keyring  2017.7+mp1  GnuPG archive keys of the Debian archive
distro-info-data        0.36+mp1    information about the distributions' releases (data files)
linux-image-4.15.2a…    4.15.2a…    Linux kernel, version 4.15.2a-02769-g6d0ce60c8d56
maplewebde              0.1~rc4-2   Web interface to communicate with mapleboard

maplewebde seems to be a web interface for interacting with the board, but it’s in Chinese so I wasn’t able to make much sense of it. I was more interested in the kernel - how much customisation had been done, and how much was already mainlined. Happily the Linux sunxi folk have done a lot of great work in getting things upstream, so although the supplied kernel was using its own drivers (largely branded Mapleboard rather than Allwinner) all the necessary pieces were in mainline. I did a little bit of cleanup of the supplied device tree configuration file, and am pleased to say that as of 5.0-rc1 a multi_v7_defconfig will build a kernel and a sun8i-h3-mapleboard-mp130.dtb file which Just Work™ on the device.

What about the board itself? I haven’t thrown a lot at it, but it feels reasonably responsive compared to some other ARM boards I’ve played with. Quad 1GHz cores and 1GB RAM is a nice enough base spec, though it is ARMv7 so 32-bit only. Unfortunately the “Economy Edition” doesn’t have HDMI on board or I’d have seen about trying to use it as a media player - the video decoding engine apparently has Free drivers based on a vdpau backend, which is pretty cool. There’s no SATA, so it can’t be pressed into service as a build machine easily. I suspect in the long run I’ll press it into service as part of my home automation setup, somewhere I need more oompf than an ESP8266 but still want low power consumption - there are a number of GPIOs conveniently brought out to a 40 pin header.

In general terms of the target market my understanding is the board is largely pitched as a development board, with Centrum being prepared to do customisations for sufficiently sized runs. The open nature of the board, and availability of good support upstream (even if it’s come from the community rather than Allwinner) certainly makes it an attractive option if you’re looking for a 32-bit ARM platform, and although I’m not aware of any option other than ordering from Taiwan at present I’ve found Jia-Bin to be helpful with any queries I’ve had. Also I quite like supporting companies that are using Debian. :) Worth a look if you’re in the market for such a thing.

January 08, 2019 07:07 PM

January 05, 2019

Chris Lamb

Favourite books of 2018

I managed to read 53 books 2018 (up from fifty in 2017) but here follows eleven of my favourite, in no particular order.

Disappointments this year included Attack of the 50 Foot Blockchain: I am finding snark and sarcasm to be subject to severe diminishing returns these days, so whilst entertaining at first it got a little too much too fast. I was not altogether surprised that the author is "a proud editor of RationalWiki" too.

In addition, whilst I really enjoyed The Martian back in 2016 I didn't find Weir's Artemis nearly as compelling. Whilst it was a good enough yarn, everything about the protagonist felt somewhat forced and ultimately hollow. Yuval Noah Harari's 21 Lessons for the 21st Century also did not match up with his previous two but still warrants the investment if you enjoyed them.

The worst book I finished this year was probably Nasssim Nicholas Taleb's Fooled by Randomness. I admit this was a guilty pleasure to some degree; a car crash of arrogance at its finest but ironically quite a compelling read if you can stomach it. However, How to Own the World "bests" it that whilst it delivers some fairly sensible financial advice at first the book finally reveals itself as a tedious encomium to gold about halfway through.


Countdown to Zero Day (2014)

Kim Setter

A genuine thriller or cyberpunk "novel", this book tells the true story behind the virus that sabotaged Iran's nuclear efforts. Not content to focus on Stuxnet itself, it discusses the wider issues with regards to the market for exploits, cyberwarfare and geopolitics.

Although at times it goes into somewhat-unnecessary technical detail on the exploits themselves (".lnk" files, anyone?) this should absolutely not deter recommending it to non-technical folks as these asides are not essential to appreciating this fine book. Indeed, this is absolutely riveting and eye-opening, even for someone who is reasonably up-to-date with security issues.

Highly recommended, I ended gifting this book as a number of Christmas presents.


A Year in Provence (1989)

Peter Mayle

Whilst waxing lyrical to a friend about Kate Fox's Watching the English from my 2017 highlights, they immediately enquired whether I had read any of Peter Mayle's Provence series. Answering in the negative, they explained that it uses the authors's renovation of a house in a small village in France as a way of hanging an amusing socio-anthropological yarn. I ended up binge-reading this in a number of wine bars and bistros in the XVIIIe arrondissement guessing that was as good a set and setting I was going to achieve, especially as that would avoid the dreaded Mistral that is personified as a human actor throughout the tale.

Singularly impressed by the quality of the writing ("… by nine o'clock it was already too hot to wear a watch…") and the author's ability to find the «le mot juste», it is an unalloyed joy to read primarily due to the interactions into the natives:

"What’s your best price?" she asked the dealer. "My best price, Madame, is a hundred francs. However, this now seems unlikely and lunch approaches. You can have it for fifty."

The immediate sequel, Toujours Provence, is already high on my queue for January. Mayle tragically passed away in January 2018, but not before quipping "I've often thought the best time to die would be after a long lunch — just before the bill arrives."


Confessions of a Conjuror (2009)

Derren Brown

Until recently, Derren was somewhat of a UK-centric celebrity magician who essentially redefined the public perception of the genre to modern audiences by foregrounding psychological manipulation and spectacle over mere "tricks".

An autobiography of sorts, Confessions is structured around a single performance from the days when Brown was an unknown magician working the tables in a middlebrow Bristol restaurant, and uses this narrative conceit as a springboard to break into rambling yet highly-revealing tangents into parts of his world and mind.

Clearly highly tuned to social dynamics, Derren offers a fair amount of observational humour too:

The Parmesan Moment: when the most animated chatter enters, sometimes mid-word, a cryogenic phase equal in length to the time it takes the waiter to shave hard cheese on to the plates of the erstwhile vivacious diners. No conversation is too mundane, no babble too banal for it to be suddenly classified as anything less than entirely confidential once the rotary grater invades the periphery.

As you might be able to surmise, one hurdle to really enjoying this book is Brown's use of unnecessarily fancy prose which — like Russell Brand's similar pretentious affections — serves only to keep the reader at a distance. It is refreshing that Brown's later works don't appear to have this trait however and his Happy is very much on my to-read for 2019.

Nevertheless, this is a bizarre, intriguing and (almost entirely…) brilliant insight into the mind of a remarkable artist.


On Tyranny: Lessons from the Twentieth Century (2017)

Timothy Synder

I first discovered Synder many years ago through his harrowing Bloodlands which describes the Nazi and Soviet killing fields of the Black Sea and the Baltic Republics where both parties were complicit in such atrocities that are so huge and so awful that grief could almost grow numb. However, this year he popped up on an episode of the Sam Harris podcast to promote his Lessons from the Twentieth Century.

This book comprises of a number of short chapters with titles like "Remember Professional Ethics" and "Beware the One-Party State", each purporting to illustrate some angle of the 20th-century to readers in the 21st. At only 128 pages, this slender and easy-to-read volume was engaging enough to enjoyed over a the course of a single beverage.

I am now straining to elucitate exactly why I liked this as a whole but in hindsight it seemed to hit home at the right time and was motivational in terms of re-affirming confidence in ones established beliefs. It certainly makes some mordant criticisms of our approaches to current world events, including remarking that whilst our generic cynicism makes us feel alternative, given this is what everyone else is doing we are actually part of a morass of indifference. The positive (but "adolescent") connotations of the doctrine of disruption are also given a knock with the observation that:

The man who runs naked across a football field disrupts, but he does not change the rules of the game.

… and for those with more of a penchant for privacy-related topics, Synder reminds us that totalitarianism is not necessarily the clichéd all-powerful state but rather the erasure of the difference between the private and public life: we are free only insofar as we exercise control over what people know about us and in what circumstances they come to know it.


Why We Sleep: The New Science of Sleep and Dreams (2017)

Matthew Walker

We remain shockingly ignorant of how we spend at least a third of our lives and how much it affects the other two-thirds. But perhaps more worrying are the severe physical and mental health considerations of foregoing sleep as well as the degree a deficiency prevents us from perceiving said negative effects in a kind of bizarre "Dunning—Morpheus" effect.

Sleep (or rather; the "science of sleep") was definitely a meme of 2018 popular science and garnered a lot of attention in the podcast world — so what stood out about this particular contribution?

Indeed, in terms of specific advice there nothing here you haven't come across before (regular schedule, no screens, cooler room, avoid sleeping pills…) this book rises above the rest in that it isn't a step-by-step manual (isn#t "advice is what we ask for when we already know the answer but wish we didn't…" anyway?). In contrast, Walker foregrounds explanations about dreams, REM & NREM sleep, the evolution of sleep, jetlag, the history of sleep as well as the ever-changing relationship between society and the act of sleeping. There is unfortunately not enough causal data on a population level at the moment to make definitive statements, but enough highly correlative stuff and thus ironically ripe for the pop science treatment.


Zen and the Art of Motorcycle Maintenance (1974)

Robert Pirsig

After at least a decade, I finally got around to reading this. I am not sure why I had avoided it up until now, perhaps worrying such a "hypercanonical" book in this space would come across as highly-derivative given that I've read so many books that occupy the same space or have otherwise taken it as inspiration.

However, it was probably the acquisition of an actual motorbike this year that prompted an ironic purchase (along with the associated Haynes manual) and was quickly rewarded by its take on the philosophy of science and other prosaic or romantic thoughts.

Reviewing such a book in any detail in late-2018 seems a little odd (do we need another "review" of GEB on Hacker News?) so I will only add that I not found myself associating my thoughts on maintenance closer to the Sutherlands than our protagonist and my copy is now irredeemably littered with highlighted quotations for which it is impossible to find a favourite. However, he's one, perhaps, suitable for the upcoming year:

You are never dedicated to something you have complete confidence in. When people are fanatically dedicated to political or religious faiths or any other kinds of dogmas or goals, it's always because these dogmas or goals are in doubt.


The Dig (2007)

John Preston

This novel dramatises the events behind the discovery of the Sutton Hoo treasure in 1939 which included a "ship burial" and a wealth of undisturbed Anglo-Saxon artefacts. Eerily similar to when I was reading the author's A Very English Scandal, I started out not aware it was based on a true story but some very slightly incongruous or unnecessary facts encouraged me look up the background online.

A quick, short and enjoyable read, recommended to anyone interested in history or a portrait of antebellum England.


The Long Way to a Small, Angry Planet (2014)

Becky Chambers

This is one of those books whose appeal and interest is curiously in its flaws. Or: if this was a better book it would be curiously less compelling to recommend. To get it out of the way up-front, Chambers clearly has a particular target audience in mind and this regrettably means a certain amount of pandering, wish-fulfillment and compromises on behalf of the art of the novel.

For example, one discovers that in the Android is actually black and it seemed clear to me you weren't really meant to notice and thus raise an eyebrow at one's own prejudice when it is faux-casually revealed to you. There, of course, is nothing really wrong with these sorts of games — or perhaps the book's overt use of non-standard pronouns — but this sort of oft-laboured detail ends up simply tripping up the (good!) core narrative rather than offering delightful background colour, at least violating the principle of Chechov's Gun and getting in the way of the plot; these social elements are invariably not "world-building" as it is in, say, episodic and early Star Trek or Stargate is.

Despite all of the above, I would still highly recommend this to anyone remotely-interested in modern sci-fi; indeed I found its loosely-associated followup almost as compelling and the third installment on my 2019 list once I can stomach the cheeky "get 'em hooked" drug dealer pricing strategy of the trilogy.


The Road to Wigan Pier (1937)

George Orwell

Defying strict classification, this book is split into two quite distinct parts; the first discusses the living conditions among the working class in Yorkshire whilst the second half is a long and rambling essay on a myriad of subjects including socialism, politics, his middle-class upbringing.

A huge fan of Orwell, I also read his Burmese Days (1934) too, finally finishing my entire journey through his oeuvre. However, unlike his other works, Orwell uncharacteristically comes across a bit cuckoo in this second part:

It would help enormously if the smell of crankishness which still clings to the socialist movement could be dispelled. If only the sandals and the pistachio-coloured shirts could be put in a pile and burnt, and every vegetarian, teetotaller, and creeping Jesus sent home to Welwyn Garden City to do his yoga exercises quietly!

Curiously, this second part was almost entirely cut by an original editor. It is not, however, without a bit of humour or even entirely unrelatable:

I am a degenerate modern semi-intellectual who would die if I did not get my early morning cup of tea and my "New Statesman" every Friday.

For fans of 1984 or Animal Farm I could not unreservedly recommend this but if you have enjoyed any of his splendid essays then it is definitely worth checking out.


How to Change Your Mind: The New Science of Psychedelics (2018)

It appears that widespread adoption of psychedelic drugs, at least for therapeutic purposes, always appears to be just another year away, but 2018 definitely represented a convergence of literature on the topic.

Pollan's work stands above the rest with its compelling explanation of the substances' storied histories, the author's own personal experiences with it whilst weaving in the neuroscience without putting the reader at distance.

As somewhat of a paean to such "cures", I would find myself unable to recommended this unreservedly to all and sundry, but most will find the combination of science, spirituality (from the perspective of a skeptic) and narrative adds up to something far more than the sum of its parts.


American Kingpin (2017)

Nick Bilton

Subtitled "Catching the Billion-Dollar Baron of the Dark Web", this gripping tale tells the story of Ross Ulbricht, better known as the owner/operator of the Silk Road online black market. It describes the background of the creation of the site, the fascinating and at-times completely immoral & illegal activities of the law enforcement sent after him, all the way through to his arrest and subsequent trial.

Like Countdown to Zero Day reviewed above, you probably couldn't make a believable film about episode in our history without requiring something on the scope and quality of 2010's The Social Network. Eerily reminiscent or suggestive of the film itself, this book is perhaps at its best when critically dissecting Russ' personality, describing the bizarre antics happening and getting somewhat weaker as it moves into the more-humdrum court proceedings.

Regardless, neither fans nor detractors of cryptocurrencies or the ethics of online black markets should be deterred from checking out this superb work.

January 05, 2019 08:50 AM

December 31, 2018

Chris Lamb

Free software activities in December 2018

Here is my monthly update covering what I have been doing in the free software world during December 2018 (previous month):


Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws almost all software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

This month I:


I also made the following changes to our tooling:


diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.


strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.


Debian


Patches contributed


Debian LTS

This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.


Uploads


Debian bugs filed

I also filed bugs against packages that use vendor-specific patch series files for deluge, fail2ban, filezilla, hexchat, libfreenect, libxfce4util, liferea, mate-power-manager, mate-terminal, mixxx, numix-gtk-theme, packagekit, smuxi, xchat & xfce4-smartbookmark-plugin.


FTP Team


As a Debian FTP assistant I ACCEPTed 141 packages: ansible, bambootracker, birdtray, bitlbee-mastodon, blis, capnproto, centreon-broker, chargebee-python, chargebee2-python, dar, darknet, dask-sphinx-theme, dav4tbsync, davs2, displaycal, django-anymail, dsmidiwifi, eas4tbsync, emerald, emerald-themes, erlang-horse, fusion-icon, ghostwriter, gitlab, go-cpe-dictionary, go-exploitdb, golang-1.12, golang-github-datadog-zstd, golang-github-justinas-alice, golang-github-namsral-flag, google-compute-image-packages, grim, grpc, haskell-gi-atk, haskell-gi-cairo, haskell-gi-dbusmenu, haskell-gi-dbusmenugtk3, haskell-gi-gdk, haskell-gi-gdkpixbuf, haskell-gi-gdkx11, haskell-gi-gio, haskell-gi-glib, haskell-gi-gobject, haskell-gi-gtk, haskell-gi-gtk-hs, haskell-gi-pango, haskell-gi-vte, haskell-gi-xlib, haskell-gtk-sni-tray, haskell-gtk-strut, haskell-status-notifier-item, haskell-system-posix-redirect, haskell-termonad, haskell-xml-html-qq, i3pystatus, jaxb, lablgtk3, libcloudflare-client-perl, libconfig-model-backend-yaml-perl, libcpan-common-index-perl, libhostfile-manager-perl, libhttp-tinyish-perl, libjs-jquery-center, libjs-jquery-markitup, libmenlo-legacy-perl, libmenlo-perl, libmoox-locale-passthrough-perl, libnewlib-nano, libnss-unknown, liborcus, libparse-binary-perl, librtr, libsearch-elasticsearch-client-1-0-perl, libsearch-elasticsearch-client-2-0-perl, libtie-handle-offset-perl, libzstd, lvm2, matplotlib2, med-fichier, meep, meep-lam4, meep-mpi-default, meep-mpich2, meep-openmpi, mir-core, mle, movim, netplan.io, node-lunr, node-ramda, node-react-audio-player, nodejs, oakleaf, olive, openrazer, puppet-module-heini-wait-for, puppet-module-octavia, puppet-module-voxpupuli-ssh-keygen, pylibtiff, pymilter, pyspectral, python-cytoolz, python-dpkt, python-envs, python-flask-cors, python-geotiepoints, python-glad, python-hgapi, python-ifaddr, python-internetarchive, python-markdown2, python-msgpack-numpy, python-netdisco, python-pipx, python-project-generator, python-project-generator-definitions, python-pywebview, python-sparkpost, python-sshoot, python-thinc, python-tornado4, pytroll-schedule, rcm, redberry-pipe, ruby-kitchen-salt, ruby-vcr, rust-crossbeam-channel, rust-crossbeam-utils-0.5, rust-ena, rust-hyphenation, slurp, theme-d-gnome, ticcutils, trollimage, trollsift, ulfius, vim-puppet, vland, voluptuous-serialize, vulkan-tools & xavs2.

I additionally filed 11 RC bugs against packages that had potentially-incomplete debian/copyright files against centreon-broker, dav4tbsync, eas4tbsync, emerald, i3pystatus, lvm2, olive, python-pywebview, ruby-kitchen-salt, rust-crossbeam-channel & trollsift.

December 31, 2018 06:06 PM

Jonathan McDowell

Maxcio W-UK007S Power Monitoring Smart Plug notes

The house server I built in 2013 is getting on a bit, so I’d like to replace it. That’s currently held up on availability of Ryzen 7 2700E CPUs, which look to be the best power consumption/performance trade-off available at present. While I wait for stock I figured I should see how the current i3-3220T is doing.

To do so I decided to buy a Smart Plug that advertised energy monitoring, planning to integrate it into my current setup for the monitoring and then being able to use it for general power control once the upgrade comparison is complete. I ended up with a pair of Maxcio Smart Plugs - pricing and availability worked out and I’d found confirmation that the W-US002S was ESP8266 based.

The model I ended up with is externally marked as a W-UK007S. It’s a fairly neat form factor (slightly smaller than the SonOff S26 I already have, which doesn’t do power monitoring). It also turned out to be easy to take apart; there is a circular cover in the middle which can be carefully popped out, revealing the single screw holding the device together.

Front of Maxcio Smart Plug

The back plate has 4 clips holding it together at the corners and can be gently pried off. Inside there’s a main circuit board labelled “W-US0007S-V0.3” which has the relay on it and a perpendicular board with the ESP8266 module and power monitoring chip on it.

Inside of Maxcio Smart Plug

Sadly the layout didn’t match anything I was familiar with, or could find any details of. That meant I had to do some detective work to figure out how to talk to the ESP8266. It was easy enough to work out GND + VCC by following PCB tracks. Likewise the relay, the button and the LED (underneath the button, and separately controlled from the relay, unlike the S26). Next move was to hook up power (just a low voltage supply to GND/VCC, I did not engage in any experimentation involving mains voltages!) and monitor each unknown pin in turn in the hope I’d find TX (even if the supplied firmware didn’t print anything out the ESP8266 prints a message on boot, so I’d definitely see something if it was there).

Thankfully TX was brought out to the module connection to the main PCB, so I had something I could monitor.

Maxcio boot log
 ets Jan  8 2013,rst cause:1, boot mode:(3,7)

load 0x40100000, len 1396, room 16 
tail 4
chksum 0x89
load 0x3ffe8000, len 776, room 4 
tail 4
chksum 0xe8
load 0x3ffe8308, len 540, room 4 
tail 8
chksum 0xc0
csum 0xc0

2nd boot version : 1.4(b1)
  SPI Speed      : 40MHz
  SPI Mode       : QIO
  SPI Flash Size & Map: 8Mbit(512KB+512KB)
jump to run user1 @ 1000

OS SDK ver: 1.4.2(23fbe10) compiled @ Sep 22 2016 13:09:03
phy v[notice]user_main.c:268 SDK version:1.4.2(23fbe10)
[notice]user_main.c:270 tuya sdk version:1.0.6
[notice]user_main.c:275 tuya sdk compiled at Jul 26 2017 15:27:36
[notice]user_main.c:277 BV:5.20 PV:2.1 LPV:3.1
reset reason: 0
epc1=0x00000000, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000,depc=0x00000000
mode : softAP(de:4f:22:2c:76:93)
dhcp server start:(ip:192.168.4.1,mask:255.255.255.0,gw:192.168.4.1)
add if1
bcn 100
bcn 0
del if1
usl
mode : sta(dc:4f:22:2c:76:93)
add if0
scandone

[notice]device.c:1694 fireware info name:esp_hys_qx_single_light_jlplug version:1.0.0
[notice]device.c:1695 PID=gDYvLrWhRoqpgHMj

[notice]device.c:1696 FW Compiled date:Feb  5 2018
[notice]gw_intf.c:240 Authorization success
[notice]device.c:1722 ##  AUTH DONE ##

[err]bl0937.c:1013 ### get_coefficient get err: 28
[err]device.c:1767 get ele data err...
[err]device.c:1772 get tem ele data err...
del if0
usl
mode : null
force slp enable,type: 2
fpm open,type:2 0
[notice]device.c:2041  # STAT_STA_UNCONN STAT_LOW_POWER#

[notice]device.c:2042  # STAT_STA_CONN #

[notice]device.c:648 #####I_VAL:0#####

[notice]device.c:654 ***************************Check Start 1**********************************
[notice]device.c:655 ### cur:0 mA power:0 W vol:0 V
[notice]device.c:656 ****************************Check Stop**********************************

Next I took the remaining unknown pins and tried grounding them on boot, in an attempt to find GPIO0 (which needs to be grounded to access the ROM serial bootloader). I ended up finding GPIO2 first, and then eventually figuring out the LED was using GPIO0 - learning the lesson not to assume pins don’t have multiple uses. Now I had TX + GPIO0 I could hold GPIO0 on boot and look for RX by probing the remaining pins and seeing if esptool could talk to the bootloader. Again, I was successful.

At that point I was able to download the firmware from flash, and poke it in the hope of working out the GPIO assignments (I’m a software guy, I’m happier with an assembly dump than probing randomly around a board in the hope of enlightenment). I generated a crude .elf from the flash dump using esp-bin2elf, hacking it up to cope better with an OTA ROM image and skip the boot loader. I initially used objdump to examine the result, which wasn’t that helpful, and then found ScratchABit, which made things much easier. What would have been ideal is some way to load in the .a static libraries from the SDK and automatically map those to the code; as well as providing some useful symbols it would have avoided work looking at functions that were irrelevant. The ESP8266 seems to want various levels of indirection to access functions and memory locations so it’s not just a simple matter of looking for an I/O request to a specific location, but I was able to figure out that the button was on GPIO13 and the relay on GPIO15.

All that left was the bit I was actually interested in - the power monitoring. The appropriate chip (clearly attached to a low resistance bridge from one of the AC power pins, and also attached to the other pin) on the PCB was marked “HJL-01 / J1749CYH / D797480E”. Whatever you find on the web this is not the same as the HLW8012. It’s very similar in operation but is actually a BL0937. Electrodragon’s Energy Meter page was the best resource I found - it has a link to the Chinese datasheet for the BL0937, which in combination with Google Translate allows the basic principles to be understood. Both devices work by having a pin (CF) which outputs pulses based on monitored power consumption, and a pin (CF1) which can be switched between monitoring current and voltage via a third pin (SEL). For the BL0937 you can just count pulses; the pulse width is fixed at 38µS and it’s just the frequency which varies. I’d found the GPIO interupt handler in my flash disassembly which indicated that GPIO5 was connected to CF and GPIO14 to CF1. Additionally the handler around GPIO14 needs to check which mode the chip is currently in, which let me discover GPIO12 was connected to this pin.

That resulted in the following pin mapping of the daughter PCB; the remaining 4 pins weren’t of interest once I had the ones I needed, so I didn’t do further investigation:

PCB EDGE

                +----+
LED    / GPIO0  |1 12| 3.3V
                |2 11|
BUTTON / GPIO13 |3 10| GPIO2
TX              |4  9| RX
RELAY  / GPIO15 |5  8|
GND             |6  7|
                +----+

PCB CENTER

Of course the frequency values that come out of the BL0937 are not directly usable; there’s a certain amount of conversion/calibration that needs to be done. Thankfully although the datasheet has an equation that includes an odd constant value as well as the internal reference voltage this all boils down to a simple scaling value. I ended up using a multimeter to calibrate the voltage and then a standalone power meter + table lamp to calibrate power/current. Better results could be obtained with a constant voltage source and a known resistance load but this worked out close enough for my needs.

I wrote my own driver to fit within the ESP8266 MQTT framework I use, but if you’re looking for something more off the shelf ESPurna is probably a good start. Ticket #737 talks about adding support for the BL0937 (it’s close enough to the HLW8012 that you can get away with just changing the scaling factors), and the Homecube SP1 seems to use the same GPIOs for the various pieces of hardware.

I’ve put all the images from my teardown into a W-UK007S Teardown Album on Google Photos, in case it’s useful to anyone.

December 31, 2018 05:57 PM

December 24, 2018

Mick Morgan

always keep the address

I normally post a “happy birthday trivia” message at this time of year. Indeed I have been doing this for 12 years now. Of late my posting has become less frequent which is somewhat odd since I now have much more free time than I had back when I started trivia. But no matter – some things are much more important than blogging.

This year I was struck by a BBC article by the poet Ian McMillan which I read yesterday. The article recalls how McMillan briefly met a chap called “Brian” at Jersey airport on a breezy night in autumn many years ago. McMillan was apparently very worried about the impending flight but was reassured by Brian that all would be well. After chatting for a short while and just before boarding the flight, Brian and McMillan swapped addresses and said that they would stay in touch. Unfortunately McMillan then lost Brian’s address. But Brian obviously did not lose McMillan’s address because each Christmas thereafter he sent a card, despite receiving nothing back.

The article ends with McMillan saying:

“Always keep the address. Always remember where people are, and then you can translate those moments of the kindness of strangers into a winter scene and a first class stamp. “

I’d say that was good advice.

Merry Christmas all.

by Mick at December 24, 2018 03:41 PM

November 08, 2018

Steve Engledow (stilvoid)

git-get

Because I work on a lot of different projects spread across a lot of accounts at multiple git hosting providers, I try to keep my code folder in some semblance of order by having subfolders for things.

A while ago, I decided to make things even simpler by letting the git repos I was cloning dictate where they should live. I took inspiration from the way go expects you to organise your code.

Today, I decided to apply the three virtues and wrote some code to handle this for me.

Introducing git-get

git-get is an opinionated git command that helps you keep your code folder in order.

You use git-get as a replacement for git clone and it will decide where your code should live :)

git get https://github.com/stilvoid/git-get
Cloning into '/home/steve/code/github.com/stilvoid/git-get'...

Laziness is the primary virtue.

by Steve Engledow at November 08, 2018 12:00 AM

August 14, 2018

Steve Engledow (stilvoid)

Heroes: Building some old code

For the end result of this post, see my AUR package of Heroes.


The other day, something reminded me of a game I used to really enjoy playing back in my early days of getting to know Linux. That game was Heroes. It’s a clone of Snake/Tron/Nibbles but with some fun additions, a nice graphical style, and some funky visual effects.

Heroes screenshot

So, of course, I immediately decided to install it.

$ pacman -Ss heroes

No results. Nothing in the AUR either. There is only one other course of action: I’m going to create an AUR package for it!

It looks like the last change to the game was 16 years ago so it could be fun getting it to compile with a modern toolchain.

Getting Heroes to compile in 2018

I put together a basic PKGBUILD that pulls down the source and data files from the Heroes sourceforge page and then runs:

./configure
make

Here’s the first of what I’m sure are many failure messages:

hedlite.c:48:20: error: static declaration of ‘tile_set_img’ follows non-static declaration 
 static a_pcx_image tile_set_img;
                    ^~~~~~~~~~~~
In file included from hedlite.c:44:
const.h:52:20: note: previous declaration of ‘tile_set_img’ was here                        
 extern a_pcx_image tile_set_img, font_deck_img;                                            
                    ^~~~~~~~~~~~

Some forewarning: it’s been quite some time since I wrote anything serious in C and I was never an expert in it anyway. But I think I know enough to fix this and so just commented out the static declaration as, after poking around in the code a bit, it doesn’t seem like it’s necessary anyway.

Now the compilation succeeds but I get the following error during linking:

/usr/bin/ld: camera.o: undefined reference to symbol 'sin@@GLIBC_2.2.5'
/usr/bin/ld: /usr/lib/libm.so.6: error adding symbols: DSO missing from command line

Turns out that for some reason, the developers forgot to include the math(s) library. I’m guessing that perhaps it used to be linked by default in a previous version of GCC.

LDFLAGS=-lm ./configure
make

Now it at least compiles correctly! Next up, compiling the data, music, and sound effects packages.

Amazingly, those all worked correctly and I was able to play the game!

However, this game was written a while ago and originally targeted MS-DOS so it has a window size of 320x200 which looks rather ridiculous on my 1920x1080 desktop ;)

Tiny Heroes window screenshot

So I set about trying to set the default screen mode so that the game starts in full screen…

Fortunately, it looks like this is relatively easy. I just modified a few variables and changed a command line flag from -F | --full-screen to -W | --windowed.

Next up, rather than rely on SDL’s built-in scaling (it looks blurry and weird), I need to enable Heroes’ quadruple flag -4 by default. In fact, I removed all the scaling options and just left it to default to scaling 4-fold as that leaves the game with a resolution of 1280x800 which seems a reasonable default these days. I’m sure I’ll receive bug reports if it’s not ;)

The very last thing I’ve done is to enable the high quality mixer by default and remove the command line option from the game. CPU is a little more abundant now than it was in 2002 ;)

Here’s my final patch file.

Submitting the AUR package

Things have changed since I last submitted a package to the AUR so here’s a brief writeup - if only to remind myself in future ;)

First step was to update the SSH key in my AUR account as it contained a key from my old machine.

Next up, I added a remote to my repository:

$ git remote add aur ssh://aur@aur.archlinux.org/heroes.git
$ git fetch aur  # This step causes AUR to create a record for the package

The next step is to generate AUR’s .SRCINFO file and rebase it into every commit (AUR requires this).

$ git filter-branch --tree-filter "makepkg --printsrcinfo > .SRCINFO"

And then push it to the AUR repository:

$ git push -u aur master

Testing it out

I use packer to make using AUR easier (I’m lazy).

$ packer -S heroes

SUCCESS!

All in all, this wasn’t anywhere near as painful as I’d expected. Time to play some Heroes :D

by Steve Engledow at August 14, 2018 12:00 AM

July 03, 2018

Daniel Silverstone (Kinnison)

Docker Compose

I glanced back over my shoulder to see the Director approaching. Zhe stood next to me, watched me intently for a few moments, before turning and looking out at the scape. The water was preturnaturally calm, above it only clear blue. A number of dark, almost formless, shapes were slowly moving back and forth beneath the surface.

"Is everything in readiness?" zhe queried, sounding both impatient and resigned at the same time. "And will it work?" zhe added. My predecessor, and zir predecessor before zem, had attempted to reach the same goal now set for myself.

"I believe so" I responded, sounding perhaps slightly more confident than I felt. "All the preparations have been made, everything is in accordance with what has been written". The director nodded, zir face pinched, with worry writ across it.

I closed my eyes, took a deep breath, opened them, raised my hand and focussed on the scape, until it seemed to me that my hand was almost floating on the water. With all of my strength of will I formed the incantation, repeating it over and over in my mind until I was sure that I was ready. I released it into the scape and dropped my arm.

The water began to churn, the blue above darkening rapidly, becoming streaked with grey. The shapes beneath the water picked up speed and started to grow, before resolving to what appeared to be stylised Earth whales. Huge arcs of electricity speared the water, a screaming, crashing, wall of sound rolled over us as we watched, a foundation rose up from the depths on the backs of the whale-like shapes wherever the lightning struck.

Chunks of goodness-knows-what rained down from the grey streaked morass, thumping into place seamlessly onto the foundations, slowly building what I had envisioned. I started to allow myself to feel hope, things were going well, each tower of the final solution was taking form, becoming the slick and clean visions of function which I had painstakingly selected from among the masses of clamoring options.

Now and then, the whale-like shapes would surface momentarily near one of the towers, stringing connections like bunting across the water, until the final design was achieved. My shoulders tightened and I raised my hand once more. As I did so, the waters settled, the grey bled out from the blue, and the scape became calm and the towers shone, each in its place, each looking exactly as it should.

Chanting the second incantation under my breath, over and over, until it seemed seared into my very bones, I released it into the scape and watched it flow over the towers, each one ringing out as the command reached it, until all the towers sang, producing a resonant and consonant chord which rose of its own accord, seeming to summon creatures from the very waters in which the towers stood.

The creatures approached the towers, reached up as one, touched the doors, and screamed in horror as their arms caught aflame. In moments each and every creature was reduced to ashes, somehow fundamentally unable to make use of the incredible forms I had wrought. The Director sighed heavily, turned, and made to leave. The towers I had sweated over the design of for months stood proud, beautiful, worthless.

I also turned, made my way out of the realisation suite, and with a sigh hit the scape-purge button on the outer wall. It was over. The grand design was flawed. Nothing I created in this manner would be likely to work in the scape and so the most important moment of my life was lost to ruin, just as my predecessor, and zir predecessor before zem.

Returning to my chambers, I snatched up the book from my workbench. The whale-like creature winking to me from the cover, grinning, as though it knew what I had tried to do and relished my failure. I cast it into the waste chute and went back to my drafting table to design new towers, towers which might be compatible with the creatures which were needed to inhabit them and breath life into their very structure, towers which would involve no grinning whales.

by Daniel Silverstone at July 03, 2018 03:13 PM

June 07, 2018

Brett Parker (iDunno)

The Psion Gemini

So, I backed the Gemini and received my shiny new device just a few months after they said that it'd ship, not bad for an indiegogo project! Out of the box, I flashed it, using the non-approved linux flashing tool at that time, and failed to backup the parts that, err, I really didn't want blatted... So within hours I had a new phone that I, err, couldn't make calls on, which was marginally annoying. And the tech preview of Debian wasn't really worth it, as it was fairly much unusable (which was marginally upsetting, but hey) - after a few more hours / days of playing around I got the IMEI number back in to the Gemini and put back on the stock android image. I didn't at this point have working bluetooth or wifi, which was a bit of a pain too, turns out the mac addresses for those are also stored in the nvram (doh!), that's now mostly working through a bit of collaboration with another Gemini owner, my Gemini currently uses the mac addresses from his device... which I'll need to fix in the next month or so, else we'll have a mac address collision, probably.

Overall, it's not a bad machine, the keyboard isn't quite as good as I was hoping for, the phone functionality is not bad once you're on a call, but not great until you're on a call, and I certainly wouldn't use it to replace the Samsung Galaxy S7 Edge that I currently use as my full time phone. It is however really rather useful as a sysadmin tool when you don't want to be lugging a full laptop around with you, the keyboard is better than using the on screen keyboard on the phone, the ssh client is "good enough" to get to what I need, and the terminal font isn't bad. I look forward to seeing where it goes, I'm happy to have been an early backer, as I don't think I'd pay the current retail price for one.

by Brett Parker (iDunno@sommitrealweird.co.uk) at June 07, 2018 01:04 PM

February 21, 2018

MJ Ray

How hard can typing æ, ø and å be?

Petter Reinholdtsen: How hard can æ, ø and å be? comments on the rubbish state of till printers and their mishandling of foreign characters.

Last week, I was trying to type an email, on a tablet, in Dutch. The tablet was running something close to Android and I was using a Bluetooth keyboard, which seemed to be configured correctly for my location in England.

Dutch doesn’t even have many accents. I wanted an e acute (é). If you use the on screen keyboard, this is actually pretty easy, just press and hold e and slide to choose the accented one… but holding e on a Bluetooth keyboard? eeeeeeeeeee!

Some guides suggest Alt and e, then e. Apparently that works, but not on keyboards set to Great British… because, I guess, we don’t want any of that foreign muck since the Brexit vote, or something(!)

Even once you figure out that madness and switch the keyboard back to international, which also enables alt i, u, n and so on to do other accents, I can’t find grave, check, breve or several other accents. I managed to send the emails in Dutch but I’d struggle with various other languages.

Have I missed a trick or what are the Android developers thinking? Why isn’t there a Compose key by default? Is there any way to get one?

by mjr at February 21, 2018 04:14 PM

March 01, 2017

Brett Parker (iDunno)

Using the Mythic Beasts IPv4 -> IPv6 Proxy for Websites on a v6 only Pi and getting the right REMOTE_ADDR

So, more because I was intrigued than anything else, I've got a pi3 from Mythic Beasts, they're supplied with IPv6 only connectivity and the file storage is NFS over a private v4 network. The proxy will happily redirect requests to either http or https to the Pi, but this results (without turning on the Proxy Protocol) with getting remote addresses in your logs of the proxy servers, which is not entirely useful.

I've cheated a bit, because the turning on of ProxyProtocol for the hostedpi.com addresses is currently not exposed to customers (it's on the list!), to do it without access to Mythic's backends use your own domainname (I've also got https://pi3.sommitrealweird.co.uk/ mapped to this Pi).

So, first step first, we get our RPi and we make sure that we can login to it via ssh (I'm nearly always on a v6 connection anyways, so this was a simple case of sshing to the v6 address of the Pi). I then installed haproxy and apache2 on the Pi and went about configuring them, with apache2 I changed it to listen to localhost only and on ports 8080 and 4443, I hadn't at this point enabled the ssl module so, really, the change for 4443 didn't kick in. Here's my /etc/apache2/ports.conf file:

# If you just change the port or add more ports here, you will likely also
# have to change the VirtualHost statement in
# /etc/apache2/sites-enabled/000-default.conf

Listen [::1]:8080

<IfModule ssl_module>
       Listen [::1]:4443
</IfModule>

<IfModule mod_gnutls.c>
       Listen [::1]:4443
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

I then edited /etc/apache2/sites-available/000-default.conf to change the VirtualHost line to [::1]:8080.

So, with that in place, now we deploy haproxy infront of it, the basic /etc/haproxy/haproxy.cfg config is:

global
       log /dev/log    local0
       log /dev/log    local1 notice
       chroot /var/lib/haproxy
       stats socket /run/haproxy/admin.sock mode 660 level admin
       stats timeout 30s
       user haproxy
       group haproxy
       daemon

       # Default SSL material locations
       ca-base /etc/ssl/certs
       crt-base /etc/ssl/private

       # Default ciphers to use on SSL-enabled listening sockets.
       # For more information, see ciphers(1SSL). This list is from:
       #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
       ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
       ssl-default-bind-options no-sslv3

defaults
       log     global
       mode    http
       option  httplog
       option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
       errorfile 400 /etc/haproxy/errors/400.http
       errorfile 403 /etc/haproxy/errors/403.http
       errorfile 408 /etc/haproxy/errors/408.http
       errorfile 500 /etc/haproxy/errors/500.http
       errorfile 502 /etc/haproxy/errors/502.http
       errorfile 503 /etc/haproxy/errors/503.http
       errorfile 504 /etc/haproxy/errors/504.http

frontend any_http
        option httplog
        option forwardfor

        acl is_from_proxy src 2a00:1098:0:82:1000:3b:1:1 2a00:1098:0:80:1000:3b:1:1
        tcp-request connection expect-proxy layer4 if is_from_proxy

        bind :::80
        default_backend any_http

backend any_http
        server apache2 ::1:8080

Obviously after that you then do:

systemctl restart apache2
systemctl restart haproxy

Now you have a proxy protocol'd setup from the proxy servers, and you can still talk directly to the Pi over ipv6, you're not yet logging the right remote ips, but we're a step closer. Next enable mod_remoteip in apache2:

a2enmod remoteip

And add a file, /etc/apache2/conf-available/remoteip-logformats.conf containing:

LogFormat "%v:%p %a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" remoteip_vhost_combined

And edit the /etc/apache2/sites-available/000-default.conf to change the CustomLog line to use remoteip_vhost_combined rather than combined as the LogFormat and add the relevant RemoteIP settings:

RemoteIPHeader X-Forwarded-For
RemoteIPTrustedProxy ::1

CustomLog ${APACHE_LOG_DIR}/access.log remoteip_vhost_combined

Now, enable the config and restart apache2:

a2enconf remoteip-logformats
systemctl restart apache2

Now you'll get the right remote ip in the logs (cool, huh!), and, better still, the environment that gets pushed through to cgi scripts/php/whatever is now also correct.

So, you can now happily visit http://www.<your-pi-name>.hostedpi.com/, e.g. http://www.srwpi.hostedpi.com/.

Next up, you'll want something like dehydrated - I grabbed the packaged version from debian's jessie-backports repository - so that you can make yourself some nice shiny SSL certificates (why wouldn't you, after all!), once you've got dehydrated installed, you'll probably want to tweak it a bit, I have some magic extra files that I use, I also suggest getting the dehydrated-apache2 package, which just makes it all much easier too.

/etc/dehydrated/conf.d/mail.sh:

CONTACT_EMAIL="my@email.address"

/etc/dehydrated/conf.d/domainconfig.sh:

DOMAINS_D="/etc/dehydrated/domains.d"

/etc/dehydrated/domains.d/srwpi.hostedpi.com:

HOOK="/etc/dehydrated/hooks/srwpi"

/etc/dehydrated/hooks/srwpi:

#!/bin/sh
action="$1"
domain="$2"

case $action in
  deploy_cert)
    privkey="$3"
    cert="$4"
    fullchain="$5"
    chain="$6"
    cat "$privkey" "$fullchain" > /etc/ssl/private/srwpi.pem
    chmod 640 /etc/ssl/private/srwpi.pem
    ;;
  *)
    ;;
esac

/etc/dehydrated/hooks/srwpi has the execute bit set (chmod +x /etc/dehydrated/hooks/srwpi), and is really only there so that the certificate can be used easily in haproxy.

And finally the file /etc/dehydrated/domains.txt:

www.srwpi.hostedpi.com srwpi.hostedpi.com

Obviously, use your own pi name in there, or better yet, one of your own domain names that you've mapped to the proxies.

Run dehydrated in cron mode (it's noisy, but meh...):

dehydrated -c

That s then generated you some shiny certificates (hopefully). For now, I'll just tell you how to do it through the /etc/apache2/sites-available/default-ssl.conf file, just edit that file and change the SSLCertificateFile and SSLCertificateKeyFile to point to /var/lib/dehydrated/certs/www.srwpi.hostedpi.com/fullchain.pem and /var/llib/dehydrated/certs/ww.srwpi.hostedpi.com/privkey.pem files, do the edit for the CustomLog as you did for the other default site, and change the VirtualHost to be [::1]:443 and enable the site:

a2ensite default-ssl
a2enmod ssl

And restart apache2:

systemctl restart apache2

Now time to add some bits to haproxy.cfg, usefully this is only a tiny tiny bit of extra config:

frontend any_https
        option httplog
        option forwardfor

        acl is_from_proxy src 2a00:1098:0:82:1000:3b:1:1 2a00:1098:0:80:1000:3b:1:1
        tcp-request connection expect-proxy layer4 if is_from_proxy

        bind :::443 ssl crt /etc/ssl/private/srwpi.pem

        default_backend any_https

backend any_https
        server apache2 ::1:4443 ssl ca-file /etc/ssl/certs/ca-certificates.crt

Restart haproxy:

systemctl restart haproxy

And we're all done! REMOTE_ADDR will appear as the correct remote address in the logs, and in the environment.

by Brett Parker (iDunno@sommitrealweird.co.uk) at March 01, 2017 06:35 PM

October 18, 2016

MJ Ray

Rinse and repeat

Forgive me, reader, for I have sinned. It has been over a year since my last blog post. Life got busy. Paid work. Another round of challenges managing my chronic illness. Cycle campaigning. Fun bike rides. Friends. Family. Travels. Other social media to stroke. I’m still reading some of the planets where this blog post should appear and commenting on some, so I’ve not felt completely cut off, but I am surprised how many people don’t allow comments on their blogs any more (or make it too difficult for me with reCaptcha and the like).

The main motive for this post is to test some minor upgrades, though. Hi everyone. How’s it going with you? I’ll probably keep posting short updates in the future.

Go in peace to love and serve the web. 🙂

by mjr at October 18, 2016 04:28 AM

March 09, 2015

Ben Francis

Pinned Apps – An App Model for the Web

(re-posted from a page I created on the Mozilla wiki on 17th December 2014)

Problem Statement

The per-OS app store model has resulted in a market where a small number of OS companies have a large amount of control, limiting choice for users and app developers. In order to get things done on mobile devices users are restricted to using apps from a single app store which have to be downloaded and installed on a compatible device in order to be useful.

Design Concept

Concept Overview

The idea of pinned apps is to turn the apps model on its head by making apps something you discover simply by searching and browsing the web. Web apps do not have to be installed in order to be useful, “pinning” is an optional step where the user can choose to split an app off from the rest of the web to persist it on their device and use it separately from the browser.

Pinned_apps_overview

”If you think of the current app store experience as consumers going to a grocery store to buy packaged goods off a shelf, the web is more like a hunter-gatherer exploring a forest and discovering new tools and supplies along their journey.”

App Discovery

A Web App Manifest linked from a web page says “I am part of a web app you can use separately from the browser”. Users can discover web apps simply by searching or browsing the web, and use them instantly without needing to install them first.

Pinned_apps_discovery

”App discovery could be less like shopping, and more like discovering a new piece of inventory while exploring a new level in a computer game.”

App Pinning

If the user finds a web app useful they can choose to split it off from the rest of the web to persist it on their device and use it separately from the browser. Pinned apps can provide a more app-like experience for that part of the web with no browser chrome and get their own icon on the homescreen.

Pinned_apps_pinning

”For the user pinning apps becomes like collecting pin badges for all their favourite apps, rather than cluttering their device with apps from an app store that they tried once but turned out not to be useful.”

Deep Linking

Once a pinned app is registered as managing its own part of the web (defined by URL scope), any time the user navigates to a URL within that scope, it will open in the app. This allows deep linking to a particular page inside an app and seamlessly linking from one app to another.

Pinned_apps_linking

”The browser is like a catch-all app for pages which don’t belong to a particular pinned app.”

Going Offline

Pinning an app could download its contents to the device to make it work offline, by registering a Service Worker for the app’s URL scope.

Pinned_apps_offline

”Pinned apps take pinned tabs to the next level by actually persisting an app on the device. An app pin is like an anchor point to tether a collection of web pages to a device.”

Multiple Pages

A web app is a collection of web pages dedicated to a particular task. You should be able to have multiple pages of the app open at the same time. Each app could be represented in the task manager as a collection of sheets, pinned together by the app.

Pinned_app_pages

”Exploding apps out into multiple sheets could really differentiate the Firefox OS user experience from all other mobile app platforms which are limited to one window per app.”

Travel Guide

Even in a world without app stores there would still be a need for a curated collection of content. The Marketplace could become less of a grocery store, and more of a crowdsourced travel guide for the web.

Pinned_apps_guide

”If a user discovers an app which isn’t yet included in the guide, they could be given the opportunity to submit it. The guide could be curated by the community with descriptions, ratings and tags.”

3 Questions

Pinnged_apps_pinned

What value (the importance, worth or usefulness of something) does your idea deliver?

The pinned apps concept makes web apps instantly useful by making “installation” optional. It frees users from being tied to a single app store and gives them more choice and control. It makes apps searchable and discoverable like the rest of the web and gives developers the freedom of where to host their apps and how to monetise them. It allows Mozilla to grow a catalogue of apps so large and diverse that no walled garden can compete, by leveraging its user base to discover the apps and its community to curate them.

What technological advantage will your idea deliver and why is this important?

Pinned apps would be implemented with emerging web standards like Web App Manifests and Service Workers which add new layers of functionality to the web to make it a compelling platform for mobile apps. Not just for Firefox OS, but for any user agent which implements the standards.

Why would someone invest time or pay money for this idea?

Users would benefit from a unique new web experience whilst also freeing themselves from vendor lock-in. App developers can reduce their development costs by creating one searchable and discoverable web app for multiple platforms. For Mozilla, pinned apps could leverage the unique properties of the web to differentiate Firefox OS in a way that is difficult for incumbents to follow.

UI Mockups

App Search

Pinned_apps_search

Pin App

Pin_app

Pin Page

Pin_page

Multiple Pages

Multiple_pages

App Directory

App_directory

Implementation

Web App Manifest

A manifest is linked from a web page with a link relation:

  <link rel=”manifest” href=”/manifest.json”>

A manifest can specify an app name, icon, display mode and orientation:

 {
   "name": "GMail"
   "icons": {...},
   "display": "standalone",
   "orientation": “portrait”,
   ...
 }

There is a proposal for a manifest to be able to specify an app scope:

 {
   ...
   "scope": "/"
   ...
 }

Service Worker

There is also a proposal to be able to reference a Service Worker from within the manifest:

 {
   ...
   service_worker: {
     src: "app.js",
     scope: "/"
   ...
 }

A Service Worker has an install method which can populate a cache with a web app’s resources when it is registered:

 this.addEventListener('install', function(event) {
  event.waitUntil(
    caches.create('v1').then(function(cache) {
     return cache.add(
        '/index.html',
        '/style.css',
        '/script.js',
        '/favicon.ico'
      );
    }, function(error) {
        console.error('error populating cache ' + error);
    };
  );
 });

So that the app can then respond to requests for resources when offline:

 this.addEventListener('fetch', function(event) {
  event.respondWith(
    caches.match(event.request).catch(function() {
      return event.default();
    })
  );
 });

by tola at March 09, 2015 03:54 PM

December 11, 2014

Ben Francis

The Times They Are A Changin’ (Open Web Remix)

In the run up to the “Mozlandia” work week in Portland, and in reflection of the last three years of the Firefox OS project, for a bit of fun I’ve reworked a Bob Dylan song to celebrate our incredible journey so far.

Here’s a video featuring some of my memories from the last three years, with Siobhan (my fiancée) and me singing the song at you! There are even lyrics so you can sing along 😉

“Keep on rockin’ the free web” — Potch

by tola at December 11, 2014 11:26 AM

July 10, 2014

James Taylor

SSL / TLS

Is it annoying or not that everyone says SSL Certs and SSL when they really mean TLS?

Does anyone actually mean SSL? Have there been any accidents through people confusing the two?


July 10, 2014 02:09 PM

Cloud Computing Deployments … Revisited.

So its been a few years since I’ve posted, because its been so much hard work, and we’ve been pushing really hard on some projects which I just can’t talk about – annoyingly. Anyways, March 20th , 2011 I talked about Continual Integration and Continual Deployment and the Cloud and discussed two main methods – having what we now call ‘Gold Standards’ vs continually updating.

The interesting thing is that as we’ve grown as a company, and as we’ve become more ‘Enterprise’, we’ve brought in more systems administrators and begun to really separate the deployments from the development. The other thing is we have separated our services out into multiple vertical strands, which have different roles. This means we have slightly different processes for Banking or Payment based modules then we do from marketing modules. We’re able to segregate operational and content from personally identifiable information – PII having much higher regulation on who can (and auditing of who does) access.

Several other key things had to change: for instance, things like SSL keys of the servers shouldn’t be kept in the development repo. Now, of course not, I hear you yell, but its a very blurry line. For instance, should the Django configuration be kept in the repo? Well, yes, because that defines the modules and things like URLs. Should the nginx config be kept in the repo? Well, oh. if you keep *that* in then you would keep your SSL certs in…

So the answer becomes having lots of repo’s. One repo per application (django wise), and one repo per deployment containing configurations. And then you start looking at build tools to bring, for a particular server or cluster of servers up and running.

The process (for our more secure, audited services) is looking like a tool to bring an AMI up, get everything installed and configured, and then take a snapshot, and then a second tool that takes that AMI (and all the others needed) and builds the VPC inside of AWS. Its a step away from the continual deployment strategy, but it is mostly automated.


July 10, 2014 02:09 PM

June 12, 2014

Paul Tansom

Beginning irc

After some discussion last night at PHP Hants about the fact that irc is a great facilitator of support / discussion, but largely ignored because there is rarely enough information for a new user to get going I decided it may be worth putting together a howto type post so here goes…

What is irc?

First of all, what on earth is it? I’m tempted to describe it as Twitter done right years before Twitter even existed, but I’m a geek and I’ve been using irc for years. It has a long heritage, but unlike the ubiquitous email it hasn’t made the transition into mainstream use. In terms of usage it has similarities to things like Twitter and Instant Messaging. Let’s take a quick look at this.

Twitter allows you to broadcast messages, they get published and anyone who is subscribed to your feed can read what you say. Everything is pretty instant, and if somebody is watching the screen at the right time they can respond straight away. Instant Messaging on the other hand, is more of a direct conversation with a single person, or sometimes a group of people, but it too is pretty instantaneous – assuming, of course, that there’s someone reading what you’ve said. Both of these techonologies are pretty familiar to many. If you go to the appropriate website you are given the opportunity to sign up and either use a web based client or download one.

It is much the same for irc in terms of usage, although conversations are grouped into channels which generally focus on a particular topic rather than being generally broadcast (Twitter) or more specifically directed (Instant Messaging). The downside is that in most cases you don’t get a web page with clear instructions of how to sign up, download a client and find where the best place is to join the conversation.

Getting started

There are two things you need to get going with irc, a client and somewhere to connect to. Let’s put that into a more familiar context.

The client is what you use to connect with; this can be an application – so as an example Outlook or Thunderbird would be a mail client, or IE, Firefox, Chrome or Safari are examples of clients for web pages – or it can be a web page that does the same thing – so if you go to twitter.com and login you are using the web page as your Twitter client. Somewhere to connect to can be compared to a web address, or if you’ve got close enough to the configuration of your email to see the details, your mail server address.

Let’s start with the ‘somewhere to connect to‘ bit. Freenode is one of the most popular irc servers, so let’s take a look. First we’ll see what we can find out from their website, http://freenode.net/.

freenode

There’s a lot of very daunting information there for somebody new to irc, so ignore most of it and follow the Webchat link on the left.

webchat

That’s all very well and good, but what do we put in there? I guess the screenshot above gives a clue, but if you actually visit the page the entry boxes will be blank. Well first off there’s the Nickname, this can be pretty much anything you like, no need to register it – stick to the basics of letters, numbers and some simple punctuation (if you want to), keep it short and so long as nobody else is already using it you should be fine; if it doesn’t work try another. Channels is the awkward one, how do you know what channels there are? If you’re lucky you’re looking into this because you’ve been told there’s a channel there and hopefully you’ve been given the channel name. For now let’s just use the PHP Hants channel, so that would be #phph in the Channels box. Now all you need to do is type in the captcha, ignore the tick boxes and click Connect and you are on the irc channel and ready to chat. Down the right you’ll see a list of who else is there, and in the main window there will be a bit of introductory information (e.g. topic for the channel) and depending on how busy it is anything from nothing to a fast scrolling screen of text.

phph

If you’ve miss typed there’s a chance you’ll end up in a channel specially created for you because it didn’t exist; don’t worry, just quit and try again (I’ll explain that process shortly).

For now all you really need to worry about is typing in text an posting it, this is as simple as typing it into the entry box at the bottom of the page and pressing return. Be polite, be patient and you’ll be fine. There are plenty of commands that you can use to do things, but for now the only one you need to worry about is the one to leave, this is:

/quit

Type it in the entry box, press return and you’ve disconnected from the server. The next thing to look into is using a client program since this is far more flexible, but I’ll save that for another post.

The post Beginning irc appeared first on Linuxlore.

by Paul Tansom at June 12, 2014 04:27 PM

May 06, 2014

Richard Lewis

Refocusing Ph.D

Actual progress on this Ph.D revision has been quite slow. My current efforts are on improving the focus of the thesis. One of the criticisms the examiners made (somewhat obliquely) was that it wasn't very clear exactly what my subject was: musicology? music information retrieval? computational musicology? And the reason for this was that I failed to make that clear to myself. It was only at the writing up stage, when I was trying to put together a coherent argument, that I decided to try and make it a story about music information retrieval (MIR). I tried to argue that MIR's existing evaluation work (which was largely modelled on information retrieval evaluation from the text world) only took into account the music information needs of recreational users of MIR systems, and that there was very little in the way of studying the music information seeking behaviour of "serious" users. However, the examiners didn't even accept that information retrieval was an important problem for musicology, nevermind that there was work to be done in examining music information needs of music scholarship.

So I'm using this as an excuse to shift the focus away from MIR a little and towards something more like computational musicology and music informatics. I'm putting together a case study of a computational musicology toolkit called music21. Doing this allows me to focus in more detail on a smaller and more distinct community of users (rather than attempting to studying musicologists in general which was another problematic feature of the thesis), it makes it much clearer what kind of music research can be addressed using the technology (all of MIR is either far too diverse or far too generic, depending on how you want to spin it), and also allows me to work with the actually Purcell Plus project materials using the toolkit.

May 06, 2014 11:16 PM

March 27, 2014

Richard Lewis

Taking notes in Haskell

The other day we had a meeting at work with a former colleague (now at QMUL) to discuss general project progress. The topics covered included the somewhat complicated workflow that we're using for doing optical music recognition (OMR) on early printed music sources. It includes mensural notation specific OMR software called Aruspix. Aruspix itself is fairly accurate in its output, but the reason why our workflow is non-trivial is that the sources we're working with are partbooks; that is, each part (or voice) of a multi-part texture is written on its own part of the page, or even on a different page. This is very different to modern score notation in which each part is written in vertical alignment. In these sources, we don't even know where separate pieces begin and end, and they can actually begin in the middle of a line. The aim is to go from the double page scans ("openings") to distinct pieces with their complete and correctly aligned parts.

Anyway, our colleague from QMUL was very interested in this little part of the project and suggested that we spend the afternoon, after the style of good software engineering, formalising the workflow. So that's what we did. During the course of the conversation diagrams were drawn on the whiteboard. However (and this was really the point of this post) I made notes in Haskell. It occurred to me a few minutes into the conversation that laying out some types and the operations over those types that comprise our workflow is pretty much exactly the kind of formal specification we needed.

Here's what I typed:

module MusicalDocuments where

import Data.Maybe

-- A document comprises some number of openings (double page spreads)
data Document = Document [Opening]

-- An opening comprises one or two pages (usually two)
data Opening = Opening (Page, Maybe Page)

-- A page comprises multiple systems
data Page = Page [System]

-- Each part is the line for a particular voice
data Voice = Superius | Discantus | Tenor | Contratenor | Bassus

-- A part comprises a list of musical sybmols, but it may span mutliple systems
--(including partial systems)
data Part = Part [MusicalSymbol]

-- A piece comprises some number of sections
data Piece = Piece [Section]

-- A system is a collection of staves
data System = System [Staff]

-- A staff is a list of atomic graphical symbols
data Staff = Staff [Glyph]

-- A section is a collection of parts
data Section = Section [Part]

-- These are the atomic components, MusicalSymbols are semantic and Glyphs are
--syntactic (i.e. just image elements)
data MusicalSymbol = MusicalSymbol
data Glyph = Glyph

-- If this were real, Image would abstract over some kind of binary format
data Image = Image

-- One of the important properties we need in order to be able to construct pieces
-- from the scanned components is to be able to say when objects of the some of the
-- types are strictly contiguous, i.e. this staff immediately follows that staff
class Contiguous a where
  immediatelyFollows :: a -> a -> Bool
  immediatelyPrecedes :: a -> a -> Bool
  immediatelyPrecedes a b = b `immediatelyFollows` a

instance Contiguous Staff where
  immediatelyFollows :: Staff -> Staff -> Bool
  immediatelyFollows = undefined

-- Another interesting property of this data set is that there are a number of
-- duplicate scans of openings, but nothing in the metadata that indicates this,
-- so our workflow needs to recognise duplicates
instance Eq Opening where
  (==) :: Opening -> Opening -> Bool
  (==) a b = undefined

-- Maybe it would also be useful to have equality for staves too?
instance Eq Staff where
  (==) :: Staff -> Staff -> Bool
  (==) a b = undefined

-- The following functions actually represent the workflow

collate :: [Document]
collate = undefined

scan :: Document -> [Image]
scan = undefined

split :: Image -> Opening
split = undefined

paginate :: Opening -> [Page]
paginate = undefined

omr :: Page -> [System]
omr = undefined

segment :: System -> [Staff]
segment = undefined

tokenize :: Staff -> [Glyph]
tokenize = undefined

recogniseMusicalSymbol :: Glyph -> Maybe MusicalSymbol
recogniseMusicalSymbol = undefined

part :: [Glyph] -> Maybe Part
part gs =
  if null symbols then Nothing else Just $ Part symbols
  where symbols = mapMaybe recogniseMusicalSymbol gs

alignable :: Part -> Part -> Bool
alignable = undefined

piece :: [Part] -> Maybe Piece
piece = undefined

I then added the comments and implemented the part function later on. Looking at it now, I keep wondering whether the types of the functions really make sense; especially where a return type is a type that's just a label for a list or pair.

I haven't written much Haskell code before, and given that I've only implemented one function here, I still haven't written much Haskell code. But it seemed to be a nice way to formalise this procedure. Any criticisms (or function implementations!) welcome.

March 27, 2014 11:13 PM

February 06, 2014

Adam Bower (quinophex)

I finally managed to beat my nemesis!

I purchased this book http://www.amazon.co.uk/dp/0738206679 (Linked, by Barabasi) on the 24th of December 2002, I had managed to make 6 or 7 aborted attempts at reading it to completion where life had suddenly got busy and just took over. This meant that I put the book down and didn't pick it up again until things were less hectic some time later and I started again.

Anyhow, I finally beat the book a few nights ago, my comprehension of it was pretty low anyhow but at least it is done. Just shows I need to read lots more given how little went in.




comment count unavailable comments

February 06, 2014 10:40 PM

February 01, 2014

Adam Bower (quinophex)

Why buying a Mio Cyclo 305 HC cycling computer was actually a great idea.

I finally made it back out onto the bike today for the first time since September last year. I'd spent some time ill in October and November which meant I had to stop exercising and as a result I've gained loads of weight over the winter and it turns out also become very unfit which can be verified by looking at the Strava ride from today: http://www.strava.com/activities/110354158

Anyhow, a nice thing about this ride is that I can record it on Strava and get this data about how unfit I have become, this is because last year I bought a Mio Cyclo 305 HC cycle computer http://eu.mio.com/en_gb/mio-cyclo-305-hc.htm from Halfords reduced to £144.50 (using a British Cycling discount). I was originally going to get a Garmin 500 but Amazon put the price up from £149.99 the day I was going to buy it to £199.99.

I knew when I got the Mio that it had a few issues surrounding usability and features but it was cheap enough at under £150 that I figured that even if I didn't get on with it I'd at least have a cadence sensor and heart rate monitor so I could just buy a Garmin 510 when they sorted out the firmware bugs with that and the price came down a bit which is still my longer term intention.

So it turns out a couple of weeks ago I plugged my Mio into a Windows VM when I was testing USB support and carried out a check for new firmware. I was rather surprised to see a new firmware update and new set of map data was available for download. So I installed it think I wasn't going to get any new features from it as Mio had released some new models but it turns out that the new firmware actually enables a single feature (amongst other things, they also tidied up the UI and sorted a few other bugs along with some other features) that makes the device massively more useful as it now also creates files in .fit format which can be uploaded directly to Strava.

This is massively useful for me as although the Mio always worked in Linux as the device is essentially just a USB mass storage device but you would have to do an intermediate step of having to use https://github.com/rhyas/GPXConverter to convert the files from the Mio-centric GPX format to something Strava would recognise. Now I can just browse to the folder and upload the file directly which is very handy.

All in it turns out that buying a Mio which reading reviews and forums were full of doom and gloom means I can wait even longer before considering replacement with a garmin.

comment count unavailable comments

February 01, 2014 02:11 PM

January 01, 2014

John Woodard

A year in Prog!


It's New Year's Day 2014 and I'm reflecting on the music of past year.

Album wise there were several okay...ish releases in the world of Progressive Rock. Steven Wilson's The Raven That Refused To Sing not the absolute masterpiece some have eulogised a solid effort though but it did contain some filler. Motorpsyco entertained with Still Life With Eggplant not as good as their previous album but again a solid effort. Magenta as ever didn't disappoint with The 27 Club, wishing Tina Booth a swift recovery from her ill health.

The Three stand out albums in no particular order for me were Edison's Children's Final Breath Before November which almost made it as album of the year and Big Big Train with English Electric Full Power which combined last years Part One and this years Part Two with some extra goodies to make the whole greater than the sum of the parts. Also Adrian Jones of Nine Stones Close fame pulled one out of the bag with his side Project Jet Black Sea which was very different and a challenging listen, hard going at first but surprisingly very good. This man is one superb guitarist especially if you like emotion wrung out of the instrument like David Gilmore or Steve Rothery.

The moniker of Album of the Year this year goes to Fish for the incredible Feast of Consequences. A real return to form and his best work since Raingods With Zippos. The packaging of the deluxe edition with a splendid book featuring the wonderful artwork of Mark Wilkinson was superb. A real treat with a very thought provoking suite about the first world war really hammed home the saying "Lest we forget". A fine piece that needs to be heard every November 11th.


Gig wise again Fish at the Junction in Cambridge was great. His voice may not be what it was in 1985 but he is the consummate performer, very at home on the stage. As a raconteur between songs he is as every bit as entertaining as he is singing songs themselves.

The March Marillion Convention in Port Zealand, Holland where they performed their masterpiece Brave was very special as every performance of incredible album is. The Marillion Conventions are always special but Brave made this one even more special than it would normally be.
Gig of the year goes again to Marillion at Aylesbury Friars in November. I had waited thirty years and forty odd shows to see them perform Garden Party segued into Market Square Heroes that glorious night it came to pass, I'm am now one very happy Progger or should that be Proggie? Nevermind Viva Progressive Rock!

by BigJohn (aka hexpek) (noreply@blogger.com) at January 01, 2014 07:56 PM

December 01, 2013

Paul Tansom

Scratch in a network environment

I have been running a Code Club at my local Primary School for a while now, and thought it was about time I put details of a few tweaks I’ve made to the default Scratch install to make things easier. So here goes:

With the default install of Scratch (on Windows) projects are saved to the C: drive. For a network environment, with pupils work stored on a network drive so they always have access whichever machine they sit at, this isn’t exactly helpful. It also isn’t ideal that they can explore the C: drive in spite of profile restrictions (although it isn’t the end of the world as there is little they can do from Scratch).

save-orig

After a bit of time with Google I found the answer, and since it didn’t immediately leap out at me when I was searching I thought I’d post it here (perhaps my Google Fu was weak that day). It is actually quite simple, especially for the average Code Club volunteer I should imagine; just edit the scratch.ini file. This is, as would be expected, located in:

C:\Program Files\Scratch\Scratch.ini

Initially it looks like this:

ini-orig

Pretty standard stuff, but unfortunately no comments to indicate what else you can do with it. As it happens you can add the following two lines (for example):

Home=U:
VisibleDrives=U:

To get this:

ini-new

They do exactly what is says on the tin. If you click on the Home button in a file dialogue box then you only get the drive(s) specified. You can also put a full path in if you want to put the home directory further down the directory structure.

save-new1

The VisibleDrives option restricts what you can see if you click on the Computer button in a file dialogue box. If you want to allow more visible drives then separate them with a comma.

save-new2

You can do the same with a Mac (for the home drive), just use the appropriate directory format (i.e. no drive letter and the opposite direction slash).

There is more that you can do, so take a look at the Scratch documentation here. For example if you use a * in the directory path it is replaced by the name of the currently logged on user.

Depending on your network environment it may be handy for your Code Club to put the extra resources on a shared network drive and open up an extra drive in the VisibleDrives. One I haven’t tried yet it is the proxy setting, which I hope will allow me to upload projects to the Scratch website. It goes something like:

ProxyServer=[server name or IP address]
ProxyPort=[port number]

The post Scratch in a network environment appeared first on Linuxlore.

by Paul Tansom at December 01, 2013 07:00 PM

January 16, 2013

John Woodard

LinuxMint 14 Add Printer Issue


 LinuxMint 14 Add Printer Issue



 

I wanted to print from my LinuxMint 14 (Cinnamon) PC via a shared Windows printer on my network. Problem is it isn’t found by the printers dialog in system settings. I thought I’d done all the normal things to get samba to play nice like rearranging the name resolve order in /etc/samba/smb.conf to a more sane bcast host lmhosts wins. Having host and wins, neither of which I’m using first in the order cocks things up some what. Every time I tried to search for the printer in the system setting dialog it told me “FirewallD is not running. Network printer detection needs services mdns, ipp, ipp-client and samba-client enabled on firewall.” So much scratching of the head there then, because as far as I can tell there ain’t no daemon by that name available!

It turns out thanks to /pseudomorph this has been a bug since LinuxMint12 (based on Ubuntu 11.10). It’s due to that particular daemon (Windows people daemon pretty much = service) being Fedora specific and should have no place in a Debian/Ubuntu based distribution. Bugs of this nature really should be ironed out sooner.

Anyway the simple fix is to use the more traditional approach using the older printer dialog which is accessed by inputting system-config-printer at the command line. Which works just fine so why the new (over a year old) printer config dialog that is inherently broken I ask myself.

The CUPS web interface also works apparently http://localhost:631/ in your favourite browser which should be there as long as CUPS is installed which it is in LinuxMint by default.

So come on Minty people get your bug squashing boots on and stamp on this one please.

Update

Bug #871985 only affects Gnome3 so as long as its not affecting Unity that will be okay Canonical will it!

by BigJohn (aka hexpek) (noreply@blogger.com) at January 16, 2013 12:39 AM

August 20, 2012

David Reynolds

On Music

Lately, (well I say lately, I think it’s been the same for a few years now) I have been finding that it is very rare that an album comes along that affects me in a way that music I heard 10 years ago seem to. That is not to say that I have not heard any music that I like in that time, it just doesn’t seem to mean as music that has been in my life for years. What I am trying to work out is if that is a reflection on the state of music, of how I experience music or just me.

Buying

Buying music was always quite an experience. I would spend weeks, months and sometimes longer saving up to buy some new music. Whether I knew exactly what I wanted or just wanted “something else by this artist” I would spend some time browsing the racks weighing up what was the best value for my money. In the days before the internet, if you wanted to research an artist’s back catalogue, you were generally out of luck unless you had access to books about the artists. This lead to the thrill of finding a hidden gem in the racks that you didn’t know existed or had only heard rumours about. The anticipation of listening to the new music would build even more because I would have to wait until I had travelleled home before I could listen to my new purchases.

Nowadays, with the dizzying amount of music constantly pumped into our ears through the internet, radio, advertising and the plethora of styles and genres, it is difficult to sift through and find artists and music that really speak to you. Luckily, there are websites available to catalogue releases by artists so you are able to do thorough research and even preview your music before you purchase it. Of course the distribution methods have changed massively too. No longer do I have to wait until I can make it to a brick and mortar store to hand over my cash. I can now not only buy physical musical releases on CD or Vinyl online and have it delivered to my door, I can also buy digital music through iTunes, Amazon or Bandcamp or even stream the music straight to my ears through services like Spotify or Rdio. Whilst these online sales avenues are great for artists to be able to sell directly to their fans, I feel that some of the magic has been removed from the purchasing of music for me.

Listening

Listening to the music used to be an even greater event than purchasing it. After having spent the time saving up for the purchase, then the time carefully choosing the music to buy and getting it home, I would then sit myself down and listen to the music. I would immerse myself totally in the music and only listen to it (I might read the liner notes if I hadn’t exhausted them on the way home). It is difficult to imagine doing one thing for 45+ minutes without the constant interruptions from smartphones, tablet computers, games consoles and televisions these days. I can’t rememeber the last time I listened to music on good speakers or headphones (generally I listen on crappy computers speakers or to compressed audio on my iPhone through crappy headphones) without reading Twitter, replying to emails or reading copiuous amounts of information about the artists on Wikipedia. This all serves to distract from the actual enjoyment of just listening to the music.

Experience

The actual act of writing this blog post has called into sharp focus the main reason why music doesn’t seem to affect me nowadays as much as it used to - because I don’t experience it in the same way. My life has changed, I have more resposibilities and less time to just listen which makes the convenience and speed of buying digital music online much more appealing. You would think that this ‘instant music’ should be instantly satisfying but for some reason it doesn’t seem to work that way.

What changed?

I wonder if I am the only one experiencing this? My tastes in music have definitely changed a lot over the last few years, but I still find it hard to find music that I want to listen to again and again. I’m hoping I’m not alone in this, alternatively I’m hoping someone might read this and recommend some awesome music to me and cure this weird musical apathy I appear to me suffering from.

August 20, 2012 03:33 PM

On Music

Lately, (well I say lately, I think it’s been the same for a few years now) I have been finding that it is very rare that an album comes along that affects me in a way that music I heard 10 years ago seem to. That is not to say that I have not heard any music that I like in that time, it just doesn’t seem to mean as music that has been in my life for years. What I am trying to work out is if that is a reflection on the state of music, of how I experience music or just me.

Buying

Buying music was always quite an experience. I would spend weeks, months and sometimes longer saving up to buy some new music. Whether I knew exactly what I wanted or just wanted “something else by this artist” I would spend some time browsing the racks weighing up what was the best value for my money. In the days before the internet, if you wanted to research an artist’s back catalogue, you were generally out of luck unless you had access to books about the artists. This lead to the thrill of finding a hidden gem in the racks that you didn’t know existed or had only heard rumours about. The anticipation of listening to the new music would build even more because I would have to wait until I had travelleled home before I could listen to my new purchases.

Nowadays, with the dizzying amount of music constantly pumped into our ears through the internet, radio, advertising and the plethora of styles and genres, it is difficult to sift through and find artists and music that really speak to you. Luckily, there are websites available to catalogue releases by artists so you are able to do thorough research and even preview your music before you purchase it. Of course the distribution methods have changed massively too. No longer do I have to wait until I can make it to a brick and mortar store to hand over my cash. I can now not only buy physical musical releases on CD or Vinyl online and have it delivered to my door, I can also buy digital music through iTunes, Amazon or Bandcamp or even stream the music straight to my ears through services like Spotify or Rdio. Whilst these online sales avenues are great for artists to be able to sell directly to their fans, I feel that some of the magic has been removed from the purchasing of music for me.

Listening

Listening to the music used to be an even greater event than purchasing it. After having spent the time saving up for the purchase, then the time carefully choosing the music to buy and getting it home, I would then sit myself down and listen to the music. I would immerse myself totally in the music and only listen to it (I might read the liner notes if I hadn’t exhausted them on the way home). It is difficult to imagine doing one thing for 45+ minutes without the constant interruptions from smartphones, tablet computers, games consoles and televisions these days. I can’t rememeber the last time I listened to music on good speakers or headphones (generally I listen on crappy computers speakers or to compressed audio on my iPhone through crappy headphones) without reading Twitter, replying to emails or reading copiuous amounts of information about the artists on Wikipedia. This all serves to distract from the actual enjoyment of just listening to the music.

Experience

The actual act of writing this blog post has called into sharp focus the main reason why music doesn’t seem to affect me nowadays as much as it used to - because I don’t experience it in the same way. My life has changed, I have more resposibilities and less time to just listen which makes the convenience and speed of buying digital music online much more appealing. You would think that this ‘instant music’ should be instantly satisfying but for some reason it doesn’t seem to work that way.

What changed?

I wonder if I am the only one experiencing this? My tastes in music have definitely changed a lot over the last few years, but I still find it hard to find music that I want to listen to again and again. I’m hoping I’m not alone in this, alternatively I’m hoping someone might read this and recommend some awesome music to me and cure this weird musical apathy I appear to me suffering from.

August 20, 2012 03:33 PM

June 25, 2012

Elisabeth Fosbrooke-Brown (sfr)

Black redstarts

It's difficult to use the terrace for a couple of weeks, because the black redstart family is in their summer residence at the top of a column under the roof. The chicks grow very fast, and the parents have to feed them frequently; when anyone goes out on the terrace they stop the feeding process and click shrill warnings to the chicks to stay still. I worry that if we disturb them too often or for too long the chicks will starve.

Black redstarts are called rougequeue noir (black red-tail) in French, but here they are known as rossignol des murailles (nightingale of the outside walls). Pretty!

The camera needs replacing, so there are no photos of Musatelier's rossignols des murailles, but you can see what they look like on http://fr.wikipedia.org/wiki/Rougequeue_noir.

by sunflowerinrain (noreply@blogger.com) at June 25, 2012 08:02 AM

June 16, 2012

Elisabeth Fosbrooke-Brown (sfr)

Roundabout at Mirambeau

Roundabouts are taken seriously here in France. Not so much as traffic measures (though it has been known for people to be cautioned by the local gendarmes for not signalling when leaving a roundabout, and quite rightly too), but as places to ornament.

A couple of years ago the roundabout at the edge of  Mirambeau had a make-over which included an ironwork arch and a carrelet (fishing hut on stilts). Now it has a miniature vineyard as well, and roses and other plants for which this area is known.

Need a passenger to take photo!

by sunflowerinrain (noreply@blogger.com) at June 16, 2012 12:06 PM

September 04, 2006

Ashley Howes

Some new photos

Take a look at some new photos my father and I have taken. We are experimenting with our new digital SLR with a variety of lenses.

by Ashley (noreply@blogger.com) at September 04, 2006 10:42 AM

August 30, 2006

Ashley Howes

A Collection of Comments

This is a bit of fun. A collection of comments found in code. This is from The Daily WTF.

by Ashley (noreply@blogger.com) at August 30, 2006 01:13 AM