Low level fun.

As a wise man said, “low level is easy”.

There aren’t that many things below that can break except for hardware (and compiler!) bugs, they are becoming increasingly common but still the level of pressure and control to keep quality up on processors are orders of magnitude greater than those on the stacks above the OS.

A minor update of Ruby broke everything? The layout is totally wrong when a browser locale is set to Spanish but no in English? Who’s to blame for that? Even if we have most of the sources fixing the problem for everyone but ourselves isn’t that easy.

The other Monday I went to the Uni to take an exam on digital circuits (graded with 9/10) and oh boy, programming in assembler again was refreshing.

The tooling is just as awful as the last time I took that course (not as bad as the first versions of mplab but the assembler is still dumb when it comes to parsing spaces and the integrated editor is notepad in disguise) but the difference is that now they allowed us to use our own notebooks. The only saving grace is that the assembler, debugger and simulator are different executables, they accept a sane set of command line parameters and they run fine under wine.

So I was able to use vim and a simple makefile to streamline the process and finish in almost no time. Sadly the bar to pass has declined a lot in the last years.

When I’m say I’m alright

… don’t look in my eyes ’cause you’ll see I’m not alright.

But I’m trying.

About a month and half ago I snapped and couldn’t take it anymore. I sought professional help and got myself a Real Shrink and a Therapist (both extremely good and expensive). And I’m on meds now.

For the most part of the last three to four years what I described as “happy” or “fine” was actually different shades of miserable. I had my moments of truly happiness, but those were really few. And there was always that tiny rainy cloud looming above me.

I had a breakdown like this back in 2015 and I almost wanted to give up and get away from programming (I kinda did, I biked many times to the countryside trying to look for a quiet spot).

This time it’s not so bad on its own but compounded with other things happening in my life (one of the biggest is leaving Berisso) it totally wrecked me. The difference now it’s that I asked for help. I still run away from the civilization to take day long bike rides and reflex but I’m trying to get a grip on the problem and fix myself.

Lately I’ve lost interest in most of what my life is/was centered about (technology, engineering) and that bothers me a lot. Not completely, I have many drafts and toy projects that prove otherwise.

The knack it’s there but the thrill is gone. At least for a while.

As Dave Thomas put it, this is supposed to be fun. And it stopped being such for me, instead of an almost daily stream of marvel it gradually turned into a grind with an occasional touch of enjoyment, and not always at work.

I believe that we are defined by what we do (more than what we think / believe), work being a significant part of that. On that reference frame, this is a very tough situation.

So what now? I’m going to spend all the time I need to get better and take care of myself. I’m taking a bit of time off work and going back to uni for real, coming clean of my schedule to not rush things pretending I’m doing something when I’m not (attending lectures and using slack on the phone is awful). I’m saying no instead of hesitating. I’m not committing to do things just to please someone. I take a break when I feel I need it.

I know this stuff (antidepressants) takes time to make effect and it’s not a silver bullet that somehow will make everything fine overnight but I’m anxious, waiting for something to happen.

On a totally unrelated thing, the quote from the beginning is from a Storm Large song (it can be found here). I really miss her days with The Balls, the energy and vocal range from that era reminds me a lot to Skunk Anansie. But her smooth side of nowadays is gorgeous.

What nobody tells

Nobody tells this to people who are beginners, I wish someone told me.

All of us who do creative work, we get into it because we have good taste.

But there is this gap. For the first couple years you make stuff, it’s just not that good.

It’s trying to be good, it has potential, but it’s not. But your taste, the thing that got you into the game, is still killer.
And your taste is why your work disappoints you.

A lot of people never get past this phase, they quit.

Most people I know who do interesting, creative work went through years of this.
We know our work doesn’t have this special thing that we want it to have. We all go through this.

And if you are just starting out or you are still in this phase, you gotta know its normal and the most important thing you can do is do a lot of work. Put yourself on a deadline so that every week you will finish one story.

It is only by going through a volume of work that you will close that gap, and your work will be as good as your ambitions.

And I took longer to figure out how to do this than anyone I’ve ever met.

It’s gonna take awhile.
It’s normal to take awhile. You’ve just gotta fight your way through.

Ira Glass

Also, Andy Hunt used to say:

It is the artistry,
not the art.
It is the process,
not the product.
It is the journey,
not the destination.

Burning down the house

(Watch out)

So many drafts, some stories and pictures from the last PyCon at Bahía Blanca.

I was happily hacking on the kitchen the other Saturday when I hear a strange noise coming from the garden.

To my dismal surprise I see that the shed is on fire and part of the roof collapsed. I went in to take out a propane can to avoid an impending catastrophe and called the firemen (lucky us, they are a few blocks away).

We lost the roof, tools, vinyls and books on an adjacent room but nothing that can’t be replaced. Still fuck.

Some pictures of PyCon at flickr (not mine) https://www.flickr.com/photos/70871182@N04/sets/72157677377824525

Bike basket

A reclaimed milk crate, some wood leftover from another project and pipe clamps.
Now I can roam freely, it feels so nice to my back not having to carry the backpack. It’s particularly useful when doing errands or picking up “trash” (but for me it’s a treasure).

All it takes it’s asking.

About a month ago I was at the bank.

Despite arriving a quarter before the opening hour there was already a long queue and once inside my ticket was number 29. That more or less means three hours of wait.

I carried on with the daily chores and went back, there was still a lot of people before me and I took a seat to rest for a while.

Next to me there was this couple talking, I didn’t quite listen to them but all of a sudden I catched that they had two tickets and were going to the same teller as me anyways. So I asked if they wanted to give me one.

And they said yes, moving me back to the 16. I was glowing in satisfaction. The rest of the people looked at me like some sort of criminal.

Not totally unrelated, last week we went to another bank to do some paperwork on a shared account. I arrived earlier and grabbed a ticket. The other owner was delayed and so I stepped on the middle of the hall asking if anyone wanted to trade my ticket for theirs. Everybody looked at me in disbelief, like, “what’s this wacko asking?”. Finally one slowly raised his hand, we traded places and everyone was happy.

Fixing a microwave oven with a broken keypad

This is by far one of the most productive things I did this week outside of work (at least the one I can write about here).

A couple of months ago my ex gave it to me, it started with intermittent display issues and one day it stopped completely. I picked it up and stored it.

The other weekend I was in a bit of cleaning frenzy and I remembered that it was using valuable space on the shack doing nothing so I set to see if it had any hope of working again. Otherwise I’d take the transformer and dish motor, the magnetron would go to a friend and the rest sold as scrap.

This is the second time I fix a microwave oven and I’m amazed at the amount of grease and acid stench that accumulates inside them.

I bridged the safety interlock pads on the control board and powered it with an isolation transformer. It kinda turned on but was not responsive and only some digits were dimly lit. It was also very sticky.

After that I cleaned it using lukewarm water, detergent and a toothbrush, a scoop with a hair drier and then another bath with alcohol.

Now it works!

The keypad is a mess, besides being sticky and stenchy too the conductive traces were broken, like dissolved, on the connector side. For some models there are still replacements on the market but they aren’t cheap and also what’s the fun on that?

I peeled away the layers, traced it and make a replacement using tact switches. The decal will be glued on top of that. It works fine, there’s less waste (but I’m short of a spot welder) and off it goes to Radio Futura.

Doomed

So, I’m facing an issue and the best tools so far (or the ones that are the less worse) seem to involve both php and xslt. And a braindead webservice. Go team.
Not totally unrelated, I’m surprised at the amount of stuff that can be found with “depressed developer”.

Yummy.

The other week I felt like cooking.

I made cubes of calabaza in syrup like grandma used to, with ash or lime to harden the outside. About half day sitting in water and lime, a thorough clean and then five hours give or take on the stove with lots of sugar.

After that I roasted a sweet potato and just for kicks I also fried a banana in a mixture of honey and butter. That was really tasty.

Guidelines for C source code auditing and other tales.

The papers and articles at this site are quite interesting, even if a little dated. Somehow I had many of them opened from a couple of days ago but just now took the time to really read them.

Guidelines for C source code auditing: http://www.ouah.org/mixtercguide.html

Syscall Proxying – Simulating remote execution: http://www.ouah.org/SyscallProxying.pdf

An Overview of Unix Rootkits: http://www.ouah.org/iRootkits.pdf

Know your Enemy: http://www.ouah.org/motives.html

…and part the other series: http://web.archive.org/web/20010607083412/http://project.honeynet.org/papers/

10 times.

Somehow this is a grossly exaggerated notion within the software community but in every field we find people that performs better (by some measure) than the average by a very high margin.

I witnessed that with my very own eyes and they are not magical creatures. There’s a lot written about them but they tend to share some common traits of overachievers that make the real difference from the rest:

  • They have a clue about what they are doing.
  • They are focused.
  • They work on things that matter.

(Or as Yosef puts it in http://yosefk.com/blog/10x-more-selective.html, things that aren’t going down the toilet. I like it when he says, “The hardest part of “managing” these 10x folks – people widely known as extremely productive – is actually convincing them to work on something. (The rest of managing them tends to be easy – they know what’s what; once they decide to do something, it’s done.“)

Now, I’m not the brightest bulb but when I tackle a problem I try as much as I can to understand its domain. I ask myself frequently if there’s a better way to approach it, as it’s very rare to come across something so unique that nobody worked on anything that barely resembles it (or that can be applied to the current problem).

I’m working on a system that it’s getting older but the foundation is solid and it shows. Everything makes sense, even if you have no idea about a piece is often it can be found in an intuitive way and the core looks beautiful, even if it’s made with dying technologies. The architecture is very well designed and implemented.

But then people came and started adding little things here and there without very much thought. They built XML files concatenating strings, they copied the routines into 34 (that’s real) places and each one has a little difference (that’s kinda ok, they talk to things so horrible that can’t process CDATA fields and use a custom encoding instead). The idea of having all the common stuff in one place never crossed their minds or, shiver, use a standard library (they existed and were mature back when those things were implemented).

They also wrote lots, and lots, of functions like (php):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
private function frobnicate($the_frob)
{
    // Selects the baz of the frob.
 
    // If it's one kind of baz
    if($the_frob->is_of_type('one kind of baz'))
    {
        return $this->baz = 'Baz1';
    }
 
    // If it's a special baz
    if($the_frob->is_of_type('a special baz'))
    {
        return $this->baz = 'Special';
    }
 
    // If it's a straw one
    if($the_frob->is_of_type('a straw one'))
    {
        return $this->baz = 'nuts';
    }
 
    // ... snip about 200 lines of the same ...
 
    return $this->baz = 'the default value';
}

And this is repeated in about 56 places, intertwined with many more conditionals. I only hope this was generated code and not typed by hand.

Anyway, I nuked it and turned that wall of if statements into:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
private function frobnicate($the_frob)
{
    // Selects the baz of the frob.
 
    $map = array(
        'one kind of baz' => 'Baz1',
        'a special baz'   => 'Special',
        'a straw one'     => 'nuts',
        // ... you get the picture ...
    );
 
    foreach($map as $type => $baz) {
        if ($the_frob->is_of_type($type)) {
            return $this->baz = $baz;
        }
    }
 
    return $this->baz = 'the default value';
 
}

And that’s even not clever (table driven programming has been around for quite some time).

I mean, after doing it three times I question if there’s a better, more concise way of expressing the same. But for some people that moment never comes.

Focus.

These days I’m having trouble to keep focused for more than four hours straight, make that six in a very very god day. I guess that’s a given with age and more responsibilities. Doing boring stuff doesn’t help either, but it’s a good incentive to finish as soon as possible without mistakes.

Do things that matter.

This deeply touches me.

Nowadays most of the stuff I do at work to put food on the plate is meaningless and boring to death. It makes life easier for a lot of people but nothing will change if it goes away overnight. Many will cry but nothing terrible.

It doesn’t give me a technical challenge any more. At best it teaches me patience and how to deal with utterly broken and stupid systems that were not designed to be used (not by computers and certainly not by humans). It drains my energy and I’m past the point where it makes sense to put up with it.

It doesn’t make the world a better place, not even by chance.

Do I do things that matter? Yes, on weekends and sometimes by night.
Can I make a living out of them? Not now.
Not yet.

A statistical insight

I’ve been working during the weekends on an instrumentation frontend to precisely measure the resistance of an RTD sensor using a ratiometric approach.

After building it and waiting a prudential time to let it warm I saved an hour of samples (3600) and fired Octave.

The mean and standard deviation looked ok and while a plot showed a bit of noise it was well within reasonable limits.

Just for the sake of it I did a histogram and, oh the horror:

This is clearly not OK. It should be more like a Gaussian (the real formula is quite daunting but still retains symmetry) and that looks a lot like a bimodal distribution. Changing the number of bins does not help.

The ADC I used does not have a reference input so I make two differential reads and then take the quotient (I know… but it was the only one available when started).

Perhaps the input multiplexer is at fault? (the unused channels are grounded, so I discarded that as a cause). I repeated the experiment but this time doing a full run on each channel instead of switching them and this is the result:

Well, both are skewed so there’s something else going on.

Scoping at the inputs shows what seems to be AM at around 70MHz even without power applied (that’s on the tv broadcast band here) and it kind of makes sense because I didn’t use a shield. Head bangs on the desk.

Anyways, using a quick digital filter makes everything look nicer but I’ll still have to shield this:

The transient at the beginning is not going to be an issue, as in real life I don’t expect such a step change (from 0 to ~3k) and in any case the antialias filter will get rid of it.

On a second thought, those chunks skewed up are really interesting and I should spotted that as a failure symptom earlier.

Learning again

I got into computers at the age of 8 (1994), when a very weird lady came to our school and offered to teach us. A week after that she was back with a bunch of friends and unloaded a stack of XT’s and AT’s on an empty room.

We spent two months doing exercises on paper and then she introduced us to QBasic (and DOS).

After that I continued with Visual Basic (and stopped at 6, I made a couple of programs for a very good amount of money), assembler for PIC micros. I also used C with TurboC, DJGPP and even Visual C.

I stumbled upon Python and its community and I fell in love. It is still my go to language for almost everything, even if only to try out solutions and then port them to the target language.

The last major event was a couple (and a bit more…) years ago when I started to use more and more Javascript, from plain servers using express to fully encompassing frameworks like LoopBack. From big iron to smaller micros (besides code isomorphism is really interesting)

After that I toyed with some languages but nothing really caught my attention. Among them I used Ruby, Haskell, Go and a bit of Java just for kicks or helping out long time friends. Also TCL, I   liked it as an extension language but I wouldn’t write an application from scratch using it. Go is really interesting (mainly because Docker and all the tools revolving around it) but I really need some itch that calls for it.

The major breakthrough came when I rediscovered Forth and Erlang.

Forth is really interesting as an embedded language because it’s so easy to implement and there are so many things powered by it.

Erlang was on my radar for a while and has, at least on my circle of friends, this aura of mystery.

After a post on PyAr I decided to join a local group and give it a go. I started with the exercises of Learn You Some Erlang and I couldn’t be happier.

It makes my head twitch and think in new ways. I just lust for the time of the day devoted to it.

Breaking a simple captcha with Python and Pillow

A while ago one of our long time customers approached us to automate tasks on a government portal. At least here most of them are kind of ugly, work on a specific set of browser versions and are painfully slow. We already helped him with problems like this before, so instead of having someone enter manually all the data they just populate a database and then our robot does all the work, simulating the actions on the web portal.

This one is a bit different, because they introduced a captcha in order to infuriate users (seriously, it looks like they don’t want people logging in).

Most of the time they look like this:

The first thing I tried was to remove the lines and feed the result into an ocr engine. So I made a very simple filter using Pillow:


#!/usr/bin/python

from PIL import Image
import sys, os

def filter_lines(src):
    w,h = src.size

    stripes = []
    ss = {}

    for x in range(w):
        count = 0
        for y in range(h):
            if src.getpixel( (x,y) ) != (248, 255, 255):
                count += 1
        if count == h:
            stripes.append(x)

    for x in stripes:
        for y in range(h):
            src.putpixel( (x,y),  (248, 255, 255) )
    return src

if __name__ == '__main__':
    src = Image.open(sys.argv[1])
    region = filter_lines(src)
    region.save(sys.argv[2])

Now it looks better but after trying gocr and tesseract it still needs more work:

Just for kicks I decided to filter 100 images and overlap them, this is what I got:

That is interesting… I used this script (not the most efficient approach, but still..)


#!/usr/bin/python

from PIL import Image
import sys, os

dst = Image.new('RGB', (86, 21) )

w,h = 86, 21

for x in range(w):
    for y in range(h):
        dst.putpixel( (x,y),  (255, 255, 255) )

for idx in range(30):
    src = Image.open('filtradas/%i.bmp'%idx)

    for x in range(w):
        for y in range(h):
            if src.getpixel( (x,y) ) != (248, 255, 255):
                dst.putpixel( (x,y),  (255, 0, 0) )

dst.save('overlapeada.bmp')

With this piece of information I can focus my efforts on that area only.
That font, even distorted, looks quite familiar to me. And indeed it is, it’s Helvetica.
This makes the problem a lot easier.

I grabbed a bitmapped version of the same size and made a grid that shows were can a number land assuming 8×13 symbols:

This shows that there is a slightly overlap between digits.
I went for a brute force approach, dividing the captcha in cells and comparing each one with every digit on the font with a small amount of overlap between them.
The symbols are smaller than the cell, so for every one of them I build regions on the cell and assign a score for the number of pixels that are equal on both.
The one that has a highest score is (likely) the correct number.

This is really simple, event tough we do a lot of comparisons performs ok (the images are quite small), and without tunning we got about 30% success rate (the server also adds noise and more aggressive distortions from time to time).

Have a difficult or non conventional problem? Give us a call, we are like the A-Team of technology.

This is the complete algorithm (it’s in Spanish but shouldn’t be hard to follow), can also be found here: https://gist.github.com/pardo-bsso/a6ab7aa41bad3ca32e30


#!/usr/bin/python

from PIL import Image
import sys, os


imgpatrones = []
pixelpatrones = []

for idx in range(10):
    img = Image.open("patrones/%i.png" % idx).convert('RGB')
    imgpatrones.append(img)
    pixelpatrones.append( list(img.getdata()) )


def compara(region, patron):
    pixels = list(region.getdata())
    size = min(len(pixels), len(patron))

    res = 0.0
    for idx in range(size):
        if pixels[idx] == patron[idx]:
            res = res + 1

    return res / size


def elimina_lineas(src):
    cropeada = src.crop( (4, 1, 49, 19) )
    w,h = cropeada.size
    stripes = []

    for x in range(w):
        count = 0
        for y in range(h):
            if cropeada.getpixel( (x,y) ) != (248, 255, 255):
                count += 1

        if count == h:
            stripes.append(x)

    for x in stripes:
        for y in range(h):
            cropeada.putpixel( (x,y),  (248, 255, 255) )
            cropeada.putpixel( (x,y),  (255, 0, 0) )

    return cropeada

def crear_crops(src, celda):
    limites = range(38)
    xceldas = [0, 8, 16, 24, 32, 40]
    xoffsets = range(-3,4)
    yceldas = range(6)
    boxes = []
    crops = []

    x = xceldas[celda]
    x = [ (x+off) for off in xoffsets if (x+off) in limites ]

    for left in x:
        for top in yceldas:
            boxes.append( (left, top, left+8, top+13) )

    for box in boxes:
        crops.append( src.crop(box) )

    return crops

def compara_crops_con_patron(crops, patron):
    scores = []
    for crop in crops:
        scores.append( compara(crop, pixelpatrones[patron] ))
    return max(scores)

def decodifica_celda(src, celda):
    pesos = []
    crops = crear_crops(src, celda)

    for patron in range(10):
        pesos.append( compara_crops_con_patron(crops, patron) )

    return pesos.index( max(pesos) )

def decodifica(filename):
    original = Image.open(filename)
    src = elimina_lineas(original)
    res = []

    for celda in range(6):
        res.append( decodifica_celda(src, celda) )

    return ''.join( str(x) for x in res )

if __name__ == '__main__':
    print decodifica(sys.argv[1])