How do I learn to do what you do

A friend asked me how I got to where I was.

The largest problem with answering this question is my path can never be your path. At its most extreme this makes advice useless: the coach at the pro shop grew up thwacking tennis balls as a toddler, his advice for a 45-year-old getting in to the game is probably worse than useless.

But to set some parameters, I feel I have some advice for people who are:

  • over 30 or so
  • don’t have a Computer Science degree
  • have responsibilities outside of work that make a full time bootcamp or similar very hard to do
  • have not excelled in another field – I wasn’t a doctor or lawyer or a movie mogul before I started programming

Separate posts detail what my path was exactly, but here are 10 things I learned on my path from “guy who wants to write some code for the company site and how do we make him stop?” and “making $70 an hour as a contractor”

1) Learn something that is easy to learn

You’re really going to struggle to learn Java from Stack Overflow. In general it is super easy to find online tutorials in

  • Frontend Javascript
  • Ruby
  • React
  • SQL
  • Node

It’s also easier to find MOOC’s, discussion board posts, and interactive ‘games’ to learn the above list.

2) Learn from a practical source

Phonebook-sized textbooks on a software are a common sight, and it’s so appealing! Surely if you work your way through a giant compendium on HTML and CSS you’ll be a developer by the end. The problem is that these tend not to be real curriculum in how to do a job, but more like the user manual for an airplane cockpit. They’ll load you down with unecessary information about methods that no one uses frequently, and won’t give you critical advice about how to get something done

There’s a critical difference between telling you how to use every tool in the wood shop, and showing you how to make a chair

In general, try to get a text that a real human working as a programmer has recommended to you, and then check out the ‘who this book is for’ section to make sure it’s aimed at a beginner.

3) Learn in the easiest way possible

When you were in school, which topic did you find easiest and why? For me it was definitely physics class, where the math and theory came secondary to almost daily demonstrations of small topics. The subject had a certain ‘cool’ and every theoretical point was underpinned with a memorable visual lesson.

When I first went to learn programming I made myself sit alone at a desk with a poorly written textbook. It shouldn’t be that surprising that I learned very little even after trying hard for a year or so.

While I think some resources are just bad, there’s also many systems that are bad for me personally.  You have to find something that motivates you and do that, even if it’s more ‘difficult’ in absolute terms.

Maybe you need to pay for video tutorials when there’s a cheaper written option.

Maybe you need to schlep over to an in-person meetup because group work is the only way you stay motivated

Remember you are trying to learn something that will change your whole career path and help ensure you’ll die wealthy. Saving $300 on textbooks is not actually that smart a move.

5)  ‘No one does PHP any more’ and other advice you should ignore

Every developer is full of advice like this, and it’s the most critical to ignore. In general it is based on personal anecdote rather than real stats, (a student of mine had a friend tell her it was Very Difficult to get a first job programming if you lived in San Francisco!), but much more importantly: these judgements are about how good a particular language or framework is for job opportunities relative to other software jobs

I did tech support for almost five years. Answering phones and writing emails, I experienced every possible frustration and became a master of my craft. The most I ever made was $50,000 a year, and I was lucky to get that much. If I’d learned PHP backwards and forwards I could have doubled that salary within two years.

Heck I could have found a better job making honest-to-god video games

jobs in software working on games are notoriously the worst in the industry, but they beat the shit out of waiting tables. Pursue something you want to do in software!

I said at the start of this that this advice is for people who aren’t crazy successful. If you’re currently a well paid market analyst, you should think carefully about which software jobs would be a real improvement. For the rest of us, it’s all gravy so we should consider other factors than absolute demand.

5) try to focus on a few skills

This excellent advice is one that I absolutely failed to follow, and I know it’s hurt my career. had I focused on Data Science, or Front-end, or webservices, I would have an easier time now.

If you’re trying to break into the industry in the least time, you need to study on what a single person at a given job does, and spend most of your time on just those skills.

6) It’s a skill tree not a skill ladder -or- HTML is not the easiest thing to learn

the most common advice newbies get is to learn HTML and CSS first and it’s absolute horse shit. HTML and CSS are complex languages and while you’ll need a very basic knowledge of them at some point, they’re in no way more ‘basic’ than any backend language.

7) a few universals

Once you’ve sussed out a target role and chosen a language, there are a few skills that almost every job would like you to know. Try to do a few online courses and independent study that covers:

  • git
  • agile workflow methodologies (see a later point on how to get these without having a software job)
  • automated testing
  • interfacing with an API like the twitter or Salesforce api

Almost everything else is stuff that will vary between shops, and almost every other skill is something that a shop should be willing to teach you.

8) ways to break in

One of the hardest parts about getting a first job is that any professional work is so totally unlike lonely study: you’ll have to work in a team and document your work so others can re-use it. This is hard to simulate! But there are numerous opportunities to experience similar

  • game jams
  • taking small jobs via upwork (upwork sucks and the pay is shit, you should only do this to get experience of client relations)
  • ‘hack days,’ ‘code challenges,’ and ‘hackathons’

Events happen fairly often in larger cities, and you can find online events as well with a little googling. They’re great ways to get exposure to concepts like Kanban boards, bug documentation, and team collaboration.

The Javascript30 Day 1: adding instruments to the Drum Machine

I’ve long enjoyed his blog posts, so I was really excited for Wes Bos’s ‘Javascript 30’: 30 short tutorials on building cool stuff in vanilla JS. Since I’m not a real frontend developer, the lessons have included all kinds off stuff that was news to me, and beyond that it’s just fun to have fun coding.

I’m going to be posting extension ideas for the first few lessons, with an eye to adding each one to a portfolio. Day 1 is a drum machine that kicks off sound effects from keyboard input, and restarts the sound if you trigger it more than once.

screen shot of Wes's original drum machine page.

All I’ll do for this first project as extension is:
* change the background image to a local file
* add an instrument on a new keyboard button

New background!

To start with, I didn’t like a web request going out for a background image, and since the image was something grabbed from imgur, I didn’t really know its copyright status. No good for a portfolio, so I looked for a good online art archive that was releasing open-source images: turns out the Getty Museum has a great search tool for their archives

Okay, now I’ve downloaded the image I want to use, how do I place it as the background? I looked around the sample files and found this line in the CSS:

background:url(http://i.imgur.com/b9r5sEL.jpg) bottom center;

it took some digging to figure out the right syntax to link to a local image, turned out it was:

background: url("./images/dance-party-lo.jpg") bottom center;

note the double quotes and a ./ before the path. There are so few files in this thing it would be fine not to use an /images folder but that just felt too sloppy.

Okay, now that the image is replaced, time to add an instrument

adding a button

It’s always better to make changes incrementally. In this case, I’ll do two steps:
1. add a new keyboard button that plays the same instrument as another key
2. add a new sound effect for that button

Wes calls out in the tutorial you can use (keycode.info)[http://keycode.info/] to find every keyboard button’s code, and it seemed logical to add semicolon since all the other home-row buttons are there. I think the reason Wes didn’t do this originally is internationally issues: not every keyboard has ; to the right of L.

once I had the new keyboard code, there was a bunch of copy-paste as I added the button in the html. It worked first try!

new sound effect

Its kind of daunting trying to figure out what audio formats are supported, but all the other sound effects were .wav so I figured that would work. The Freesound Project makes its files available as .wavs, so I grabbed an appropriate ‘airhorn’ noise.

It wasn’t as confusing adding a new local sound, since the syntax was the same as the ones Wes already had. The final result looks nice:

image of new drum machine

Ideas for more stuff

  1. I tried making the sound play back at different pitches at each time. There’s even a call for ‘playback speed’ but it didn’t work the way I thought it was. Ended up dropping this
  2. some different arrangements would be cool
  3. change the look of a button the whole time the sound is playing

The shopify exporter!

When a friend asked “is there a way I can automatically backup the important info from my Shopify store?” I figured there had to be an easy way, and sure enough there were a ton of available plugins/services to do just that. Since they’d asked me to do it myself I’d much rather write code than buy a thing and try and configure it. So I wrote a node app to grab it all and send it to S3. Behold the Shopify exporter!

Lessons learned:

  1. Heroku automatically lets you schedule node jobs! I thought I had to write all that code myself! Look out Twitter bots! 😛
  2. There really wasn’t a simple enough walkthrough of s3 uploads for me. They were all covering how to upload user-submitted data, which adds a whole front-end piece I wasn’t worried about.
  3. From the s3 dashboard there’s a button to upload files directly. A parenthetical warns you “Under 5 TB”. lol

Upgrade your security in 10 minutes

The text of a 10-minute talk first given to donut-js in November 2016. Here’s a .pdf of the presentation

Where do we start?

Normally, when we talk about security, we have to start with a discussion of a ‘threat model’

‘Threat Model’?

A threat model helps us decide which security measures to use, taking into account:

  • likelihood of attack
  • cost of a successful attack
  • reduction of likelihood given a countermeasure
  • cost of a countermeasure

We won’t be doing that in 10 minutes!

That’s okay, because the countermeasures here are all cheap!

  • Fast setup
  • no $$$ cost
  • no added hassle to doing ‘stuff’ online

Top 5 things to do

  1. Secure your email
  2. Effective Ad-Blocking
  3. (Windows) Automatic updates!
  4. Signal for secure conversations
  5. Secure your other services

1. Use ‘2-factor Authentication’ for email

Why: if someone accesses your email, they can probably get access to everything else

How: for Gmail, you’ll enable 2-factor Authentication, and install the ‘authenticator’ app on your phone.

2. Effective Ad-Blocking

Why: Beyond sites becoming readable, ads are the primary vector for privacy and security problems

How: Switch to Chrome or Firefox and install uBlock Origin

3. (Windows) Update and use Microsoft Antivirus

Why: Most attacks on computer secure target Windows PC’s that are not fully up-to-date

How: A great guide is at decentsecurity.com, but in general:
* upgrade to windows 7 or higher
* use included “Windows Defender”
* enable automatic updates

4. Signal for secure conversations

Why: In a conversation via email or social media, all your security is worthless if the other person has been hacked

How: Search your app store for Signal and use it when you want your conversation to be private

5. 2-factor authorization for other services

Why: Once your email is secure, time to make sure no one can access your Facebook, Twitter, LinkedIn, etc without your permission!

How: A few services only have 2-factor login, including Snapchat and Squarecash, for others Google ‘enable 2 factor authentication on [social network]’

What’s next?

  • check out the EFF guides for different security needs with much better explanation of the how’s and why’s of each of these steps
  • Toby offers security consults to do some personalized threat modeling. These consultations are free if you’re a political activist or fear being targeted by hate groups.

Science Hack Day pt. 2: The stepper motor

My first post about Science Hack Day covered the fits and starts of working on the hackathon day and covers why the raspberry Pi turned out to be way less than ideal. This post is drawn from the last part of the hackathon, when we were driving the stepper, and what I learned about accuracy, and timing, in that experimentation

caveat: I am 100% not a mechanical engineer

Dozens of concerns were mentioned to me about the design of this tool, and I found a few others with a little googling, that implied we might have difficulty while trying to ‘drive’ the tool at the right speed with sufficient accuracy. None of these turned out to be actual problems, but I’m so ignorant on the subject I might not even have the right terms as I try to describe them. Sorry!

Okay what does this device even look like?

The process we’re talking about is stereotactic surgery, where bones are used to lock in a coordinate system for finding an exact point in soft tissue. The term could refer to any kind of surgical procedure, but usually we’re talking about the brain, since the skull is such a convenient ‘hard point’ to start your coordinate system.

To perform surgeries like this, they use a tool like this one:

The instrument. Image taken from lab supply company http://www.stoeltingco.com/
The instrument. Image taken from lab supply company http://www.stoeltingco.com/

The part of this that really matters are the two ‘knobs’ you see at the top left. The scientist twists these to move the probe. If you’ve ever worked with a 3d printer, this probably looks familiar. You’re looking at a hand-cranked 3d control system.

The upgraded version of this tool looks like this:

There are a lot of ways to 'computerize' this tool, but I chose this one since the conversion is more obvious. ibid
There are a lot of ways to ‘computerize’ this tool, but I chose this one since the conversion is more obvious. ibid

This version should look very familiar to someone who has worked with either a 3d printer or a CNC: the hand cranks are now motors, and a controller for those motors is hooked up to PC control.

The paper that Daniela found was about building a full 3d tool with open source hardware at a significant cost saving. She would find such a tool useful someday, but for the moment she just wanted one that moved on the Z-axis automatically, with manual controls for the X and Y.

So this whole goal is to build something that uses a stepper motor to turn a single knob.

Again, I had the advantage of a machinists’ work before the hack day to build the mechanical connection between motor and tool.

My concerns about the motor

  • Accuracy
    The probe needs to move ‘in’ by a total of 2mm over the course of a minute. Since we’re doing this in a surgery, it’s prooooooobably not good to end up moving 2.1mm. The ‘stepper’ part of stepper motor means it moves forward in discrete steps. My big fear was that the right distance would be between steps. If the system moved 3mm in 17 steps, there wouldn’t be a right number of steps to go exactly 2mm. Thankfully, since we were using a ‘geared’ stepper motor that steps down the speed of turning with a gearbox, this wasn’t a problem. A full rotation of the motor took a few thousand steps. Plenty of resolution.

  • Smoothness
    Again taking the example where the stepper moves the whole length over a dozen or so steps, this motion might look smooth to the naked eye, but microscopically they’d look like separate thrusts, which might not look good. Again, the resolution of steps resolved this.

  • Calibration
    I knew there were ways to look at the machines involved and calculate the expected distance of movement but 1) I didn’t know how to do that and 2) it sounded hard so I came up with the clever idea of just making the tool move, measuring the distance, and ‘calibrating’ to find the right number of steps to go 2 mm. At first I thought it would be tough to measure these tiny distances, but the version of the tool that Daniela brought had little dial readout of the distance of movement. In the end this was super easy!

  • Speed and sync
    The stepper driver in use probably had a spec available somewhere of the maximum rate it could accept instructions. The controller (raspberry Pi and/or Arduino) also have a maximum accuracy with which they can send signals out. One request from Daniela was that the probe move faster coming back out than it did going in, and I was worried that this higher speed might end up with signals ‘lost’. If you send ‘step’ signals faster than the motor can actually get them, you could miss steps and end up not moving the full distance.

I tested this extensively by throwing a ton of steps at the system at speeds much higher than required. Even with the Pi, which should have much more difficulty sending signals at high frequency, had no trouble. This wasn’t that surprising, after all we were going super slowly.

  • heat
    Stepper motors eat up a lot of current. Both the motors themselves and the stepper driver get really hot
See that big metal bit toward the type? It's a rather large heatsink to dissipate the heat. Image from tronixstuff.com
See that big metal bit toward the top? It’s a rather large heatsink to keep this part cool. Image from tronixstuff.com

Where a lot of the other problems I came up with were obviated by the slow speed and high number of steps, energy usage and concomitant heat buildup are actually worse for a more accurate tool.

Stepper motors can accept a big range of voltage. The more voltage, the stronger the grip of their magnets and the more accurate each step would be. That meant we ran our stepper at its max allowed voltage. We actually killed two of the tiny ‘cute’ stepper motor drivers in our testing.

We ended up using a version of the driver above with a huge heatsink. The motor itself also got pretty hot hot. Similar rigs to do 3d printing have cooling fans mounted on the frame to cool the motor itself. But I calculated that on the hack day we moved the probe more distance, at a higher speed, than it would move over a year of use. The system didn’t melt, we were okay.

stuff other people mentioned

  • free-wheeling
    If you turn off all current to the coils of a stepper motor, it should allow the wheel to turn freely. This would be super duper bad if the surgical probe just started moving around during a procedure. Normally this is handled by the hardware/software driving the stepper motor: giving it no instructions actually leaves one coil on holding the motor. I started to do a bunch of checks on this before I tried just turning off the power to my motor entirely. It turned out that the gearbox and the probe hardware presented way too much resistance for the thing to move on its own

  • reversing
    I’m not going to go to deep on this, but when you try to move back the way you came, there’s some ‘slack’ in the gears that needs to be taken up, losing you some steps. This difference proved too small to measure, and the probe gets ‘zeroed’ before every procedure anyway, so a problem can’t build up over a bunch of procedures.

  • timing from the Pi
    See my previous post: the Raspberry Pi was totally capable of running the driver.

what we learned

Optimization of a system, whether that system is electical, mechanical, or software, is only really meaningful after you have a working prototype. Problems like heat on the stepper motor driver are easily predicted and are worth handling early. Other problems should come up in trial runs, and if not, any time spent preventing them was largely wasted.

So build, and test! When you test, test a lot of stuff. Try your best to simulate uses more extreme than the real world should throw at them, and document what you find and notice.

the next and final post about science hack day will cover the control software I wrote for the arduino, and why an open-source stereotactic surgical might not be that great an idea.

Science Hack Day!!! -or- The Raspberry Pi, Stepper Motors, and Man’s Folly

Science hack day Portland was, without a doubt, my favorite hackathon event I’ve ever attended. So many things to love about it:

  • Slack channel invite a few days before to get on the same page/find teams
  • lots of real scientists attending, some bringing real problems
  • generous spread of hardware
  • not organized 100% by dudes
  • food was amazing

The Plan

I signed up thinking I might make some simple electronic whatnot or maybe some kind of ‘science-related’ complete project. Like a… weather station? Those are still a thing, right? Thanks to the slack channel set up beforehand, I found a scientist who was trying to get a real piece of lab equipment made, and even had an open-source project that they were basing their project on. So I’d be making a tool to carefully manipulate a probe. As it pushes into… an animal’s brain.

Science hack day had officially gotten real.

The version that Daniela Saderi wanted for her lab wasn’t going to grind bone like the open-source original, it was just going to handle a probe, moving only on one axis. The good news was that she already had an Electrical Engineer and Machinist on the project, otherwise the coupling between the probe and the stepper motor would have taken me the whole weekend.

So she was bringing all the necessary parts, and a stepper motor connected to a probe manipulator, all I needed to do was design some software to control the thing.

The Huge Mistake

I decided to try and control the system with a Raspberry Pi. This was a huge mistake for reasons I’ll get into, but I picked it for what seemed like good reasons:

  1. Mozilla had generously donated 35 of the buggers to the event.
  2. the user (Daniela) wanted a lot of customization on the tool. Adjustable speed, depth, pausing and restarting, all needed to be controllable.

The question I got asked throughout was ‘why not use the Arduino?’ And I did my usual scoff at this kind of question: there’s always a better tool out there, just let me use what I have dammit!

But really, motivation was the complexity of the interface Daniela wanted. I knew that displays and button inputs existed for the Arduino, but I didn’t really know much about them, I didn’t have the hardware necessarily, and I felt like the programming would be more complex.

Prompting the user to enter a speed and depth on a command line interface is easy for me, but making some little LCD display data and handling button input both sounded… hard.

Arduino and Raspberry Pi side by side
The Raspberry Pi is a complete PC (below) next to an Arduino (above) which, while programmable, isn’t much smarter than the electronics in your microwave’s keypad.

Oh and also: while Johnny-Five lets you write in Javascript for the Arduino it doesn’t actually… work the way you think it does: it requires the Arduino be hooked to the PC. To write code that just runs on the Arduino, you gotta write C.

I’m not that terrified of swapping programming languages? But it just seemed fun to be able to get the thing working in Javascript! And Johnny-Five can run directly on the Raspberry Pi (of course it can, it’s a PC!). I even plugged Johnny-five on the Pi to a few people in Slack in the days leading up to the event.

Why that was a mistake

a PC is a PC

In the end, I never got Johnny-Five to work on the Pi, and ended up having to write the Raspberry Pi controls in Python. I’d never written any Python, so that wasn’t super.

Also the thing about using a super-compact computer to be your computer is it just took so much time getting that fukkin’ PC to work. I was all excited about a tutorial that would let me get the Pi on WiFi and let me SSH over to it, but one of the steps was to update the OS. Of course, the OS update took half an hour, and with other little problems with hardware and video it was noon before I was even really working on my Pi.

Compared to setting up a full-blown new PC 3 hours isn’t terrible but it really shows why people prefer making simple projects on the Arduino.

just one letter separates ‘networking’ from ‘not working’

One thing that REALLY bit me: I followed a lovely tutorial on getting the Pi set up for SSH and Johnny-Five development, and the last few steps did not work right. First, I couldn’t get Mongo to serve the Pi at a hostname, and wrestled with that for a while.

After that, I couldn’t get the fileshare set up with the method they described, and ended up having to use… something else (I’m sorry, I’m sorry, I have no idea what it was and all the results I found were for Samba and smb and that did not work for me)

Once I had that done, I was able to remote into the Pi and write code from there. With the fileshare I could use Atom on my laptop and not be stuck with whatever weird IDE the Pi had. Great right? Not so much.

Since I didn’t get Samba set up right, I was accessing the Pi by its IP address, and this got reassigned by the network once or twice during the event. When this happened my attempts to load new code failed silently, and the sequence I was triggering would fail because it was running code from 10 minutes ago.

When I write code and it doesn’t work, network issues preventing the system from getting the latest version of the code are not my first guess about the cause. As a result this cost me an hour of debugging before I figured out the real problem.

Oh Johnny why you gotta do me like you do

After all that, possibly related: I couldn’t get Johnny-Five to work. I would get a few steps forward and then hit new roadblocks. The final version would run the code and throw me into a REPL. Was that it working? It sent the signals. But by throwing me into the REPL I couldn’t do console logging like I expected. It sent me some error messages that I… couldn’t understand.

johnny-five

Johnny-Five is an awesome project and it has a ton of rad uses. There’s also an “Espruino” hardware platform that takes Javascript direction. There was actually a presentation on this at the hack day that I missed on the Flasher.js sdk to do just this.

But don’t knock the Raspberry Pi

One thing I heard a few times, even read from some people who were writing hardware control software for the Pi, is that the Pi can’t control a stepper motor. This concern turned out to be quite overblown, I’ll explain:

A stepper motor is a wheel of magnets surrounded on four sides by electro magnets. As the magnets around the edge switch off and on, the wheel inside moves forward.

Stepper Motor animation, described in the paragraph above

for precision, they’re the dominant design, since at low speed it’s a bit like rolling a square wheel: you’re never unsure exactly how many times that wheel has turned.

If you’re curious: steppers aren’t the best at everything since they’re very power inefficient. They also require a ton of power to stay still (this wasn’t a problem for me since our motor had a gearbox that required force to move, so our probe wasn’t just wobbling around when the motor was off.)

All of this to say: controlling a stepper motor directly requires you to turn off and on these magnets around the edge. Obviously, the switching has to be really precise to ‘drop’ the wheel from one magnet and ‘catch’ it with the next.

the Raspberry Pi is a cut down PC, and it lacks a bunch of hardware involved with timing.

Here a sidenote: the chips on a normal PC that precisely control timing and allow for multiple tasks to work on in parallel while remaining in sync is called a ‘clock chip’ and is unrelated to telling time. My Father has worked for several decades making this exact part for a large semiconductor company.

Since the Pi can’t sync up its signals perfectly, it can’t handle controlling these magnets.

Okay, but these are really strong magnets, surely we’re not going to run them with power directly from the board, right? At the very least, you’d want a transistor to let a little bit of current from your controller direct a bunch more to the motor.

closeup photo of the pins on an Arduino
The General-Purpose Input/Output or GPIO pins can do a lot, but they can’t power your toaster oven

And here’s where we come to a basic thing with hardware hacking: powering LED’s or getting button input are all things that these controllers can do directly, but for more complex stuff you have a choice!
* design your own fancy circuit
* send simpler signals from your microcontroller to specialized hardware.

These are the famous “shields” for the Arduino. A lot of them aren’t necessarily required to do whatever, but they make it a ton easier to write the software to control whatever.

In this case, there exist stepper motor drivers that take very simple signals instead of requiring you to control each magnetic coil. Set up right, these ‘drivers’ will just take signals on 2 pins: a ‘step’ signal and a ‘direction’ signal, and handle all the fiddly “turn on this magnet, then turn on the next one right away, no, NOW”

By the by, we borrowed a Stepper Motor Driver from… someone… on the day of the event and I haven’t actually found another one that works just like this. I’ll figure it out eventually, but anyway!

Lessons learned

doing too much at once makes debugging a nightmare

At my job, where I’m using a library I’m very familiar with, I still rarely write more than a few lines of code without checking some debug output. Trim a string of a few characters, escape it for HTML, and then send it in a JSON object? I’ll often console.log() both the input and the escaped string before I move on to step 3.

By contrast, at 1pm on the hack day, I was SSH’ed into a Raspberry Pi running a Linux build I’d never used before, with an unfamiliar Javascript Library, to make the Pi control a stepper motor driver, which I’d first heard of the day before.

The first attempts to turn the motor was 27 lines of code. Not even counting the other tools, this was extreme.

After an hour of frustration, I disconnected the Pi from the motor, and went back to basics: trying to make the Pi blink some LED’s

It was doing the easiest possible task that revealed that Johnny-Five wasn’t working correctly, and that Python was.

In the end

The hackathon was a single day. By the end of the day I had the actual probe hardware moving the exact right distance in, pausing, and then moving back out. This wasn’t really a minimum viable product, since it needed to wait on input before going to the next stage. Still pretty cool to know the hardware setup and the Pi could manage such precise control, to move a probe just 2mm in 100 seconds!

If I get to it, I’ll write up how I went from that to the Arduino version in both hardware and software. For now you can check out my code for in Javascript, Python, and C

The Ever-Elusive Full-text search -or- Postgres magic

Search is, by it’s very nature, a complex problem. Just when you feel proud of yourself for getting everything pertinent stored in a DB, getting the right columns indexed, making find operations fast and efficient, someone’s going to ask ‘now I want to search the complete text of every post, and-
Okay, so that’s difficult, right?

And you’ll see some pretty kludgey solutions out there: using google indexing for your site, or just restricting searches to titles rather than full text. What are the key points in getting a really complete site search feature working?
1) Do we really need full-text search?

A lot of the wizardry we’re going to talk about feels like just that: wizardry. It’s easy to get caught up in algorithmic speed testing before thinking about what the business needs of this search really are.
If we’re letting a user search our e-commerce site, searching the text of reviews might be super unhelpful.

If we’re going to end up wanting to manipulate these results later, it would be a lot easier to just have ‘search’ go through manually selected tags than full text. E.g: when the user searches for ‘Ray Ban Sunglasses’ we want to offer him the cheap (and higher margin) substitutes. Rather than putting in some weird exception to our full-text search system, it would be a lot easier to just have search look at tags first, then we could tag whatever we really wanted without having to put our ‘thumb on the scale.’
2) do we really really need full text search?

trying to search, and even worse live-search, to work with a complex data set is a complex task, and we need to have user stories that justify the time and effort.
One question most ‘search’ user stories need to answer: would an extract or frequency table work equally well?

To be more specific, what about exporting a frequency table of words from the full text, removing the particles, and just suggesting the ‘matches’ from the top 10,000 words? What about an object containing frequently searched terms, with an array of post ID’s? All of these could be indexed ahead of time, with hourly re-indexing, for super fast performance, they all look a lot like full-text search, and they’re easier to manipulate.

You’re still here?
Okay, so let’s talk about how to make Postgres really sing through full-text search.
1) Get your indexes tuned

I came to programming from professional card playing, not a CS program, so I don’t always have the theoretical knowledge I’d like on every subject.
I do know this though: the speed with which you can search a set depends a lot on what you know about the set before hand
A simple add index for any column you’ll repeatedly query against is a really smart first step, but you should go back and mark every index for columns that you know are unique.
CREATE UNIQUE INDEX name ON table (column [, …]);
The reason why this helps is pretty easy to imagine: even with an index that shows you where you can find records of type ‘x’ is great, but it saves a ton of time if you know this listing is the only listing of its type in the index.
2) make what structure you can

I call this the ‘title subtitle optimization:’ if you’re concerned about search performance later, as much as possible give your stored information structure. That means if each post has a title and a subtitle, store them in separate columns rather than one text blob with a few tags.
The advantage of this approach mainly comes up in page ranking: it’s kind of a gimme that every match to a page title should be ranked above pages with the search term deep in the description.
Also: the shorter the blobs you’re searching in Postgres the faster your results will be.
3) your data has structure whether you know it or not

tl:dr version: use JSONB to store arbitrary structured data
It’s not actually possible to index (or for that matter compress) fully random data but good news: you’re a thinking machine not a bag o’ dice, you really can’t produce random output even if you wanted to.
JSONB lets you store structured data that can be indexed without knowing anything about its structure. and vastly improves performance
4) GIN indexing is your friend

I’d go so far as to call it your best friend. While it can index arbitrary structured data as JSONB, it’ll also let you index arbitrary text
CREATE INDEX idxgin ON api USING GIN (jdoc);

gin indexing was significantly improved in 9.4
Remember that you’ll need to format the query to tsvector to get to actually use this index. e.g.
NEW.tsv = to_tsvector(array_to_string(array( select json_array_elements(NEW.items)->>’name’ ),’ ‘));
I could not have written this article, nor had my career in programming, without the writing of David Bryant Copeland on Postgres

Watch a Rails app wither

My favorite parts of visualizations that show a commit history are the moments of destruction: that moment when whole trees are zapped into oblivion.

I don’t have anything so dramatic for my latest Rails toy but it WAS fun to make a full-fledged rails app and slowly winnow it down to just an API serving stuff to Angular.

I’ve talked about Angular a few times and we all know the total teardown coming with Angular 2 has made its future look grim. For the moment it was kinda cool to see how well it handled half a dozen PUTs all at once while keeping the UI clean, and ui-router module makes pretty neat multi-part forms.

Re-factoring

Always wanted to travel back in time to try fighting a younger version of yourself? Software development is the career for you!
-@Loh

It’s so weird to come back to something you were happy with, so happy with you posted it to your site, and say ‘gah, this is terrible.’ There were so many things wrong with that last calculator.

It’s super important to say here that there is no absolute way to determine whether code is good or bad. Beyond ‘working’ vs. ‘not working’ there are a few other standards: code that is easy for most other developers to understand, that handles error conditions in a predictable way, code that uses fewer server resources to run. Every thing except ‘doesn’t run’ is just a matter of degree, and often you gain performance by sacrificing readability etc. etc. My gripes about my own code are more a list ‘red flags’ than any absolute problem

Let’s talk about some ways to evaluate code that is in general, not so good:

  • tons of comments – comments are good and helpful when they are good and helpful. If every third line of your code needs explanation, there’s a chance your code path is pretty convoluted

  • multiple safeties – my first version had two different ‘checker’ methods for a few dozen code lines. Two methods to check one line of user input isn’t great :/ At a basic level: methods have error conditions already, or ways to handle weird input. Double-triple-checking that you’re not calling pop() on an empty array isn’t actually necessary, just make your other functions handle undefined.

  • long – I’ve talked about this before, hell everyone has talked about it, but there’s usually a good chance that going from 60 to 12 lines for the same functionality will be easier to read and understand.

so here’s the ‘better Reverse Polish Calculator’:

var stack = []
function handleInput(expressions){
  if (expressions == "q"){
    console.log("recieved 'q,' quitting")
    process.exit(0)
  }
  var inputArray = expressions.split(' ')
  inputArray.forEach(function (expression){
    if (/\d/.test(expression)){
      stack.push(parseFloat(expression))
    }
    switch (expression) {
      case '+':
        stack.push(stack.pop() + stack.pop())
        break
      case '-':
        stack.push(stack.pop() - stack.pop())
        break
      case '*':
        stack.push(stack.pop() * stack.pop())
        break
      case '/':
        stack.push(stack.pop() / stack.pop())
        break
    }
    console.log(stack)
  })
}


process.stdin.resume()
process.stdin.setEncoding('utf8')
process.stdout.write('\033[32m> \033[0m') //input prompt
process.stdin.on('data', function(datum) {
  handleInput(datum.trim())
  process.stdout.write('\033[32m> \033[0m')
});


One thing I love about this refactoring is it doesn’t actually do anything new. I did some research into whether it would be possible to just use the "+" as an operator without having to recognize the string, but there was no easy way to get it done.

My own Postfix Calculator

Job hunt time and that means writing some code to do random stuff to things!

Given the requirements that the code to do this be ‘readable, easy to extend, and understand’ I wrote a Reverse Polish Notation calculator.

//reminder: 'operand' == some kinda number, 'operator' == a math symbol
var operandCount = 0 //for checking if we have > 2 operands to do stuff to
var stack = [] //list of everything entered, with operands as floats and operators as strings
var validCharacters = ['1', '2', '3', '4', '3', '4', '5', '6', '7', '8', '9', '10',
' ', '*', '+', '/', '-' ]
function handleInput (datum){
  if (datum == 'q'){
    console.log("recieved 'q,' quitting")
    process.exit(0);
  }
  var response = ''
  if (!checkInvalid(datum)){
    console.log ('Sorry, you can only enter '+validCharacters)
  } else { //the input wasn't invalid characters
    datum = datum.replace(/^\s*|\s*$/, '') //Pattern's just removing initial or terminal whitespace
    //if more thn one value was given at once, grab each
    //TODO: if you enter more than one value the 'operandCount' checker will
    //become inaccurate. Basically the entering of multiple values isn't fully
    //supported in this version.
    datum.split(' ').forEach(function (value){stack.unshift(value)})
    var lastVal = stack.shift()
    if (isNumeric(lastVal)){
      //console.log('oh my, that's a number')
      operandCount++
      stack.unshift (parseFloat(lastVal))
    } else{
      if (operandCount < 2){
        console.log ('please enter '+(2-operandCount)+' more value!')
      } else {
        operandCount -- //one operand is destroyed in any function
        switch (lastVal){
          case '+' : performAddition() ; break
          case '/' : performDivision() ; break
          case '-' : performSubtraction() ; break
          case '*' : performMultiplication() ; break
        }

      }
    }


  }
  response = stack[0]
  console.log (response)
}

//this function could be written on one line with functional JS but it would be harder to read
function checkInvalid(datum){
  var reply = true
  datum.trim().split('').forEach(function(character){
    if (validCharacters.indexOf(character) < 0){
      reply = false
    }
  })
  return reply
}

function isNumeric(n){
  n = parseFloat(n) //everything's coming in as a string from stdin.on('data') :shrug emoji:
  return (typeof n == 'number' && !isNaN(n))
}
//are you at all worried I'm shift()ing the operator here? nope! I shifted it already
function performAddition (){
  stack.unshift(stack.shift() + stack.shift())
}

function performSubtraction (){
  stack.unshift(stack.shift() - stack.shift())
}

function performMultiplication (){
  stack.unshift(stack.shift() * stack.shift())
}

function performDivision (){
  stack.unshift(stack.shift() / stack.shift())
}





process.stdin.resume();
process.stdin.setEncoding('utf8');
process.stdout.write('\033[32m> \033[0m'); //input prompt
process.stdin.on('data', function(datum) {
  handleInput(datum.trim())
  process.stdout.write('\033[32m> \033[0m');
});

Install

  1. Make Node work on your system
  2. download this script to TFPostfixCalc.js
  3. run node TFPostFixCalc.js
  4. enter numbers or operators into the prompt q or EOF should both quit

TODO

In order to seem like a real human being I left one of the issues as a TODO in the code, because if you enter 10 42 the code will sorta work, but to improve it I’d also have to handle 10 * where I’m not totally clear on for the standard… I think it’s just an error?

Other stuff I don’t love:

  • all those separate functions for the operators could at least be written more densely
  • several checks could be rewritten with chaining and Functional patterns to be way more dense, but this is the way I write it first and I think this ‘fluffy’ version is easier to read
  • I feel it’s correct on bad input to just ignore the last entered line and maintain the state otherwise. Wasn’t in the spec, just said ‘won’t accept junk’ so :shrug emoji:. The other option would be to quit at that point.
  • Might want to color the output? IDK hardly major…