XOR Media

Puzzle Solver

2015-01-09T00:00:00-06:00

Backstory

My brother got me a pretty cool puzzle for Christmas which claims to have “Over 300,000 wrong ways to assemble the pieces, but only One right way!”. The only hint/instructions are that the final solution will be a 3x3 square with six points and six holes on the outside.

Both my wife and I played around with solving it for a decent amount of time each and together some, probably a total of a couple hours. We both could get 8 of the nine pieces placed (multiple times) but could never find a setup that would fit the final piece. In talking about the puzzle we discussed the brute force approach to solving it, trying every piece in every possible place and rotation, …

I realized that it would be pretty cool to actual code up a solution. So I took the piece and threw them all out in the same orientation on the kitchen floor and took a picture of them. I then cut them up with gimp, and used convert to generate the 90, 180, and 270 rotated version. After that it was a bit of html and css, and a decent amount of hacking around in js (thought it would be the best way to do it so that others could see/watch.) I started off making things more complicated than they needed to be trying to keep track of what’d be placed where. After that wrong turn I backtracked and realized that it would be much cleaner and simpler to just have the pieces know when they’ve been placed and try to place them again and just ask if that’s possible.

After that change and getting rid of the rest of the wrong-turn it probably took about 30m, maybe an hour to finish things up and what you see to the right is the result, and the live version is below. The example is running at 256 tries a second (if you’re browser is up to it.)

I was fully expecting the solver to run for a couple hours trying all of the possibilities, but by random chance I managed to lay out the pieces, cut them up, and enter them in to the code in an order that gets to the correct answer after only 3554 placements. So what could of been hours takes less than a minute.

It was thrown together as quickly and is in the style of a hack rather than a good/clean solution, but I’m reasonably happy with it. Let me know what you think.

Ticks: 0

String Truncate Middle With Ellipsis

2014-09-21T00:00:00-05:00

There are times when you need to middle truncate a string. In many cases it’s for UX/human purposes, though in some situations it’s the best way to generate unique string for a length-limited field. This is the case I ran in to recently in trying to automate submission of IAP to both Google Play and the App Store which require short unique names for each SKU.

The Setup

Consider the following titles, each of which is 39 characters long.

Midsomer Murders - Series 1 - Episode 1
Midsomer Murders - Series 1 - Episode 2
Midsomer Murders - Series 1 - Episode 3
Midsomer Murders - Series 1 - Episode 4
Midsomer Murders - Series 1 - Episode 5
Midsomer Murders - Series 2 - Episode 1
Midsomer Murders - Series 2 - Episode 2
Midsomer Murders - Series 2 - Episode 3
Midsomer Murders - Series 2 - Episode 4
Midsomer Murders - Series 2 - Episode 5
Midsomer Murders - Series 3 - Episode 1
Midsomer Murders - Series 3 - Episode 2
Midsomer Murders - Series 3 - Episode 3
Midsomer Murders - Series 3 - Episode 4
Midsomer Murders - Series 3 - Episode 5
...

Assume we had to fit these strings in to a field we had no control over that requires them to be 32 characters or less and unique, or perhaps we’re displaying them in a UI where there’s not enough room for the full title. The naive approach would be to truncate them to 32 characters in length and add ellipsis to make it clear that the title has been truncated.

Midsomer Murders - Series 1 -...
Midsomer Murders - Series 1 -...
...

That doesn’t work particularly well as it results in duplicates across each series. Taking a closer look at the format of the titles, which in this case are consistent, we notice that there’s two points that will uniquely identify an episode. The series number and the episode number. So what if we truncate things in the middle rather than the end.

Midsomer Murderi...1 - Episode 1
Midsomer Murders...1 - Episode 2
...

The code to do this is fairly simple and looks something like the following.

def truncate_middle(s, n):
    if len(s) <= n:
        # string is already short-enough
        return s
    # half of the size, minus the 3 .'s
    n_2 = int(n) / 2 - 3
    # whatever's left
    n_1 = n - n_2 - 3
    return '{0}...{1}'.format(s[:n_1], s[-n_2:])

This process isn’t perfect though as with a different set of titles it may truncate out the series number as result in duplicates. For UI purposes this may be acceptable (a best effort,) but for something that requires uniqueness won’t quite be enough. In my particular situation the items have hex UUID’s as unique identifiers so the simplest thing to do was to append a few characters of it to the end of the title before truncating. This for all practical purposes insures uniqueness. What other solutions can you think of?

Natural Sort Order with Zero Padding

2013-11-23T00:00:00-06:00

Which of the following lists is sorted in the most “natural” fashion?

A:
    Elementary Season 1 Episode 1
    Elementary Season 1 Episode 10
    Elementary Season 1 Episode 11
    Elementary Season 1 Episode 12
    Elementary Season 1 Episode 13
    Elementary Season 1 Episode 2
    Elementary Season 1 Episode 3
    Elementary Season 1 Episode 4
    Elementary Season 1 Episode 5
    Elementary Season 1 Episode 6
    Elementary Season 1 Episode 7
    Elementary Season 1 Episode 8
    Elementary Season 1 Episode 9

B:
    Elementary Season 1 Episode 1
    Elementary Season 1 Episode 2
    Elementary Season 1 Episode 3
    Elementary Season 1 Episode 4
    Elementary Season 1 Episode 5
    Elementary Season 1 Episode 6
    Elementary Season 1 Episode 7
    Elementary Season 1 Episode 8
    Elementary Season 1 Episode 9
    Elementary Season 1 Episode 10
    Elementary Season 1 Episode 11
    Elementary Season 1 Episode 12
    Elementary Season 1 Episode 13

The Problem

While the former is technically sorted, it’s ordering is mathematical, only taking in to account the character values at each index in turn. The latter which we’ll refer to as human ordering where higher level logic is applied to realized that “10” comes after “9” and not “1.” Note that we’re not talking about “The” or “A,” removal which is another interesting topic, we’re just focusing on numbers in strings here.

Back on track… The simple “trick” to get a mathematical sorting algorithm to result in the natural sort order is to zero pad the numbers in the strings before handing them off to sorting.

We’ll start with an example of zero padded strings so you can see what we’re after. If you run the following strings through a sorting algorithm they’ll come out in the expected order.

Elementary Season 0001 Episode 0001
Elementary Season 0001 Episode 0010
Elementary Season 0001 Episode 0002
Elementary Season 0001 Episode 0011

The Code

I’ve looked at several ways to do this using python and after a bit of somewhat scientific benchmarking landed on the use of SRE_Pattern.sub as the best balance of performance and simplicity.

from sys import maxint
import re

# optional '-' to support negative numbers
_num_re = re.compile(r'-?\d+')
# number of chars in the largest possible int
_maxint_digits = len(str(maxint))
# format for zero padding positive integers
_zero_pad_int_fmt = '{0:0' + str(_maxint_digits) + 'd}'
# / is 0 - 1, so that negative numbers will come before positive
_zero_pad_neg_int_fmt = '/{0:0' + str(_maxint_digits) + 'd}'


def _zero_pad(match):
    n = int(match.group(0))
    # if n is negative, we'll use the negative format and flip the number using
    # maxint so that -2 comes before -1, ...
    return _zero_pad_int_fmt.format(n) \
        if n > -1 else _zero_pad_neg_int_fmt.format(n + maxint)

def zero_pad_numbers(s):
    return _num_re.sub(_zero_pad, s)

To use zero_pad_numbers to sort we just provide it as the key function to sort/sorted. This transforms each element for the purpose of ordering them, but doesn’t modify the elements returned so they’re in the preferred ordering, but remain in their original form.

unordered = ['file12.txt', 'file3.txt', 'file1.txt', ...]
ordered = sorted(unordered, key=zero_pad_numbers)
# ordered
#    ['file1.txt', 'file3.txt', 'file12.txt']

What About Databases

It’s common to need to order things in this manner when they’re retrieved from a database. In most situations it’s not reasonable to pull all of data back in to the web server for sorting, nor is it practical to try and do the zero padding in the database/query.

The best route to take here is to add an extra column for sorting, i.e. name_sorted. You can then hook in to the save method of your object and compute the sorted version of name. This way you only incur the cost of zero padding once rather than every time the data is retrieved.

High Performance Web - Asynchronous HTTP

2013-09-02T00:00:00-05:00

Async-What

The secret to building high performance sites which depend on external web services is asynchronous HTTP. The trick to asynchronous HTTP (or anything with the exception of UI) is to avoid callback hell. Enter futures/promise objects. When used correctly they make doing substantial asynchronous IO, relatively straightforward.

It took me only a week or two of my tenure at Amazon to run across the mechanisms that allow its home page to depend on dozens of external (web) service calls without falling over and devolving in to utter chaos. Simplified for explanation, the page generation consists of two major phases. The first allows the components of the page to fire off their individual requests for data. Then comes page rendering. During this phase the pieces of the page are individually rendering, waiting if necessary, though ideally each section’s data is ready and waiting when it’s turn comes.

Reality Check

I’ll let you in on a secret. You’re not building Amazon.com, neither in complexity nor scale. If you get there you’ll have a whole team to build the sorts of frameworks they have to make what they do possible. What you can do is leverage a simplfied version of the same concept to improve the performance and stability of your projects.

requests-futures

python requests is a well done library and if you’re using python and need to make a HTTP request it’s the way to go. If you using requests and would like to explore an asynchronous model requests-futures (note: I’m the author) makes the process as straightforward as possible.

The idea for requests-futures came to me while sitting in a talk at PyCon 2013 that covered concurrent.futures. I started on it then and there and within a few hours of hacking had pushed it up to github as a public repo. Not a whole lot has changed since that first push, a few bug fixes and documentation improvements. It’s so simple thanks to concurrent.futures that it “just works.”

It’s probably time for an example. We’ll take the following set of serial requests and convert them to happen in parallel.

from requests import Session

session = Session()
# first requests starts and blocks until finished
response_one = session.get('http://httpbin.org/get')
# second request starts once first is finished
response_two = session.get('http://httpbin.org/get?foo=bar')
# both requests are complete
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)    

To make the same two requests in parallel we’ll switch to a FuturesSession and instead of the get call returning a response object it’ll return a Futures object. The result method can be called on that Futures object to retrieve the response. That’s it, the only noticeable API difference is that a future object is returned in place of the response.

from requests_futures.sessions import FuturesSession

session = FuturesSession()
# first request is started in the background
future_one = session.get('http://httpbin.org/get')
# second requests is started immediately as well
future_two = session.get('http://httpbin.org/get?foo=bar')
# wait for the first request to complete, if it hasn't already
response_one = future_one.result()
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
# wait for the second request to complete, if it hasn't already
response_two = future_two.result()
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)

Applying it to the Web

So that’s pretty simple, but how would we go about applying it in a real-world context, preferably one involved in rendering a web page. The following example will show how this can be applied in to a Django request.

...

def simple_view(request):

    # NOTE: I often used middleware to install a persistent session on to
    # the request object, but that's be omitted here
    session = FuturesSession()

    # these requests are being made to grab data that we'll pass in as
    # context to the template, since httpbin.org is over the internet, the
    # requests can take a non-trivial amount of time, good thing they're
    # both happening in parallel rather than one and then the other
    future_one = session.get('http://httpbin.org/get')
    future_two = session.get('http://httpbin.org/get?foo=bar')
    # we could have any number of requests happening here

    response_one = future_one.result()
    response_two = future_two.result()

    data = {'response_one': response_one.content,
            'response_two': response_two.content}
    return render_to_response('app/simple_view.html', data,
                              RequestContext(request))

Wrap-up

While simple to use, futures are powerful and the above examples just scratch the surface. You can see a few more examples in my post from a few months ago about statsd. There’s a lot more that we could go in to on the subject including: how to handle dependencies between requests, tracking performance/request-blocking, and general tuning of complex request flows. We’ll leave things here for now, but feel free to tell me what you’d like to hear more about in the comments below.

What Google gets right, and where it fails

2013-06-22T00:00:00-05:00

In the past couple days there’s been several articles about how Google’s interview process has failed. The headlines are overblown, inaccurate really, but to be expected giving the goal of getting clicks. In reality Peter Norvig was talking about counterintuitive findings. They were surprised to find that some of their best employees were not unanimously voted in. That makes a lot of sense given my personal experience with the Google interview process and my opinions of what matters in making software engineering hires.

In the end it all comes down to enthusiasm. My personal stab in the dark would put it’s weight at 75%. The rest is equally split between intelligence and the “right” way of thinking. GPA doesn’t matter. What school you attended doesn’t count for much. And yes brain teasers won’t tell you anything useful, at least beyond being a crude measure of raw intelligence. Google doesn’t ask brain teasers, nor does Amazon or any other decent tech company. I’m sure there are people going off script at both companies who do so, but it’s not coached in either place for good reason.

My personal experience with Google’s process is that it’s basically a final exam for a mid-level algorithms course at Stanford. In my opinion that’s where they’re going wrong, but not failing. Google hires really smart people, but the most intelligent people aren’t necessarily the best software engineers. And the best software engineers aren’t necessarily the most intelligent (I like to think I illustrate that axiom.)

I believe the key to finding the best people is to make the interview process as realistic as possible. By that I don’t necessarily mean asking a question/solving a problem that comes from stuff you really work on. Most of the time real problems are too intricate to fit nicely in an interview slot, it’s just not possible to transfer all of the relevant context. The next best thing is working through a simpler, in scope, problem together. Together is the key here. You need to ask a problem that the candidate won’t know the answer to. Something where she’ll make a couple false starts, need to ask a few questions, have a few things right, and importantly get a couple things wrong.

It’s easier said than done, but that sets the stage for what really matters. Yes she has to be decently intelligent and good at solving problems, but the key is… Is she’s having fun? Are you enjoying working with her and helping to move things along? Is a decent first pass at a workable solution emerging? Are good questions being asked? Would you dread coming to work everyday if this is what it involved, because more often than not that’s what it will involve.

Pure behavioral interviews will give you a half-decent read on the 75% of things that is summed up as enthusiasm, but it won’t do much for the other 25%. I’ve found, without fail, that the key is to get at both of those things in one go via the right process.

High Performance Web - Reducing Database Round Trips

2013-05-13T00:00:00-05:00

Background

There are two main sources of latency in the backend of web applications: rendering (HTML templating or data serialization) and IO (database or external service calls.) Today we’ll look at the latter and more specifically focus getting rid of extraneous database round trips. The fastest query possible is one that doesn’t happen.

Before we get in to the details I’ll mention the importance of schema, but won’t spend time on it here. No single factor will affect the overall responsiveness of your application, and more specifically your ability to tune it, than an effective data(base) schema. This is something we’ll return to with detail in the future, but I bring it up now as schema issues can often limit your options.

I’ll also mention caching. In an optimal situation the things we’ll talk about will help to reduce the amount of time required to serve a result that can’t be cached or is a cache miss, but just as reducing the number of database round trips by optimizing querying helps, avoiding them altogether via caching is a big win.

A final note, I’ll discuss this topic in the context of Python (Django,) but it applies equally well in any language and ORM. I cut my teeth with Java (Hibernate) and have worked with Perl (DBIx::Class,) Ruby (Rails,) and several others since.

The N+1 Selects Problem

The most common and egregious round-trip problem is the n+1 selects problem. In it’s simplest form it involves an object with children, a one-to-many relationship. A minimal example follows.

from django.db import models


class State(models.Model):
    name = models.CharField(max_length=64)
    country = models.ForeignKey(Country, related_name='states')

    class Meta:
        ordering = ('name',)


class City(models.Model):
    name = models.CharField(max_length=64)
    state = models.ForeignKey(State, related_name='cities')

    class Meta:
        ordering = ('name',)

We have states which have zero or more cities and our use case is to print out a nested list of states and their cities.

Alaska
    Anchorage
    Fairbanks
    Willow
California
    Berkeley
    Monterey
    Palo Alto
    San Diego
    San Francisco
    Santa Cruz
Kentucky
    Albany
    Monticello
    Lexington
    Louisville
    Somerset
    Stamping Ground

The code to accomplish this looks something like:

from django.shortcuts import render_to_response
from django.template.context import RequestContext
from locations.models import State

def list_locations(request):
    data = {'states': State.objects.all()}
    return render_to_response('list_locations.html', data,
                              RequestContext(request))

...
<ul>

</ul>
...

If we run the above code to generate an HTML page and look at the database queries, using django-debug-toolbar, we’ll see a single query to list the states and then a query for each of the states to retrieve the cities. With 3 states that’s not a huge deal, but if it were all 50 we’d be looking at 1 query, the +1, to get the states and 50, the n, to get all of the cities.

2N+1 (no that’s not a thing)

Before we fix the N+1 queries I want to add an additional piece of information for each state, it’s country. This one will go in the other direction on a one-to-many relationship. Each state will have exactly one country.

Alaska (United States)
...

...

class Country(models.Model):
    name = models.CharField(max_length=64)

class State(models.Model):
    name = models.CharField(max_length=64)
    country = models.ForeignKey(Country, related_name='states')

...

...
<li> ()
...

If we take a look at the SQL tab of django-debug-toolbar we’ll see there are now queries happening to pull in the country object for each state. Also note that we’re retrieving the same country over and over, since it’s shared by all of the states.

Now we have a couple interesting problems to solve, conveniently one for each of Django ORM’s primary solutions.

select_related

states = State.objects.select_related('country').all()

select_related works by joining in SQL between the primary object (state) and the other object (country.) In this case that will get rid of the extra queries to fetch the country objects for each state. So if a database round-trip takes 20ms to transit the network, run, and return we’ll see a N * 20ms speedup. For any reasonable size of N that’ll be noticeable.

The new query to retrieve the state objects will look something like the following.

SELECT ... FROM "locations_state"
    INNER JOIN "locations_country" ON
        ("locations_state"."country_id" = "locations_country"."id")
    ORDER BY "locations_state"."name" ASC
...

The above query replaces the original state fetching query and rids us of the secondary queries to retrieve the country objects. There is however a potential downside to solving the problem this way, namely we’re getting back the same country object repeatedly and thus having to send it’s columns through the ORM code over and over and creating lots of duplicate objects. We’ll come back to this in a moment.

One last thing before we move on, in Django’s ORM select_related does not work when there are multiple objects on the other end of an association. We can make use of it to grab the country for a state, but if we were to add ‘cities’ to the select_related call, nothing would happen. Other ORMs (Hibernate for example) don’t share this limitation, but you must be very careful when making use of it in those situations. In those cases the join will duplicate the primary object’s data for each secondary object. That can quickly balloon out of control and leave the ORM processing lots of data/rows.

Given all of this, one of the best uses of select_related is when you’re fetching a single object and want to pull back (a) related object in a single shot. It’ll do so in a single round-trip to the database and won’t end up duplicating too much data, any in the case of Django ORM’s one-to-one only support.

prefetch_related

states = State.objects.prefetch_related('country', 'cities').all()

In contrast prefetch_related works by collecting all of the ids for the related objects, fetches them in a single batch, and then transparently attaches them to the objects. The best part is that it can be used on one-to-many relationships, which is what the state to cities relationship is in our example.

The SQL will now look something like the following.

SELECT ... FROM "locations_state" ORDER BY "locations_state"."name" ASC
SELECT ... FROM "locations_country" WHERE "locations_country"."id" IN (1)
SELECT ... FROM "locations_city"
    WHERE "locations_city"."state_id" IN (1, 2, 3)
    ORDER BY "locations_city"."name" ASC

So we’ve gone from 2N+1 queries to 3. Getting rid of N is a big deal. 3 * 20ms is way less than (2 * 50 + 1) * 20ms or even (50 + 1) * 20ms in the case of using select_related alone.

The above example uses prefetch on both country and cities. As mentioned in the previous section all of the states share a single country and when using select_related that means we pulled back and processed its data N times along side each state object. In contrast using prefetch_related we’ll only pull back the country object once. This however comes at the cost of an extra db round trip so it may be preferable to use a combination, you’ll have to test in your specific circumstances. However, in this case using both select_related and prefetch_related takes us down to 2 * 20ms, it’ll probably be faster than 3 queries, but there are a lot of potential factors.

states = State.objects.select_related('country') \
    .prefetch_related('cities').all()

How Deep

What about when you want to span multiple levels? Both select_related and prefetch_related can traverse relationships using double-underscores. When you make use of the ability the intermediate objects will be included as well. This can be extremely powerful, but a little hard to grok in more complex object models.

# only works when there's a single object at each step
city = City.objects.select_related('state__country').all()[0]
# 1 query, no further db queries
print('{0} - {1} - {2}'.format(city.name, city.state.name,
                               city.state.country.name)

# works for both single and multiple object relationships
countries = Country.objects.prefetch_related('states__cities')
# 3 queries, no further db queries
for country in countries:
    for state in country.states:
        for city in state.cities:
            print('{0} - {1} - {2}'.format(city.name, city.state.name,
                                           city.state.country.name)

prefetch_related on raw queries

One final Django specific note. In last week’s post on efficiently querying for nearby things I worked through a complicated SQL query to find the closest objects to a given lat/lon point. The best way to go about doing this sort of query with Django is to use a raw sql query. The issue is that raw query sets don’t support prefetch_related, which is a bummer. There is however a work-around, you can directly make use of prefetch_related_objects, which Django uses under the hood to power prefetch_related.

from django.db.models.query import prefetch_related_objects

# prefetch_related_objects requires a list, it won't work on a QuerySet so
# we need to convert with list()
cities = list(City.objects.raw('<sql-query-for-nearby-cities>'))
prefetch_related_objects(cities, ('state__country',))
# 3 queries, no further db queries
for city in cities:
    print('{0} - {1} - {2}'.format(city.name, city.state.name,
                                   city.state.country.name)

That’s pretty cool.

Efficiently Querying for Nearby Things

2013-05-06T00:00:00-05:00

It’s a fairly common use case to have a latitude and longitude and want to find the closest objects to a given point. While there are heavyweight solutions: MySQL Spatial Extensions, PostGIS, they can be more trouble than they’re worth especially if you’re making use of an ORM.

Euclidean Distance (naive, flawed, but simple)

The most naive approach is to calculate the Euclidean distance between the objects and point. It works OK so long as the distances are small and you’re closer to the equator than you are to the poles. You will likely recognize the math from high school geometry class, a^2 + b^2 = c^2. A solution in sql (MySQL) might look something like the following.

CREATE TABLE Restaurant (
    id INTEGER AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(64) NOT NULL,
    lat DOUBLE NOT NULL,
    lon DOUBLE NOT NULL
);

set @point_lat = 37.757001;
set @point_lon = -122.420588;
set @radius = 100;
SELECT * from (SELECT id, name, lat, lon, 
               sqrt(pow(@point_lat - lat, 2) + pow(@point_lon - lon, 2)) 
               * 110574.61087757687 as distance
               FROM Restaurant) i
    WHERE distance < @radius ORDER BY distance ASC;

We’re computing the square root of the differences between the two points squared. If it’s less than our search radius we let it through and sort the passing Restaurants by their distance away from the search point. Note the use of a sub-select. Depending on the level of strictness you have MySQL running with (keep your eye out for an upcoming post) you cannot use an aggregate in a where clause. WHERE is applied before computing aggregates. Alternatively you could make use of HAVING, but I have chosen this route. The other thing that needs explaining is the number 110574.61087757687. It’s the distance in meters of a single degree of latitude. You’ll see it used throughout as we’ll be working in meters.

Performance (too much math)

There are two problems here, the first one is an issue of performance. As written the above query will run 2 subtractions, take two things to the power of 2, and then do a square root on every single row in the Restaurant table. As soon as you have a city’s worth of rows that’s going to be a performance problem and it’ll only get worse from there.

+----+-------------+------------+------+---------------+------+---------+------+-------+-----------------------------+
| id | select_type | table      | type | possible_keys | key  | key_len | ref  | rows  | Extra                       |
+----+-------------+------------+------+---------------+------+---------+------+-------+-----------------------------+
|  1 | PRIMARY     | <derived2> | ALL  | NULL          | NULL | NULL    | NULL | 35906 | Using where; Using filesort |
|  2 | DERIVED     | Restaurant | ALL  | NULL          | NULL | NULL    | NULL | 32160 |                             |
+----+-------------+------------+------+---------------+------+---------+------+-------+-----------------------------+
2 rows in set (0.08 sec)

There’s a relatively easy solution. We need to consider fewer rows and avoid doing as much computation as possible. The trick is to create a bounding box and only consider rows that fall inside of it. Since we are searching within a radius we know that anything that falls outside of a box with a width of 2 * radius centered around our point will fall outside of a circle of the same size.

set @r = (@radius / 110574.61087757687);
set @lat_min = @point_lat - @r;
set @lat_max = @point_lat + @r;
set @r = (@radius / 110574.61087757687);
set @lon_min = @point_lon - @r;
set @lon_max = @point_lon + @r;

SELECT * from (SELECT id, name, lat, lon, 
               sqrt(pow(@point_lat - lat, 2) + pow(@point_lon - lon, 2)) 
               * 110574.61087757687 as distance
               FROM Restaurant 
               WHERE lat BETWEEN @lat_min AND @lat_max AND
                     lon BETWEEN @lon_min AND @lon_max) i
    WHERE distance < @radius ORDER BY distance ASC;

We’ve added a where clause to the inner select that will filter out rows that have no chance at being inside our radius so when our search point is in San Francisco and we’re looking for stuff within 1000m we won’t bother doing the math for things in NYC, or even on the other side of town.

That’s not quite enough though. If we use explain plan we’ll see that a table scan is occurring to do the bounding box check. We’ll need an index to address this.

... explain query ...

+----+-------------+------------+------+---------------+------+---------+------+-------+----------+-----------------------------+
| id | select_type | table      | type | possible_keys | key  | key_len | ref  | rows  | filtered | Extra                       |
+----+-------------+------------+------+---------------+------+---------+------+-------+----------+-----------------------------+
|  1 | PRIMARY     | <derived2> | ALL  | NULL          | NULL | NULL    | NULL |   122 |   100.00 | Using where; Using filesort |
|  2 | DERIVED     | Restaurant | ALL  | NULL          | NULL | NULL    | NULL | 32160 |   100.00 | Using where                 |
+----+-------------+------------+------+---------------+------+---------+------+-------+----------+-----------------------------+
2 rows in set, 1 warning (0.05 sec)

CREATE INDEX Restaurant_lat_lon ON Restaurant(lat, lon);

... explain query ...

+----+-------------+------------+-------+--------------------+--------------------+---------+------+------+----------+-----------------------------+
| id | select_type | table      | type  | possible_keys      | key                | key_len | ref  | rows | filtered | Extra                       |
+----+-------------+------------+-------+--------------------+--------------------+---------+------+------+----------+-----------------------------+
|  1 | PRIMARY     | <derived2> | ALL   | NULL               | NULL               | NULL    | NULL |  122 |   100.00 | Using where; Using filesort |
|  2 | DERIVED     | Restaurant | range | Restaurant_lat_lon | Restaurant_lat_lon | 16      | NULL | 1043 |   100.00 | Using where                 |
+----+-------------+------------+-------+--------------------+--------------------+---------+------+------+----------+-----------------------------+
2 rows in set, 1 warning (0.01 sec)

Even in my tiny test dataset there’s over an order of magnitude reduction in the number of rows considered and it sped things up around 5x (though admittedly both are fast and two runs is hardly a scientific comparison.)

Correctness (now that’s much better)

So now that things are fast and we’re avoiding doing math as much as possible we can do a much better job at accurately computing the distance between points. The Earth is not flat, it’s a sphere. While a degree of latitude is constant anywhere it occurs, the same can’t be said for longitude. A degree of longitude is much smaller near the poles than the equator.

Enter the Haversine Formula which accurately computes great-circle distances, the distance between two points on a sphere, and while the Earth isn’t a perfect sphere it’s much closer to one than a plane. I won’t bother explaining the math, wikipedia already does so and I’d just mess it up. I will however translate it in to SQL for you. We’ll also adjust our bonding box calculation to account for longitudinal variation.

set @r = (@radius / 110574.61087757687);
set @lat_min = @point_lat - @r;
set @lat_max = @point_lat + @r;
set @r = (@radius / abs(cos(radians(@point_lat)) * 110574.61087757687));
set @lon_min = @point_lon - @r;
set @lon_max = @point_lon + @r;

SELECT * from (SELECT id, name, lat, lon, 12756200 *
               asin(sqrt(pow(sin(radians(@point_lat - lat) * 0.5), 2) +
                         cos(radians(@point_lat)) *
                         cos(radians(lat)) *
                         pow(sin(radians(@point_lon - lon) * 0.5), 2)))
               as distance FROM Restaurant
               WHERE lat BETWEEN @lat_min AND @lat_max AND
                     lon BETWEEN @lon_min AND @lon_max) i
    WHERE distance < @radius ORDER BY distance ASC;

If you look closely you’ll notice a new magic number in there, 12756200 which is the diameter of the earth in meters. This query will perform roughly the same as the previous performance enhanced version even though it’s doing more complicated math. If we weren’t filtering with a bounding box we’d see a much more apparent slow down.

I plan to follow this post up next with with one on how I implemented this solution with a Django model focusing on optimizing the queries run to fetch related data. (found here.)

Note: the foundation for my final solution was a MySQL presentation on the subject, but I found it to be lacking in explanation and there were quite a few things that could (needed to really) be optimized. If you’d like to take a look at that document it can be found here.

Keeping Track with Graphite and Statsd

2013-04-11T00:00:00-05:00

I recently gave a presentation on graphite and statsd to the East Bay Django meetup It’s more about the importance of stats in general when running services, web or otherwise. The slides are available and embedded here. There’s also a related demo project that turned out to be much more interesting than the slides themselves.

Django ALLOWED_HOSTS and Amazon Elastic Load Balancer

2013-02-27T00:00:00-06:00

A recent Django security release added a new configuration option, ALLOWED_HOSTS. It’s optional in 1.3 and 1.4, but required in 1.5 setups. The relevant detail is that when enabled in production Django will throw 500 errors if the Host header doesn’t match one of the values in ALLOWED_HOSTS.

The good news is that ELB preserves the Host header on requests it forwards, as it would have to. The bad news is that it doesn’t set a useful Host header during it’s health and latency checks which causes them to fail and mark the host as down. In fact, the two user different values.

The health check uses the instance’s internal IP address which can be determined and added to the ALLOWED_HOSTS list. The latency check on the other hand uses another IP address that doesn’t belong to any instances that I own. My guess would be that it’s the IP address of the ELB node itself, but I have not had a chance to test/verify that.

So by finding and adding the internal IP address to ALLOWED_HOSTS you can make the health check happy and at least get your instances serving requests, but the latency checks will still be 500’ing and there’s no clean way to fix that.

The health check problem coupled with the PITA of reliably getting the internal IP address of a host lead me to look for a better solution. What I ended up on involved munging the Host header if it looks like an IP address. To do this I added the following my nginx site config.

set $my_host $host;
# if the host header is an ip address change it to www.mysite.com
# this works around requests coming from ELB with either the instance's
# internal ip address in the case of health checks or an unknown internal
# ip address in the case of latency checks. translating them to a known
# good host header makes django's ALLOWED_HOSTS happy
if ($host ~ "\d+\.\d+\.\d+\.\d+") {
    set $my_host "www.mysite.com";
}

location @django {
    proxy_set_header Host $my_host;
    ...

The above initially sets $my_host to the $host header from ELB. It then checks to see if it looks like a 4-segment IP address. If it does set $my_host to the default “www.mysite.com.” Finally $my_host, whether defaulted or overridden, is passed along to Django.

Top 5 Percent

2013-02-11T00:00:00-06:00

LinkedIn does a pretty good job of email communication. I probably read 50% of the emails they send me and I’ve remained subscribed to them for quite a while now.

A few minutes ago I received one of the best emails they’ve sent me yet, subject:

“Ross, congratulations! You have one of the top 5% most viewed LinkedIn profiles for 2012!”

That’s pretty cool, no? I had to read that one. In the body of the message they drop the number of users they have, 200M, and repeat the 5%. Doing the math that means that I have one of the top 10M profiles. Heh, when put that way it doesn’t sound quite as good. It’s funny how top 5% sounds better than top 10M. Both are good numbers to be a part of, but 5% just sounds more exclusive.

The email had a Read More link pointing to a page that was relatively well done page with a little more detail and an infographic that breaks down user numbers by country. There are 74M users in US, so the top 5% there would come out to 3.7M. It feels better than 10M, but is still a large number.

All told this seems like a pretty good campaign for LinkedIn. It worked on me since I’m talking about it in this post. What’s more is there’s a pretty good chance that the 5% of most looked at profiles will include a lot of people with large followings. I’d assume the Bay Area has a disproportionate percentage of the top, at least in tech, given the supply constrained nature of things here.