Tech

Turf Tools is a well-engineered software system. Although it's free to use, it's state-of-the-art for performance in this software category.

There are a few key technical decisions that make it performant and enjoyable for both users and developers. These decisions have come from carefully considering the data workloads involved in these systems, evaluating and accepting good tradeoffs for a system built to purpose, and making joint decisions about product and architecture such that performance follows naturally from product decisions.

Columnar data on object storage

Turf Tools uses a split database system, with transactional application data living in PostgreSQL (segment definitions, script configurations, canvassing results, etc.) and voter data living in DuckLake, a columnar storage system. Columnar data is slow for individual voter lookups but much faster for ingestion, bulk processing, and filtering operations.

Because of this split storage system, uploading a new voter file into the system, geocoding it, and getting it ready to use takes just a few minutes on regular cloud hardware. Additionally, the use of object storage as the data backend supports horizontal scaling. Although we have not yet used it at this scale, parallelization over zip codes or other geographic features will make it possible to load and index in a few minutes whether running on a small town, New York City, or the whole country.

The choice of columnar storage with a modern query layer also lets us be more ambitious with our features and performance. The video below, for example, shows a user constructing a canvassing universe across the 5.3M registered voters in New York City. Each additional filtering operation provides immediate feedback and information within a few hundred milliseconds (just short of feeling instant).

It's cheap to compute the relationship between each filter, and as you stack and rearrange your filters, we show extra information about how they overlay and interact. This type of user experience would not be possible with a traditional database.

The use of object storage for voter data can also scale to support canvassing across an entire country. The compute-intensive work of data loading and filtering is arbitrarily scalable, while the transactional workload of logging in and recording results could be supported for an entire country from a single PostgreSQL database. So while Turf Tools is designed for anyone to easily spin it up, one instance could also support a massive user base.

In addition to DuckLake, Turf Tools builds an index of the user-provided voter data using Quickwit, which is a search system like Elasticsearch built on top of Tantivy. This index supports fast individual voter lookups. Unlike Elasticsearch, Quickwit uses a compute-storage separation architecture which means that an index can be built on one machine, uploaded to object storage, and then served from another machine. The use of Quickwit and offline index building allows the biggest stress on the operation to be horizontally scalable with no shared resource usage, perfectly suited to the fact that the system's workload has little need for online indexing of voter data, which arrives in big bulk batches as initial data loads or updates.

The geographic data model

Turf Tools does not rely on Mapbox or Google Maps for geocoding, and the automatic turf cutting does not operate on latitude/longitude coordinate pairs.

Instead, Turf Tools contains what we believe to be the fastest open source geocoding implementation in terms of throughput (rows processed within a fixed unit of time). As part of setting up a Turf Tools deployment, the relevant public geographic data — the US Census TIGER dataset and OpenStreetMap — is downloaded into the data lake. After loading voter data, addresses are mapped to TIGER street segments using normalized, scored matching and then assignments are refined using OpenStreetMap building footprints. For a performance example, when loading data in New York City, the match rate is 99% and the entire run, including geocoding, can finish in 90 seconds on good hardware.

The geocoding pipeline produces a good estimate for latitude and longitude, but those coordinates are mostly used for display on a map. The more important result is linking each address to a "blockface", which is our word for one half of one street (e.g., the left side of 5th Ave between 10th and 11th).

The blockface is the backbone of our automatic turf cutting algorithm. It allows us to model a neighborhood as a series of interconnected streets — similar to a traditional undirected graph — instead of a blob of two-dimensional points. We can then leverage the existing field of graph clustering algorithms for turf cutting. Our algorithms produce sets of continuous blockfaces optimized for doors per set and minimal walking distance, and as such, they suit the way canvassers actually move through neighborhoods, walking and turning and crossing the street, not flying like birds.

Zones for turf distribution and parallelization

Our system uses a geographic concept called a "Zone", inspired by both technical considerations and years of real-world organizing experience.

You can think of a zone as the geographic unit for a canvass launch: after meeting at the center of the zone, canvassers can leave from that spot and walk a reasonable distance to any turf within that zone. There may be other considerations to zone boundaries as well, such as limiting turfs to areas with the same down-ballot options, the same voting location, or the same language-specific literature.

Using the editor shown below, users can define zones separate from, and complementary to, the filtered universe of voters to target, making it possible to reuse zones across different universes. Zones are defined by combining smaller geographic units, like precincts or zip codes. It's also possible to preview the spatial density of a voter universe while defining zones.

Zones, once defined, guide the canvassing operation. For the administrator, each zone is either cut (has published turfs and can be canvassed) or not. You publish the turfs for an entire zone at once, and geographic operations like turf autocutting can be parallelized across zones since no individual turf crosses a zone boundary.

As the campaign progresses, rolling up activity by zone (e.g., percent of turfs walked in the zone, persuasion rates, etc.) provides operational metrics at the level of detail needed to assess field progress and goals, and to guide a volunteer turnout operation.

Turfs are immutable blobs

When an administrator clicks "Publish" after they are done manually cutting turf or using the auto cutter, the "Turfs" (collections of people in a convenient geographic layout to be walked by one or two canvassers) are built and written to the PostgreSQL database with everything needed for the client: all the person details and geographic information are bundled into a single JSON blob.

As a result of this design, loading the turf on a user’s device at the start of a canvass is a single read from the transactional database and the voter data does not need to be online. After that single blob is downloaded, the phone has all the voter and geographic data it needs for a canvass.

Beyond helping with performance and offline reliability, immutable turfs create useful constraints for thinking through the canvassing lifecycle. For example, when an administrator wants to publish new turfs after changing voter filtering, it marks the old ones as superseded, but without disrupting their use in case they are still being walked.

Performant open source web mapping

Maps are used throughout our system, from the administrator's view of an entire universe to the canvasser's view of building locations in a single turf. Throughout, we've used open source tools and pushed the boundaries of performance. All of our maps use the MapLibre library, an open source successor to Mapbox that uses vector tiles to render beautiful maps performantly, using WebGL (on desktop) or Metal and OpenGL via React Native (on devices).

The map views for administrators require rendering potentially millions of points and updating them dynamically as you filter voters, which can be a struggle for MapLibre on its own. For that use case, we built a custom WebGL point layer, and also designed a protocol for transferring binary point data from Python to the web. We believe this is the fastest possible way to get point data into the browser and rendered on a map, and it results in a smooth mapping experience with extremely large scales of data.