As a data strategy and engineering firm, our work involves processing large data sets and building complex applications to visualize and manage workflows. Over the last 10 years, we’ve used a number of different technologies and technical stacks; however, in the last four years, we've relied on a proven set of tools that has provided dependable results.
Here’s the 2019 snapshot of the tools, technologies and services we’re using the most.
When we evaluate new technologies to use with our customers, we focus on four main criteria.
Open-source: We believe in open source software. Over the years, most of our work wouldn’t be possible without the open source community, and so whenever possible we look to use open-source projects, and to create our own.
Automated: Because we are a boutique consultancy, time is our most important asset. We look for technologies and services that support a high-level of automation, and include APIs and other interfaces through which we can build our own workflow tools.
Large user base: We like software and tools that a lot of other people are also using. This is a strong indication that we can find fixes for bugs we run into, and that the codebase is being maintained and improved.
Flexible: Lastly, we seek out tools that give us the flexibility to use in many different ways. We are most successfully when we can bring a proven approach to solving new problems.
Language and Application Frameworks
For application functionality, we largely rely on our own Nxus framework for server side components, and React or Polymer (soon to be Lit-element) for client side web components.
Visualization & Mapping
Does data exist if you can’t see it? We use a number of different solutions for visualizing and displaying data in various ways.
D3 for complicated graphics and geo projection. When a custom chart component is required, we turn to D3, the industry standard open-source library for complex data-driven visualizations.
MapBox/Leaflet for maps. Maps are a great way to add context and show geographic data. We use MapBox for our basemaps, and leaflet to draw custom geographies.
Since so much of our work begins with data management, we rely on three core technologies to store data in different ways.
MongoDB: We use Mongo as a database Swiss Army knife. Mongo has great capabilities for rapid-prototyping and allows us to flexibly iterate our data model without getting stuck writing complex migrations. Mongo also includes a pretty robust set of queries for geo information to power maps and data analysis.
ElasticSearch: When we need to provide a search interface, we rely on ElasticSearch. ES is a flexible, fast tool that enables full-text and fuzzy matching.
Redis: When we need job-queueing or other fast access key-store, we turn to Redis.
As a boutique consultancy, we are very careful with how we spend our time. One area that can take a lot of time is managing the infrastructure for our applications, so we are always looking for tools that simplify our workflow and management tasks.
For years, we used AWS as our primary hosting solution, but a few years ago we transitioned to Heroku as our managed hosting provider. The thing that sets Heroku apart for us is the ease with which we can configure clusters of servers to handle large data processing tasks.
Heroku also includes a fast growing set of Continuous Integration technologies along with a strong set of add-ons from other providers. Because of the tightly integrated ecosystem, we can deploy a fully functional system with all the software dependencies we need in minutes.