Erlang TDD Example

Test driven development is truly a very different and useful approach towards development in which you think of the result and define the tests first which gives you clarity not only about the expectation from your program but also the different scenarios that needs to be handled.

Generally test driven development is easy if you have well defined API or contract, so that the tests written are stable throughout the development timeframe. Like the design pattern recommendation to prefer interface over implementation, test driven development also needs to be against the interface as much as possible.

To see it in action, here is an example of test driven development for Erlang - simple movie reservation system which has the following 3 API requirement - register movie, reserve seat and retrieve reservation. Apart from these simple API, support for basic auth is also required for API calls, but no authorisation requirement is there.

Requests

Request type   | Key            | Description
========================================================================
               | imdbId         | IMDB movie identifier
 register      | availableSeats | Total seats available for this movie
               | screenId       | Externally managed identifier of
               |                | when and where the movie is screened
---------------+----------------+---------------------------------------
               | imdbId         | IMDB movie identifier
 reserve       | screenId       | Externally managed identifier of
               |                | when and where the movie is screened
---------------+----------------+---------------------------------------
               | imdbId         | IMDB movie identifier
 retrieve      | screenId       | Externally managed identifier of
               |                | when and where the movie is screened 
---------------+----------------+---------------------------------------

Responses

Request type | Response                       | Description
==============================================================================
             | ok                             | Registration was successful
             | {error, missing_fields}        | Fields missing in the request
 register    | {error, not_allowed}           | Registration is not allowed
             | {error, exists}                | Movie already exists
             | {error, Error}                 | Registration failed
-------------+--------------------------------+-------------------------------
             | ok                             | Reservation was successful
             | {error, missing_fields}        | Fields missing in the request
 reserve     | {error, not_exists}            | Movie was not registered
             | {error, not_allowed}           | Reservation is not allowed
             | {error, not_available}         | Seat not available for movie
             | {error, Error}                 | Reservation failed
-------------+--------------------------------+-------------------------------
             | {ok,                           | Retrieval was successful
             |  {"imdbId": "tt0111161",       | IMDB movie identifier
             |   "screenId": "screen_123456", | Externally managed identifier
             |   "movieTitle": "The Movie",   | Movie title
             |   "availableSeats": 100,       | Total seats available
             |   "reservedSeats": 50          | Total seats reserved
             |  }                             |
             | }                              |
             | {error, missing_fields}        | Fields missing in the request
 retrieve    | {error, not_exists}            | Movie was not registered
             | {error, not_allowed}           | Retrieval is not allowed
             | {error, Error}                 | Retrieval failed
-------------+--------------------------------+-------------------------------

To implement this reservation system, when you use test driven development, it will require you to first write tests for API which is the functional aspect. It also requires you to provide tests for your non-functional aspects like fault tolerance using supervisor.

From implementation point of view, following functionalities are implemented

• REST API using cowboy web server
• Data storage using mnesia

Here is the complete code for this example of test driven development - https://github.com/agrawalakhil/erlang_movies

Tech Disruption - Technology Disrupting Different Sectors

It is always interesting to see the past and learn from that, the impact of technology on various sectors & industries has been tremendous in past 100 years and its only accelearting.

Disruption & Tech Disruption

• Disruption describes a process whereby a smaller company with fewer resources is able to successfully challenge established incumbent businesses.
• Tech Disruption describes a disruption which happens due to technology - it is the primary enabler, incubator, facilitator or the sole reason.

When does tech disruption happens ?

• Technology is mature enough
• Sector is ready for change
• Product timing matches the technology maturity & sector readiness

Disruption - Four Stages

• Disruption (introduce product with new point of view)
• Evolution (innovate rapidly along this new trajectory)
• Convergence (complete value proposition relative to legacy)
• Re-imagination (rethink the entire category)

Why tech disruption happens ?

Technologies evolving faster than ever changing the way things are done.

Good examples of tech disruption

• Low end market (niche & less valuable to large players) - Uber, Micromax
• New market (market doesn’t even exist before but need is there) - Square, Tally

Sector properties to predict readiness

Less regulated, Low CAPEX, no entry barrier, bad shape, urgent need, less disruption, open to technology, conducive policy, tech infrastructure

Technology properties to predict maturity

Low cost, skilled people, highly scalable, easy adoption, robust & stable, learning ecosystem, fast evolving, secure & wide application, applied to many sectors, no entry barrier, many capabilities

Kind of tech disruption

• Disrupting Business - businesses are much more open to change
• Disrupting Us - the way we live (work, interact, think)
• Disrupting Institution - change is slow but happening

Presentation & References

Got an opportunity to do a short presentation on tech disruption for the college students at IFIM during their IT Club inauguration. Here is the snap of the event & presentation.

• https://www.slideshare.net/digikrit/tech-disruption-technology-disrupting-different-sectors
• https://hbr.org/2015/12/what-is-disruptive-innovation
• https://techcrunch.com/2016/03/16/technology-is-disrupting-everything/

Erlang Supervision Trees

Erlang Supervision Trees - This presentation was made for the meet up on Erlang supervision trees which was arranged to discuss and understand how Erlang supervision trees helps bring fault tolerance, recovery and robustness to Erlang applications. The presentation covers following

• Basics - A supervisor is responsible for starting, stopping and monitoring its child processes. The basic idea of a supervisor is that it has to keep its child processes alive by restarting when necessary.

• Supervision Trees - Supervisors can supervise workers (leaf nodes) or other supervisors forming supervision tree, while workers should only be positioned under supervisor.

• Supervision Strategy - Supervision strategy consists of two steps - first to form the supervision tree and then providing restart strategy at each level to follow when child dies, which could affect other children as well.

• Complete examples - To look at the complete example of supervision tree from open source projects like rabbitmq and ejabberd to get some understanding of how supervisors are used in real world.

Here is the presentation for the Erlang Supervision Trees

https://www.slideshare.net/digikrit/erlang-supervision-trees

Here is the github repository for the basic and advanced examples of Erlang supervision tree

https://github.com/agrawalakhil/erlang-supervision-trees

Few Interesting Meetups - Erlang & Javascript

This month has been little busy, doing lot of networking with companies at incubation center as well as organizing second Erlang meetup discussing build tools and speaking on AngularJS at a Javascript meetup. Lot of these meetups were good from different perspectives, here are the details of the presentations prepared for these events.

Erlang Meetups - Bangalore Erlang-OTP-ians

Have been working on Erlang building scalable ad server from 2010 onwards and have built some depth on the topic, wanted to meet other people who were working on the erlang based development.

• Erlang Latest Version - This was the first erlang meetup I have hosted, more than 20 developers joined and lot of interesting discussion around
  • Extended Time Functionality
  • SSL & SSH Improvements
  • License Change
  • Performance & Scalability

  Here is the Erlang latest version & opensource projects presentation and snap of the event which was hosted at Bizense (my first startup) office on 23rd Apr, 2016.

  

• Erlang Build Tools - This presentation was made for the meetup on Erlang build tools which basically discussed on different build tools available in Erlang and simple examples for each. Here is the summary of build tools discussed
  • Emakefile - A make utility for Erlang providing set of functions similar to unix style make functions. This is packaged with Erlang distribution and is the default build tool.
  • Erlang.mk - It is an included file for GNU Make, meaning including it in a Makefile allows building project, building/fetching dependencies and more.
  • rebar & rebar3 - rebar is self contained Erlang script, easy to distribute and embed in the project, provides dependency management, version 3.x has lot of improvements over 2.x
  • Mix - Mix is a command line utility that manages elixir projects but can be used for managing Erlang projects as well.

  Here is the presentation for the Erlang Build Tools

• Erlang Supervision Trees - This was the last meetup on erlang I have hosted, here is blog post for it. The erlang community in India is very small, met & came to know many of the excellent erlang developers during these meetups. Here is the presentation for the Erlang Supervision Trees

Javascript

Have been associated with frontend development from the long time from college days even before jquery became popular, our product was built using angularjs framework and got an opportunity to discuss on how it works with other developers during a meetup event.

• AngularJS Anatomy & Directives - at the JS Meetup Event

  Present basic and advanced code examples for AngularJS along with an example on directives. The presentation is much more detailed as it was created for training employees on front end development in my previous company covering web development evolution & concepts, angularjs anatomy, demos, directives, testing & debugging. The presentation talked about the following

  • Evolution - Let’s start with an overview of the evolution of the web application development over time in the last 6 years from 2011 to 2016.
  • Concepts - Angular combines lot of good programming concepts together to create an effective & powerful web application development framework.
  • Anatomy - Anatomy of an AngularJS application - understanding structure of angular app with basic & advanced examples.
  • Demos - Get familar with wide range of demos on different aspects of AngularJS as well as complete apps and opensource projects.
  • Directives - Extend HTML with directives for your application specific functionality abstracted into a reusable & clean interface.
  • Testing & Debugging - How AngularJS helps in creating testable application and what tools are available for testing & debugging your app.

  Here is the AngularJS Anatomy & Directives presentation and snap of the event which was hosted at Calm.io office on 18th Jun, 2016

  

Lot of interesting technical discussions with different developers of varied background and experience. Our goal in coming months is to increase interest among developers around open source projects we are working on and also to keep the discussion going deeper on Erlang & Javascript.

Another interesting meetup I attended this month was Open API Meetup from AWS showing how to integrate with APIs from Exotel, Freshdesk & Reverie. Lot of insights around power of API from the three companies who have already solved very different but real world problems.

Lambda Architecture On Elasticsearch

Elasticsearch provides an extremely robust platform for building custom analytics application through flexible aggregations. Although, as data goes into 100gb range, the query performance starts to degrade, so a mechanism like lambda architecture is needed to ensure the queries are running fast (especially on fields with high cardinality) without increasing the infrastructure cost.

We have used similar architecture and design for a mobile advertising product, Adatrix. The raw data and batch views (created through background jobs) were both stored in elasticsearch in different indexes. UI was using only the batch views and the speed layer was not implemented as the requirement for real time statistics was not so much there, although as were using an in-memory database Aerospike, the speed layer could be implemented whenever a requirement comes up.

Cardinality is important factor in elasticsearch query performance and the fields with high cardinality seems to have higher performance degradation. In raw data, there is very high possibility of id field (like session id or message id) having high cardinality and statistics would have to be built on those fields.

Few important considerations for creating batch views

1. breaking batch views by time (like hourly, daily, weekly etc) helps in keeping the cardinality of these fields reasonable for queries which run on raw data

2. breaking batch views by dimension (master dimensions included for creating the batch view, for example if your data has attributes like location, device, advertiser, publisher, you can select some of those to create batch views and rest of the dimensions will be flattened)

Few important considerations for queries

• Direct queries on raw data (should not be done of very large data)

• Rolling up of summaries or batch views

• Batch views limits the kind of queries that can be done (as dimensions are flattened, cross dimension is not possible)

Also, for storing raw data on elasticsearch, few important optimizations are required

• only keeping reverse indexed data and not raw data (removing _source storage)

• keyed fields instead of raw text fields

• optimizing elasticsearch storage (removing _all, keeping cardinality of fields reasonable, if possible)

Lastly, one important consideration is with finding unique values for high cardinality fields like audience. The unique value cannot be rolled up, for example, the unique audience per day can’t be added to find unique per week. So for this, some approximate algorithm like HyperLogLog has to be used to find unique statistics for high cardinality fields.

Background - Lambda Architecture

In case the data generation per day goes above 3gb, your data generation is going into the big-data use case and now the most popular architecture for this kind of data generation is Lambda Architecture defined by Nathan Marz of Apache Storm (contributed by Twitter), more details provided in the end. Briefly, lambda architecture is layered architecture with speed layer (creating real time views for recent data), batch layer (raw data and creating batch views on older data) and serving layer (which combines queries through the batch views and realtime views).

Interesting video from Yieldbot (an advertising company)

• https://www.elastic.co/videos/building-lambda-architecture-elasticsearch-yieldbot
• https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html

Other important links

• https://en.wikipedia.org/wiki/Lambda_architecture
• https://www.infoq.com/interviews/marz-lambda-architecture/
• https://databricks.com/glossary/lambda-architecture
• https://eng.uber.com/kappa-architecture-data-stream-processing/