A Serverless Music Sync App
For one of our clients we were asked to supply a small team of experts to build a bespoke music sync application. The application would (very briefly)
- allow users so log-in using OAuth providers (AWS Cognito and Okta)
- allow users to upload to their library, search, create playlists, and share tracks with other users
- allow users to "pitch" playlists to third-parties, who can then stream and download these
- collect detailed usage metrics and store these in a data-lake so that they can be displayed by BI dashboard applications such as Tableau or AWS Quicksight.
In addition we determined that the application should
- be fully scripted in infrastructure-as-code (IAC) so that it could easily be deployed in any AWS environment
- utilise serverless technology, which would drive down costs, simplify infrastructure, lower maintenance costs, auto-scale, etc.
- be organised according to a micro-services architecture
- send allow logs to an ELK instance to simplify monitoring and debugging
- utilise AWS XRAY for analysis of the performance, and help debugging, of the micro-services
How we did it
We scripted the infrastructure-as-code using a combination of Terraform (for the persistent resources such as databases, shared IAM policies, shared API gateway, etc.), and Serverless Framework for the individual micro-services.
The code was laid out in a mono-repo.
For full-text search we chose the AWS offering of Elastic Search.
We ensured the Elasticsearch-index would be kept in sync with the database by utilising an architecture where an AWS Aurora trigger would send data to a lambda which would, via an SNS & SQS, trigger a lambda which in turn populated the index.
Application logs are automatically sent to Cloudwatch Logs when using AWS Lambda. To then stream them to ELK we utilized Cloudwatch Logs Subscription Filters.
The micro-services architecture we applied utilised what we called a "middle layer", responsible for keeping state such as sessions. The layer also maintains a cache layer (using AWS Redis) to cache calls to the micro-services, which mostly consisted of basic CRUD operations. The layer also allowed for aggregation of requests to the underlying layer of micro-services.