Unified Log Aggregation Tool for EKS Services

The Client

Dataddo is a cloud-based, no-code data integration and management platform that works with almost any online data service. The Dataddo app can transform and transfer data to a wide range of databases, DWH solutions, cloud storages and dashboards.

Challenge

Dataddo was aiming to migrate all of their microservices to Kubernetes to improve the stability of their services, reliability and lower their time to production. We achieved all of the above by utilizing AWS EKS  along with multiple add-ons that we supply as part of our reference architecture – LARA. One of the challenges that we had to face along the way was log aggregation. We needed to gather a large amount of logs in a very dynamic system composed from multiple customer services and system add-ons that we supplied. 

Tech Stack

• AWS EKS, Aurora, ElastiCache, OpenSearch
• Go, PHP, Vue.js
• PostgreSQL, MySQL, MongoDB, Redis, RabbitMQ
• Terraform, Helm
• GitlabCI, ArgoCD, Prometheus, Thanos, Grafana

Solution

We chose this solution to aggregate logs from all of the services running in a particular environment. We worked with the customer engineering team to enable all of the services to dispatch logs in a standardized format and then implemented a log collection pipeline using a tool called vector.dev. We chose this tool because of its low resource profile and efficiency. This approach enabled us to collect logs using a DaemonSet in Kubernetes, meaning we gather all of the logs on a per-node basis. For some services that could require a specific configuration for labeling or log processing we utilize log collector sidecars, which  run alongside individual services. 

AWS OpenSearch Service provided an easy to use solution, without additional overhead. The customer is able to view logs from all of the services using OpenSearch dashboards, which is integrated into the AWS OpenSearch Service. 

The solution was replicated to every environment the customer uses to run their services.