So, what is SAYN? In simple terms, SAYN is an open source data processing framework. We have built it to be the simplest framework whilst maintaining full flexibility. Users can select from multiple predefined task types and build their own ETL processes. SAYN is really unique and unlike anything you have seen before. Want to know more? Then read on!
Before we speak more about SAYN, let’s start with a quick refresher to place things in context. Modern analytics infrastructures are usually organised around a data warehouse using five core layers as shown on the following graph:
It is crucial to have an efficient and scalable data process in place that can easily support the creation and maintenance of hundreds and more tasks and their dependencies. There are two common ways to go about this:
What if you want to maintain high flexibility in your analytics processes at scale, but also keep things simple? Well, as it turns out, we never found a data processing framework that efficiently addressed those concerns. So we built it!
We believe simplicity to be crucial when maintaining pipelines at scale. However, we also believe that simplicity should not come at the expense of flexibility. This is why we have built our own open source data processing framework: SAYN. SAYN is designed to empower analytics teams by being simple, flexible and centralised. It democratises the contribution to data processes within an analytics team, enables full flexibility and helps save a lot of time through automation.
SAYN is built around the concept of tasks and currently has the following task types pre-built for you:
The following graph displays how we typically use SAYN in the modern analytics stack. The blue lines are orchestrated by SAYN:
SAYN is designed around three core beliefs that a modern data processing framework should empower data engineers and analysts by being simple, flexible and centralised. This is how SAYN lives up to that promise:
The best way to see how great SAYN is is to actually try it! SAYN is distributed on PyPi and works using the command line. It is executed using the sayn run command. You can literally get started in 2 minutes with the following four lines:
$ pip install sayn $ sayn init test_sayn $ cd test_sayn $ sayn run
This will install the sayn package, create a SAYN project called test_sayn, move you into the project directory and then execute SAYN. You should see the following happening:
As mentioned before, SAYN projects are organised around the concept of tasks:
Here are some example use cases of SAYN:
If you want to understand more about how SAYN works, go through our tutorials which are good starting points.
We are actively developing SAYN and it is getting even better by the day! SAYN has made our lives so much easier at 173Tech and it really unleashes our analytics proficiency. Your team can benefit from it as well! In addition, we would love to get feedback that can help us make the framework even better so please do reach out, we’re friendly :) You can contact us for questions or suggestions regarding SAYN via firstname.lastname@example.org. Speak soon!
By subscribing, you agree to receive a monthly newsletter from 173Tech.