I’m also one but I don’t work for advertising. Most data engineers work for consulting companies that work for banks. We program automatic data processing pipelines. For example, bank transactions are stored somewhere, all the historic data, that needs processing to then be graphed out for exec number 3, or for whatever.
Other companies might send you files that need to be automatically processed, cleaned, and put correctly where then other tools can pull that data correctly.
We basically do all the background work concerning data manipulation. File processing, databases… all that stuff. And by databases it can be normal ones like posture to distributed ones like hdfs/hive/athena/whatever.
Ad world is basically the same but with tracking info instead of transactions.
If you are interested in day to day work, it’s a mix of coding SQL processes, then porting them to spark/pyspark for distributed massive processing. There are new shiny tools for those that don’t know much of the technical side to manage, sorta.
*data engineer
I’m also one but I don’t work for advertising. Most data engineers work for consulting companies that work for banks. We program automatic data processing pipelines. For example, bank transactions are stored somewhere, all the historic data, that needs processing to then be graphed out for exec number 3, or for whatever.
Other companies might send you files that need to be automatically processed, cleaned, and put correctly where then other tools can pull that data correctly.
We basically do all the background work concerning data manipulation. File processing, databases… all that stuff. And by databases it can be normal ones like posture to distributed ones like hdfs/hive/athena/whatever.
Ad world is basically the same but with tracking info instead of transactions.
If you are interested in day to day work, it’s a mix of coding SQL processes, then porting them to spark/pyspark for distributed massive processing. There are new shiny tools for those that don’t know much of the technical side to manage, sorta.