The data engineer is a technical role inside our company and the core specialisation upon which our services are built. ## Involved people Active in data engineering roles: - [[Bart Dubbeldam]] (Medior) - [[Daniël Laurentius]] (Junior) - [[Jeroen de Vries]] (Medior) - [[Philip Boontje]] (Medior) - [[Xavier Paredes]] (Medior) ## Time tracking Each engineer tracks their hours using our time tracker, with for each timeslot tracked a description needs to be added. We encourage our engineers to be verbose about their tasks. This approach builds trust and enables more intelligent conversations about invoices when clients do not agree with the time spent on certain tasks. ### Task classification A common best practice at our company is to start off with the task type you are working on, followed by a more detailed description. The usual categories of work used by us are: 1. Minor issue *(an issue that can be fixed 1-2 hours)* 2. Major issue *(an issue that can be fixed in 3-8 hours)* 3. Minor item *(usually used for a 1-4 hour ad-hoc task)* 4. Major item *(usually used for a 5-8 hour ad-hoc task)* 5. Development *(larger chunks of properly scoped work 5-100+ hours)* 6. Call *(ad-hoc call with the client to discuss minor/major items or issues)* 7. Project Management *(planning, budgeting, scoping, check-ins)* <br> For example you are requested to add the billable utilisation to the data model then your time records could look like this: `Development | Billable Utilisation | Build dbt staging, mart and external model` Followed by another time record which states: `Development | Billable Utilisation | Add measures and metrics to semantic model in cube` When you are working on a short and ad-hoc task you can use the other task type categories like. For example: `Minor issue | Fix data extraction from float time-out` Or a client calls you for 15 minutes to discuss which measures should be part of the new sickness leave percentage metric: `Call | Discussed technical details sickness leave percentage with John Doe` ### Budgeting Whenever you are encountering a task that requires an above average amount of work in some cases we create a dedicated budget code. Let's say you work on the Project `PR0243 | ABC Company | Data Engineering | Hourly` you would create the budget code `BU_243_53`. The 243 references the project number and the last 2 digits represent the budget code that is next in line for being created. Budget Codes are advised to be generated on work that requires: 1. More than 1.000 EUR in dedicated budget 2. More than 50% of the monthly capacity budget. > [!abstract] Example: > When the monthly capacity budget is 3.000 EUR and the required budget is 1.400 EUR, a budget code will not be necessary. When the capacity budget is 2.000 EUR, then a budget code would be advised. Ofcourse, these guidelines are what they are, guidelines. Depending on the client a dedicated budget is hardly required, or each task larger than 500 EUR requires a dedicated budget. All these scenarios are possible and should be catered to as long as you, the engineer, are comfortable with the degree of overhead. ## Scoping Whenever we perform tasks for our clients, we are required to scope the requirements and deliverables in a consistent and intuitive way. Therefore we have created multiple templates to aid our engineers to work towards effective collaboration and deployment. We have 2 templates that we are actively using: 1. Simple Scope 2. Advanced Scope ### Simple Scope Whenever there is a task between 1 and 8 hours, usually these do not require that much details. Spending 15 minutes scoping these tasks is usually enough. > [!abstract] Details > ![[Simple Scope]] ### Advanced Scope Tasks that require 4+ hours, usually require a bit more context and one or more back and forth between the engineer and the client. Depending on the size of the task, engineers spend between 5-10% of the overall time estimate on preparation and tracking progress. So for a task of 20 hours, spending 1-2 hours scoping is common practice. > [!abstract] Details > ![[Advanced Scope]] ## Financial compensation We dive deeper into the income part of each seniority level in the [[Engineer Compensation]]. ## Education At our guild we value the education of our engineers. As they climb in profiency and grow towards the role of Data Architect they will need to develop engineering depth, contextual breadth and a sufficient understanding of human nature. Therefore we have set up an extensive [list of books and videos](https://airtable.com/appF0pxDtxF1fVm6O/shrG8BSIqrvFIWYTk) that align with this educational goal. Apart from the in-depth engineering, the provided list also provide our engineers with a foundation in philosophy, entrepreneurship, theology, strategy, history, finance and psychology. We hope that spending time on absorbing the knowledge in the books and videos our engineers will build a more broad interest in non-engineering domains so they will be able to effectively contribute to the development of society and take their responsibility in the roles that life has given to them. ## Seniority As the engineer progresses in their competence and proficiency they are gradually able to take on more complex tasks and take more responsibility. We distinguish between 4 degrees of seniority that are dependent on the number of hours and years of experience in the data engineering field. | **Role** | **Seniority** | **Hours of experience** | **Years of experience** | | ------------- | ------------- | ----------------------- | ----------------------- | | Data Engineer | Junior | 0-5.000 | 0-4 | | Data Engineer | Medior | 5.000-10.000 | 5-8 | | Data Engineer | Senior | 10.000-15.000 | 9-12 | | Data Engineer | Architect | >15.000 | 13+ | ### Junior Data Engineer When people are starting with minimal experience we are looking for the following traits: - **Creative and innovative** – Able to think outside the box and come up with novel solutions to technical challenges. Enjoys experimenting with new technologies and approaches to improve data processes and workflows. - **Independent & self-starting** – Capable of taking initiative and driving tasks forward with minimal supervision. Comfortable working in a fast-paced environment where problem-solving and adaptability are key. - **Humble and keen to learn** – Open to feedback and eager to grow both personally and professionally. Willing to ask questions, seek guidance, and continuously develop skills in data engineering and related technologies. - **Systematic thinker** – Approaches problems methodically and can break down complex data challenges into manageable components. Strives for efficiency and consistency in data pipelines, architecture, and workflows. - **Making complexity simple** – Able to transform intricate technical concepts into clear, understandable insights for both technical and non-technical audiences. Strives to build scalable and maintainable solutions that simplify data processes. - **Proficient in English** – Capable of communicating effectively in English, both written and verbal, to collaborate with international teams, document work, and present findings clearly. ### Medior Data Engineer When people have 5-8 years experience in data engineering and want to embrace the role of medior engineer, we are looking for the following traits: - **Develops extractors with ease** – Experienced in designing and implementing efficient data extraction processes from various sources, including APIs, databases, and flat files. Ensures that extractors are reliable, scalable, and optimized for performance. - **Knows SQL and Python** – Proficient in writing complex SQL queries for data manipulation, transformation, and optimization. Comfortable using Python for data processing, automation, and integration, leveraging libraries such as Pandas and Airbyte. - **Works with GIT repositories** – Well-versed in version control best practices using Git, including branching strategies, pull requests, and code reviews. Ensures collaboration and code integrity within a team development environment. - **Can set up continuous deployments** – Familiar with CI/CD pipelines and deployment automation, ensuring smooth and efficient updates to data pipelines and infrastructure. Can configure tools like GitHub Actions, Pulumi, or dbt CI for seamless deployments. - **Aces dimensional data modeling** – Strong understanding of data warehousing concepts, including star and snowflake schemas. Capable of designing and optimizing dimensional models to support analytical queries and business intelligence applications. - **Proficient in BI tooling** – Experienced in working with business intelligence tools such as Tableau, Power BI, Looker, or Metabase. Able to create insightful dashboards and reports that help stakeholders make data-driven decisions. ### Senior Data Engineer When people have 9-12 years experience in data engineering and want to embrace the role of senior engineer, we are looking for the following traits: - **Designs end-to-end systems** – Capable of architecting, implementing, and optimizing data solutions from ingestion to analytics. Ensures scalability, reliability, and performance across the entire data pipeline while considering best practices in security, governance, and cost-efficiency. - **Takes the lead on projects** – Drives data engineering projects from conception to completion, managing timelines, priorities, and resources effectively. Provides technical direction, ensures alignment with business objectives, and mitigates risks proactively. - **Lead contact for C-level executives** – Confidently communicates complex data strategies and technical solutions to executive leadership. Able to translate business needs into actionable technical roadmaps while managing expectations and fostering trust. - **Designs and manages Agile teams** – Builds and leads high-performing Agile teams, ensuring smooth collaboration between engineers, analysts, and stakeholders. Establishes best practices in Agile methodologies, such as sprint planning, stand-ups, and retrospectives, to deliver high-quality outcomes efficiently. - **A visionary on at least one domain** – Possesses deep expertise in a specific area of data engineering, such as cloud data architecture, real-time streaming, AI-driven analytics, or large-scale ETL pipelines. Continuously explores emerging technologies and drives innovation within the organization. - **Can fully run a subset of clients** – Takes ownership of client relationships, managing their data infrastructure, requirements, and evolving needs. Ensures successful delivery of solutions while maintaining a strategic vision to support their growth and future challenges. ### Data Architect When people have 12+ years experience in data engineering and want to embrace the role of senior engineer, we are looking for the following traits: - **Shapes the future of data engineering** – Defines and drives the agency’s long-term data vision, ensuring it remains at the forefront of the industry. Establishes best practices, methodologies, and frameworks that set new standards for data architecture and engineering. - **Spearheads high-impact client transformations** – Works directly with C-level executives from enterprise clients to design and execute large-scale data transformation initiatives. Provides strategic advisory on data modernization, AI/ML readiness, and cloud-native architectures. - **Designs architectures at a global scale** – Leads the design of high-performance, multi-cloud, and hybrid data ecosystems capable of handling petabyte-scale workloads. Ensures scalability, cost-efficiency, and security while balancing diverse client needs. - **Oversees and mentors high-performing architects & engineers** – Develops the next generation of senior engineers and architects within the agency. Establishes internal education programs, technical leadership tracks, and mentoring initiatives to continuously elevate the company’s talent pool. - **Drives the company’s technological and business growth** – Partners with the executive team to identify new revenue opportunities, expand service offerings, and position the agency as a market leader. Advises on strategic investments in emerging technologies, tools, and partnerships. - **Solves the hardest technical challenges** – Acts as the go-to expert for solving the agency’s most complex architectural problems, from real-time data processing at scale to next-generation AI-powered data platforms. Provides technical leadership and ensures the success of mission-critical projects.