HQ BARCELONA - Senior Data Engineer

Jobandtalent Who we are : We are a workforce on-demand company that heavily relies on tech . We integrate the complete process of selecting, hiring and managing employees, being the first in digitizing all the steps, introducing Artificial Intelligence solutions in multiple phases, obtaining faster and higher-quality service for our clients and great satisfaction and loyalty for our workers. Our Impact on Society : Our vision is to create a long-term job security condition for our workers while providing flexible solutions to our clients . We can do this by concatenating different short-term contracts and minimizing–or completely removing–their unemployed time. We want to remove the need for finding a job ever again. Data-Driven : Being the owner of the entire hiring funnel, workforce management, and working experience, give us very unique data that we transform in powerful knowledge to make better decisions , improve job-candidate matches, estimate quality and affinity, and share performance feedback with our workers and clients. We are the most advanced workforce on-demand company thanks to our data and the way we use it. Great Challenges for the Data Team : Integrating very different AI solutions to improve the experience for both our workers and employers (Shift Optimization, Machine Learning algorithms, Quality Score Estimations, Document recognition, etc.), relying on a very powerful ETL that permits endless automatizations while keeping the focus on data quality . Growth Overview : An annual growth of 170% , available in multiple countries: the UK, Spain, Germany, Sweden, Mexico, Colombia, France, and more are coming. HQ in Madrid with a second big presence in Barcelona and some remote workers in the UK, The Netherlands, and the USA. And finally, some of our great clients are Amazon , XPOLogistics, Cabify , Santander, Just Eat , etc. You will be the main referent of our data architecture , leading the design and implementation decisions of our data pipelines . The Data Team is very independent of the rest of the Tech and Product teams. We build and manage our services that run our AI models as well as our data storages to quickly process the data we need, and let the Data Scientist performing the “Time Travel” to properly train our algorithms. A quick insight about our architecture and tools: We are building our new data lake and data warehouse directly in AWS S3 , synchronizing the data via DMS , and consolidate it via Hudi . This data will be accessible by multiple Redshift clusters via Spectrum that allows us to quickly scale up across all the countries. In addition, as data is stored in S3 this could be exploited via Athena or an ad-hoc Spark job . We have multiple queue systems from which we consume data coming from the Jobandtalent Message brokers while filling up our data storage. We use Airflow as our ETL which allows us to break down any complex problem/process into smaller ones. Each component can be implemented directly in Python , or any other language since we could execute any docker container in AWS Fargate . We are integrating different 3rd party data (Kustomer, Salesforce, etc. ). We keep iterating in the best way to incorporate this data into our Datalake which allows our users to exploit this data. You need to have proven experience in these areas, being eager to learn and improve, while enjoying taking responsibility. You need to be very hands-on, able to explore different solutions to our problems and needs. Requirements and Skills Technical degree preferably in Engineering or Quantitative field. Experience managing ETL or similar Data Workflows Experience in leading data architecture decisions and similar responsibilities. Previous practical experience with Big Data technologies knowledge, batch and real-time (Hadoop, Spark, Hudi, Spark Streaming, Flink, etc. ). Extensive experience with production-ready code (not scripting code). Extremely skilled programmer ( e.g. , unittest , production/staging experience). Python proficiency, and ideally experience with other programming languages such as Java , Scala . Strong knowledge of different data storages (knows pros and cons, understand and investigate the best solution for each use-case). Strong knowledge of different storage formats (JSON, Parquet, Avro, etc.), understand and investigate the best solution for each use-case. Streaming data queue platforms knowledge (Kinesis, Kafka, RabbitMQ, etc.). Excellent verbal and written communication skills; ability to communicate effectively with different levels of management, as well as the business and technical communities. A valid work permit to be employed in Spain. Fluent in English is a must. Besides, we are looking for a senior profile that will be interested in : Sharing knowledge inside and outside Jobandtalent writing blog posts and/or giving talks, raising the quality standards of the entire team to grow together. Contributing to Open Source projects : we are using different Open Source frameworks and libraries, and one of our wishes would be to contribute to some of those projects, dedicating part of our time when possible. Offer in Short Great ownership of the projects with a direct impact on the product. Salary in the 55-70k range (wide range, depending on experience). A yearly budget for Conferences/meetup/self-learning. Working in an international and multidisciplinary team A complete benefits program that includes flexible remuneration through expenses for restaurants, transport, childcare, and health insurance . Possibility of working remotely You will be working in the Data Science team (read the last blog posts ), together with Full-stack Engineers, Data Scientists, and Data Analysts. Q&A How the Jobandtalent Engineering team is structured? What we call “Product Team” includes the Product & Design, Tech, and Data. Product & Design ⇒ all the Product Owners and Designers that belong to the different Product Features teams (cross-functional teams that take care of some specific parts of the product). Tech ⇒ includes all the main engineering profiles and guilds, such as Backend, Frontend, Android, iOS, and Platform. The majority of them are embedded into the Product Features teams to create our extremely independent and cross-functional teams. Data ⇒ includes three different teams, namely Data Analytics, Data Science, and Data Engineering. How the Jobandtalent Data team is organized? The Data team includes three different teams that work very closely together. The Data Engineering one owns the data pipelines and processes that generate the data needed by our Data Warehouse, and thus by the entire Jobandtalent. The Data Science team, which owns all the models used by the Product teams The Data Analytics team is composed of two types of profiles: (i) Product Data Analysts, which work embedded in the Product Team following a Hub’n’Spoke approach, and (ii) Data Partners, which are working closely with other departments such as Finance, Sales, Marketing, etc. How do we organize the tasks and the planning ? We are very data-driven, we have clear KPIs for each project, team, and goal. We define strong OKRs for every quarter, we work with bi-weekly Sprint (each team has its own JIRA Project), and a Data PM is leading the delivery processes and metrics.