Wisestep Madrid, Community of Madrid, Spain
Assistant Manager Recruitment at Avance Consulting
About the Role:
We are seeking a highly skilled Senior Data Engineer with more than 10 years of experience to join our team. The ideal candidate will have extensive experience in Apache Flink and other stream processing technologies, along with a strong background in designing and building large-scale data pipelines within the banking and financial services industry. You will be responsible for driving the architecture, development, and optimization of data processing systems, ensuring high-quality, real-time data solutions to support business intelligence, analytics, and machine learning initiatives.
Key Responsibilities:
1. Stream Processing Architecture: Design and develop scalable, fault-tolerant stream processing systems using Apache Flink and similar technologies to process high-throughput, low-latency financial data.
2. Data Pipeline Development: Build and manage complex data pipelines that integrate real-time and batch data, ensuring seamless data ingestion, processing, and storage for banking and financial applications.
3. Big Data Integration: Work with big data ecosystems such as Apache Kafka, Hadoop, Spark, and Elasticsearch to build robust, highly available data architectures capable of processing and analyzing large datasets in real-time.
4. Database & Data Storage Solutions: Design and implement data storage solutions that support structured and unstructured data using technologies like HBase, Cassandra, Redshift, S3, and RDS.
5. Data Governance & Security: Ensure data security, privacy, and compliance with industry standards (e.g., PCI-DSS, GDPR) while adhering to data governance policies in the banking and financial services industry.
6. Performance Optimization: Optimize data pipelines for high performance and scalability, fine-tuning Flink and other processing components to handle large-scale data processing while minimizing latency.
7. Real-Time Analytics & Machine Learning: Support data science teams by implementing data processing workflows that power real-time analytics and machine learning models to drive actionable insights.
8. Collaboration & Mentorship: Lead and mentor junior data engineers, collaborate with cross-functional teams (data scientists, data analysts, DevOps), and provide technical leadership on data engineering best practices.
9. Troubleshooting & Issue Resolution: Identify, diagnose, and resolve data issues within real-time and batch processing environments, ensuring minimal downtime and maintaining the integrity of banking applications.
Requirements:
1. Experience: 10+ years of experience as a Data Engineer or in a similar role, with at least 5+ years of experience working with Apache Flink and related stream processing tools.
2. In-depth experience in designing, developing, and optimizing real-time data pipelines for banking, financial services, or similarly regulated industries.
3. Language: Fluent Spanish & English are required.
4. Apache Flink Expertise: Extensive hands-on experience with Apache Flink, including stateful processing, event time handling, windowing, and scaling Flink applications.
5. Big Data Technologies: Proficiency with other big data frameworks and tools like Apache Kafka, Hadoop, Spark, Hive, ElasticSearch, and Cassandra.
6. Data Integration & ETL: Solid understanding of ETL/ELT processes, data integration, and the ability to work with both real-time and batch data processing.
7. SQL & NoSQL: Strong experience with SQL and database design. Familiarity with NoSQL databases like Cassandra, HBase, or MongoDB is a plus.
8. Cloud Platforms: Experience with cloud services such as AWS, Azure, or Google Cloud for hosting data pipelines and scalable storage.
9. Security & Compliance: Expertise in implementing security controls for data privacy and compliance with financial industry standards (e.g., PCI-DSS, GDPR).
10. Programming Skills: Proficiency in Java or Scala for building data processing applications. Knowledge of Python is a plus.
11. Problem-Solving: Strong problem-solving and troubleshooting skills, with a proven ability to work in high-pressure, fast-paced environments.
12. Collaboration: Excellent communication and teamwork skills, with experience working in cross-functional teams and mentoring junior engineers.
Preferred Qualifications:
1. Machine Learning Integration: Experience with integrating streaming data workflows with machine learning models for real-time predictions and analytics.
2. Cloud Certifications: AWS Certified Solutions Architect, Google Cloud Professional Data Engineer, or equivalent certifications.
3. Advanced Data Technologies: Familiarity with modern data platforms such as Snowflake, Databricks, or other emerging technologies.
4. Financial Services Regulations: In-depth knowledge of financial regulations such as SOX compliance, AML, or other banking-specific data governance standards.
#J-18808-Ljbffr