Data Engineer

Founded in 2009 and 100% self-funded, Natural Intelligence is a leader in online multi-vertical marketplaces. Our global brands, Top10.com and BestMoney.com, help millions worldwide make informed decisions daily. Our proprietary platform, xMatch, harnesses AI and ML technologies to help consumers make more accessible choices and find the right products and services tailored to their needs.

Join our dynamic team as a key player in our Data Group, leveraging your expertise to innovate and enhance our advanced big data platform. Take the lead in designing and implementing extensive data pipelines using the latest frameworks and technologies like Spark, Kafka, Iceberg, Flink, Imply, and so on. Collaborate closely with both data science and data analytics teams, guiding projects from inception to completion. Your role is crucial in driving technological excellence and establishing best practices in data operations and pipeline optimization.

We’re looking for someone passionate and experienced in developing large-scale data platforms and applications, supporting everything from analytics & reports to machine learning algorithm

Requirements:

• At least 5 years experience in Scala/Java or equivalent language.
• Experience with Python.
• Proven experience in building, optimizing, and maintaining large-scale big data pipelines using popular open-source frameworks (Spark, Kafka, Hive Metastore, Airflow, Iceberg, etc).
• Strong SQL abilities and hands-on experience with SQL and NoSQL DBs.
• Experience with AWS cloud services such as EMR, Redshift/Snowflake, S3, Athena, and Glue - advantage.
• Experience in stream processing frameworks such as Flink - advantage.
• Exceptional Team Collaboration: Demonstrated ability to work effectively in a team environment, contributing to a collaborative work culture that fosters innovation.
• Ownership and Initiative: Strong ability to own and drive solutions from concept to completion, showing initiative and a proactive approach to problem-solving.