Software Engineer (Apache / Pyspark / Python) - Contract Role

Key Responsibilities

• Design, develop, and optimize data workflows and pipelines using Apache Hadoop, Cloudera, and Spark.
• Develop microservice APIs using Python Django framework and utilized Redis as a data store

• Developed various datawarehouse applications on Mainframe with Teradata.
• Manage and optimize data storage in data warehouse solutions, including Cloudera and Hive.

• Develop a unified execution framework to oversee vCPU and memory configuration for YARN applications and logging processes.
• Implement and manage data processing workflows using tools like PySpark, Hive, MapReduce, and Impala.

• Build robust data ingestion frameworks with Kafka and other ETL tools, ensuring data quality and reliability.
• Work with structured and unstructured data using Python, Bash, and Perl scripting to analyze, process, and transform data.

• Manage MongoDB and ensure integration with other data storage and processing systems.
• Identify and address performance degradation issues such as Data Spill, Skew, and resource constraints on Spark Batch & Streaming applications in production environments.

• Use Apache Airflow to schedule, manage, and monitor ETL workflows.
• Work closely with data scientists, analysts, and other engineers to meet business requirements and drive data initiatives.

Requirements:

• Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.
• 7+ years of experience in Big Data and data engineering roles.

• Proficiency with Cloudera, Hadoop, Hive, and MapReduce.
• Advanced knowledge of Python, Bash, and Perl.

• Strong skills in PySpark, Impala, and Kafka.
• Hands-on experience with data warehouse management and optimization.

• Experience with MongoDB and relational databases.
• Proficiency with Apache Airflow or similar orchestration tools.

• Knowledge of Apache, Linux environments, and other DevOps tools.
• Familiarity with cloud platforms such as AWS, Azure, or GCP.

• Experience with real-time data processing and streaming.

Disclaimer: The company is committed to ensuring the privacy and security of your information. By submitting this form, you consent to the collection, processing, and retention of the information you provide. The data collected (which may include your contact details, educational background, work experience and skills) will be used solely for the purpose of evaluating your qualifications for the position you're applying for. Your data will be stored securely and retained for the duration necessary to fulfill our hiring process. If you are not selected for the position, your data will be kept on file for a limited period in case future opportunities arise. You have the right to access, correct, or delete your data at any time by contacting us at Quess Singapore | A Leading Staffing Services Provider in Singapore (quesscorp.sg)

“This is in partnership with the Employment and Employability Institute Pte Ltd (“e2i”).

_e2i is the empowering network for workers and employers seeking employment and employability solutions. e2i serves as a bridge between workers and employers, connecting with workers to offer job security through job-matching, career guidance and skills upgrading services, and partnering employers to address their manpower needs through recruitment, training, and job redesign solutions.e2i is a tripartite initiative of the National Trades Union Congress setup to support nation-wide manpower and skills upgrading initiatives. By applying for this role, you consent to Quesscorp Singapore’s PDPA and e2i’sPDPA. By submitting your application for this position, you consent to the collection, use, and disclosure of your personal data by Quesscorp and its recruitment partners, including Employment and Employability Institute Pte Ltd, for the purpose of i) employment facilitation and ii) data and statistical analysis. You may find out more information on the PDPA policies at Quesscorp website andhttps://www.e2i.com.sg/pdpa/."_