DATA ENGINEER
Must-have:PythonDataAI
Job Description
- Build and maintain data pipelines from various data sources.
- Develop and maintain web crawling/scraping processes to collect data.
- Perform ETL (Extract, Transform, Load) processes to prepare data for further use.
- Clean, transform, and validate data to ensure good data quality.
- Develop scripts to process documents and generate metadata from PDF files.
- Monitor and troubleshoot data crawling and processing processes.
- Automate data processing workflows using tools such as Airflow, n8n, or similar tools.
- Collaborate with the team to support data needs for applications, analytics, and AI-related projects.
Kualifikasi
- Bachelor’s degree in computer science, mathematics, statistics, or related fields.
- Minimum 1 Years experience as Data engineer, Data Processing, Web Scarping, or a related field
- Proficient in SQL and at least one programming language, preferably Python.
- Familiar with web crawling/scraping and data cleaning.
- Familiar with ETL/data pipeline processes.
- Familiarity with Airflow, n8n, or similar tools.
- Familiar with relational databases such as PostgreSQL or MySQL.
- Experience with PDF/document processing is a plus.
- Strong analytical and problem-solving skills
- Able to work independently and within a team
- Willing to continuously learn and grow
Skills: Database Systems, Data Integration, Microsoft SQL Server, Python, ETL, Redis, Hadoop, MongoDB, Data Processing, Data Engineering