Python + PySpark - Junior Developer Job at Compunnel, Inc., New York, NY

Yk1zRVlWTmdxUitLTmFlKzdBSXk5dWV3YkE9PQ==
  • Compunnel, Inc.
  • New York, NY

Job Description

The Python + PySpark + AWS Consultant will be responsible for designing, developing, and optimizing data engineering solutions using Python and PySpark. The role requires hands-on experience in building scalable ETL pipelines, working with cloud platforms such as AWS or Azure, and ensuring data integrity and performance optimization. The candidate should have strong SQL skills and a deep understanding of data engineering principles. Key Responsibilities

  • Develop and maintain data processing workflows using Python and PySpark
  • Design and implement ETL pipelines for structured and unstructured data
  • Optimize Spark-based data processing for efficiency and scalability
  • Deploy and manage data solutions on AWS or Azure
  • Write and optimize SQL queries for data transformation and analysis
  • Troubleshoot and resolve performance issues in data pipelines
  • Work closely with cross-functional teams to ensure data reliability and integrity
Required Qualifications
  • 5+ years of experience in data engineering
  • Strong proficiency in Python and object-oriented programming
  • Hands-on experience with PySpark for large-scale data processing
  • Proficiency in SQL for data manipulation and query performance tuning
  • Experience with AWS or Azure for cloud-based data solutions
  • Knowledge of ETL processes and data pipeline automation
  • Experience with Hadoop is acceptable
Preferred Qualifications
  • Experience in optimizing Spark jobs for performance and cost efficiency
  • Familiarity with DevOps practices for data engineering
  • Understanding of data governance, security, and compliance best practices
#J-18808-Ljbffr Compunnel, Inc.

Job Tags

Similar Jobs

Parkview Regional Medical Center (Parkview Health)

Internal Medicine, Population Health Physician Job at Parkview Regional Medical Center (Parkview Health)

 ...Job Description Join Our Team! Parkview Physicians Group is currently seeking experienced Board Eligible / Board Certified Family...  ...Hours are Monday-Friday from 8am 5pm Involves travel by conducting daily rounding on Parkview patients in high performing... 

York College of Pennsylvania

Esports Coach Job at York College of Pennsylvania

Position Summary DESCRIPTION : This position is responsible for managing a singular esport at York College. The Coach will help structure practices and guide students in group practices and competition. This individual will have student success at YCP as one of its... 

Get It - Marketing

SEO/Technical Content Writer - Remote Job at Get It - Marketing

 ...organic traffic and generate highly qualified leads through effective SEO strategies, compelling web content, and thoughtful research....  ...0 to 1,500 words. Essential Skills Hard Skills: Strong technical writing background. Ability to create clear, concise, and... 

Atlantis University

Data Analyst Job at Atlantis University

 ...Join to apply for the Data Analyst role at Atlantis University Join to apply for the Data Analyst role at Atlantis...  ...Acquaintance with technical tools in the industry. Seniority level ~ Seniority level Entry level Employment type ~ Employment type Full-... 

Smart Justice California

Investigative Reporter Job at Smart Justice California

 ...reporting experience preferredHistory of award-winning investigative journalism preferredValid driver's license with clean driving record...  ...media company and a leading provider of local news and sports. The Company owns, operates and/or provides services to 185 television...