Job Description
We are seeking a skilled Big Data Operations Engineer to join our team. The ideal candidate will be responsible for the comprehensive management of our big data infrastructure, ensuring optimal performance, security, and scalability.
Key Responsibilities
- Build, operate, manage, and troubleshoot big data platforms to ensure seamless operations
- Ensure efficient operation, stability, and security of the big data platform while providing solutions for cluster upgrades and expansions
- Configure and optimize various platform components including Hadoop, Yarn, Kafka, Flink, Doris, MySQL, and PostgreSQL
- Launch and deploy big data tasks while being familiar with common execution issues and developing emergency plans
- Monitor system performance and implement improvements to enhance efficiency
- Collaborate with development teams to implement best practices for data processing and storage
- Document system configurations, processes, and procedures for knowledge sharing
Job Requirements
- Bachelor's degree in Computer Science, Engineering, or related field
- 3+ years of experience in big data operations and management
- Strong expertise in Hadoop ecosystem components (HDFS, YARN, MapReduce)
- Proficiency in stream processing frameworks (Kafka, Flink)
- Experience with relational databases (MySQL, PostgreSQL) and analytical databases (Doris)
- Knowledge of Linux system administration and shell scripting
- Understanding of distributed systems principles and best practices
- Ability to troubleshoot complex system issues and implement solutions
- Excellent problem-solving skills and attention to detail
- Strong communication and collaboration skills
Preferred Qualifications
- Experience with cloud-based big data solutions (AWS EMR, Azure HDInsight, GCP Dataproc)
- Knowledge of containerization technologies (Docker, Kubernetes)
- Familiarity with infrastructure as code tools (Terraform, Ansible)
- Understanding of data security and compliance requirements
- Certifications in relevant big data technologies
