Job Description
As a Senior Web Crawler Engineer, you will play a pivotal role in maintaining and optimizing our large-scale distributed crawling systems. You will work closely with cross-functional teams to ensure stable data acquisition and deliver high-quality technical solutions aligned with business objectives.
- Lead the development and maintenance of distributed crawler systems to overcome complex anti-scraping mechanisms
- Ensure stable and reliable data supply for core business operations
- Collaborate with quantitative trading teams to translate business requirements into efficient technical solutions
- Work closely with product managers, designers, and frontend developers to ensure high-quality deliverables
- Continuously optimize crawling performance and data processing pipelines
Key Responsibilities
- Design and implement robust solutions to bypass advanced anti-scraping techniques
- Develop and maintain high-performance distributed crawling systems
- Analyze and reverse engineer JavaScript code to overcome obfuscation
- Implement effective strategies against various anti-scraping measures
- Optimize data storage and processing pipelines for different data types
Job Requirements
- Solid Engineering Foundation:
- 3+ years of Python development experience
- Strong computer science fundamentals (data structures, algorithms, networking, OS)
- Deep understanding of Python internals (GIL, memory management)
- Expertise in concurrency models (processes/threads/coroutines)
- Technical Skills:
- Proficient with FastAPI, Django or similar web frameworks
- Expert in MySQL/Redis with strong database optimization skills
- Experienced with Docker and Kubernetes deployment
- Web Scraping Expertise:
- 2+ years experience in large-scale data collection
- Proficient with Scrapy, Playwright or similar frameworks
- Strong JS reverse engineering skills
- Knowledge of encryption algorithms (RSA/AES/DES) in anti-scraping
- Experience with various anti-scraping countermeasures
- Skilled in data cleaning and storage solutions
Benefits
Please contact our HR team via Telegram for detailed benefits information.
We offer competitive compensation packages including:
- Performance-based bonuses
- Comprehensive health insurance
- Flexible work arrangements
- Continuous learning opportunities
