Job Description
We are seeking a highly skilled professional to lead the development and maintenance of our large-scale distributed crawling systems. The ideal candidate will tackle complex anti-crawling mechanisms while ensuring stable data acquisition for our core business needs.
Key Responsibilities
- Lead the breakthrough of various anti-crawling mechanisms and maintain large-scale distributed crawling systems to ensure stable supply of core data sources
- Deeply understand quantitative trading logic and work closely with the team to transform business requirements into efficient technical solutions
- Collaborate with product, design, and front-end teams to ensure high-quality deliverables
- Design and implement robust data processing pipelines for both structured and unstructured data
- Continuously optimize system performance and stability under high-concurrency scenarios
Job Requirements
- Solid Engineering Foundation:
- 3+ years of Python development experience with strong computer science fundamentals (data structures, algorithms, networking, operating systems)
- Deep understanding of Python internals (GIL, memory management) and proficiency in process/thread/coroutine models for high-performance concurrent design
- Proficient with FastAPI, Django and other web frameworks; expert in MySQL/Redis with strong database design and optimization skills
- Experienced with Docker and familiar with k8s; capable of independent deployment
- Crawling and Reverse Engineering Skills:
- 2+ years of large-scale data collection experience with frameworks like Scrapy and Playwright
- Strong JS reverse engineering capabilities to debug and deobfuscate code; familiar with encryption algorithms (RSA/AES/DES) in anti-crawling
- Thorough understanding of anti-crawling mechanisms (CAPTCHAs, IP bans, request signing, behavior detection) with proven countermeasures
- Experienced in data cleaning (missing/abnormal values) and designing appropriate storage solutions (RDBMS/NoSQL/time-series databases)
- Preferred Qualifications:
- Practical trading experience with understanding of market trends and indicators
- Experience applying ML/DL in candlestick pattern recognition, robo-advisors, or smart strategies
- Familiarity with digital currencies, futures, or securities markets
- Strong product design thinking
- AI/ML implementation experience
- Golang proficiency
Benefits
Please contact our HR team directly via Telegram for detailed compensation and benefits information.
