Job Description:
This role involves leading the development of solutions to overcome complex anti-crawling mechanisms, maintaining large-scale distributed crawling systems, and ensuring stable data supply for core business needs. The candidate will work closely with the team to translate business requirements into efficient technical solutions, collaborating with product, design, and front-end teams to ensure high-quality deliverables.
Key Responsibilities:
- Lead the development and maintenance of distributed crawling systems to ensure stable data acquisition.
- Deeply understand quantitative trading logic and collaborate with the team to implement technical solutions.
- Work closely with product, design, and front-end teams to deliver high-quality solutions.
- Analyze and reverse-engineer JavaScript code to bypass anti-crawling mechanisms.
- Design and optimize database structures for efficient data storage and retrieval.
Job Requirements:
- Solid Engineering Skills:
- 3+ years of Python development experience with strong computer science fundamentals (data structures, algorithms, networking, OS).
- Deep understanding of Python internals (GIL, memory management) and proficiency in concurrency models (processes, threads, coroutines).
- Experience with FastAPI, Django, MySQL, Redis, and database optimization.
- Proficiency in Docker and familiarity with Kubernetes for deployment.
- Crawling & Reverse Engineering:
- 2+ years of experience in large-scale data scraping using Scrapy, Playwright, etc.
- Strong JS reverse-engineering skills to debug and deobfuscate code, with knowledge of encryption algorithms (RSA, AES, DES).
- Expertise in bypassing anti-crawling measures (CAPTCHAs, IP bans, request signing, behavioral detection).
- Experience in data cleaning (missing/abnormal values) and storage design (SQL/NoSQL/time-series databases).
- Preferred Qualifications:
- Experience in live trading and understanding market trends.
- Knowledge of ML/DL applications in trading (K-line pattern recognition, robo-advisors, smart strategies).
- Familiarity with financial markets (cryptocurrency, futures, stocks).
- Strong product design thinking.
- Experience in AI-driven automation.
- Proficiency in Golang.
Benefits:
Year-end bonuses, medical subsidies, holiday allowances, paid leave, and more.