Data Engineer at Trendx

Job Description

Key Responsibilities

• Responsible for data crawling, including static web pages, dynamic web pages (JS rendering), API interface data, etc.
• Handle anti-crawling strategies such as User Agent impersonation, proxy pooling, captcha bypass, cookie encryption, body parameter encryption, etc., to improve crawling success rate.
• Analyze webpage data and extract information using techniques such as XPath, CSS selectors, regular expressions, etc.
• Store and crawl data to databases such as MySQL, MongoDB, Redis, Selectdb, etc.
• Write data cleaning and deduplication related code to improve data quality.
• Monitor the running status of crawlers, optimize crawling strategies, and ensure the stability of data crawling.

Job Requirements

• Proven experience in web scraping and data crawling techniques.
• Strong knowledge of handling anti-crawling mechanisms and strategies.
• Proficiency in data extraction techniques like XPath, CSS selectors, and regular expressions.
• Experience with various databases such as MySQL, MongoDB, Redis, or Selectdb.
• Ability to write efficient data cleaning and deduplication scripts.
• Strong problem-solving skills and attention to detail.
• Experience in monitoring and optimizing crawler performance is a plus.