在所有具有循环的页面上获取请求的最快方法

Question

我正在制定一种获取请求的方法，并将给定url中的所有页面转换为json。我在字典中解析网站的不同搜索查询，但是每个查询有不同的页面数。我最初的尝试是在获取页面总数之后创建一个长的if-else语句。

所以，就像：

for key, value, page in zip(pages_c['keys'], pages_c['values'],pages_c['pages']):
    if key == "clothing/womens-clothing/dresses" and value == '20s':
        for i in range(1, page):
            ...
    elif key == "clothing/womens-clothing/dresses" and value == '30s':
        for i in range(1, page):        
            ...
...
...
...

其中，pages_c被指定为：

pages_c = {'keys':['clothing/womens-clothing/dresses','clothing/womens-clothing/dresses','clothing/womens-clothing/dresses','clothing/womens-clothing/dresses','clothing/womens-clothing/dresses','clothing/womens-clothing/dresses','clothing/womens-clothing/dresses','clothing/womens-clothing/dresses','clothing/womens-clothing/jackets-and-coats','clothing/womens-clothing/jackets-and-coats','clothing/womens-clothing/jackets-and-coats','clothing/womens-clothing/jackets-and-coats','clothing/womens-clothing/jackets-and-coats','clothing/womens-clothing/jackets-and-coats','clothing/womens-clothing/jackets-and-coats','clothing/womens-clothing/jackets-and-coats'],
          'values':['20s','30s','40s','50s','60s','70s','80s','90s','20s','30s','40s','50s','60s','70s','80s','90s'],
          'pages':[159,169,250,250,250,250,250,250,26,33,86,161,250,250,250,250]}

然而，这种方法要花很长时间！我想知道最快的方法是什么。我的期望是填充请求，所以使用上面的代码likeso：

pall_pages = defaultdict(list)
for key, value, page in zip(pages_c['keys'], pages_c['values'],pages_c['pages']):
    if key == "clothing/womens-clothing/dresses" and value == '20s':
        for pag_num in range(1,page):
            data = {
          'log_performance_metrics': 'true',
          'specs[async_search_results][]': 'Search2_ApiSpecs_WebSearch',
          'specs[async_search_results][1][search_request_params][detected_locale][language]': 'en-GB',
          'specs[async_search_results][1][search_request_params][detected_locale][currency_code]': 'GBP',
          'specs[async_search_results][1][search_request_params][detected_locale][region]': 'GB',
          'specs[async_search_results][1][search_request_params][locale][language]': 'en-GB',
          'specs[async_search_results][1][search_request_params][locale][currency_code]': 'GBP',
          'specs[async_search_results][1][search_request_params][locale][region]': 'GB',
          'specs[async_search_results][1][search_request_params][name_map][query]': 'q',
          'specs[async_search_results][1][search_request_params][name_map][query_type]': 'qt',
          'specs[async_search_results][1][search_request_params][name_map][results_per_page]': 'result_count',
          'specs[async_search_results][1][search_request_params][name_map][min_price]': 'min',
          'specs[async_search_results][1][search_request_params][name_map][max_price]': 'max',
          'specs[async_search_results][1][search_request_params][parameters][q]': val,
          'specs[async_search_results][1][search_request_params][parameters][explicit]': '1',
          'specs[async_search_results][1][search_request_params][parameters][ship_to]': 'GB',
          'specs[async_search_results][1][search_request_params][parameters][page]': str(pag_num),
          'specs[async_search_results][1][search_request_params][parameters][ref]': 'pagination',
          'specs[async_search_results][1][search_request_params][parameters][facet]': keys,
          'specs[async_search_results][1][search_request_params][parameters][referrer]': f'https://www.etsy.com/search/{keys}/dresses?q={val}&explicit=1&locationQuery=2635167&ship_to=GB&page={pag_num}&ref=pagination',
          'specs[async_search_results][1][search_request_params][user_id]': '',
          'specs[async_search_results][1][request_type]': 'pagination_preact',
          'specs[async_search_results][1][is_eligible_for_spa_reformulations]': 'false',
          'view_data_event_name': 'search_async_pagination_specview_rendered'
        }
            r = requests.post('https://www.etsy.com/api/v3/ajax/bespoke/member/neu/specs/async_search_results', headers=headers, data=data)
            print(r)

Answer 1

如果每个if-else都有相同的代码行，则只需编写：

for key, value, page in zip(pages_c['keys'], pages_c['values'],pages_c['pages']):
    for pag_num in range(1,page):
        data = {
      'log_performance_metrics': 'true',
      'specs[async_search_results][]': 'Search2_ApiSpecs_WebSearch',
      'specs[async_search_results][1][search_request_params][detected_locale][language]': 'en-GB',
      'specs[async_search_results][1][search_request_params][detected_locale][currency_code]': 'GBP',
      'specs[async_search_results][1][search_request_params][detected_locale][region]': 'GB',
      'specs[async_search_results][1][search_request_params][locale][language]': 'en-GB',
      'specs[async_search_results][1][search_request_params][locale][currency_code]': 'GBP',
      'specs[async_search_results][1][search_request_params][locale][region]': 'GB',
      'specs[async_search_results][1][search_request_params][name_map][query]': 'q',
      'specs[async_search_results][1][search_request_params][name_map][query_type]': 'qt',
      'specs[async_search_results][1][search_request_params][name_map][results_per_page]': 'result_count',
      'specs[async_search_results][1][search_request_params][name_map][min_price]': 'min',
      'specs[async_search_results][1][search_request_params][name_map][max_price]': 'max',
      'specs[async_search_results][1][search_request_params][parameters][q]': val,
      'specs[async_search_results][1][search_request_params][parameters][explicit]': '1',
      'specs[async_search_results][1][search_request_params][parameters][ship_to]': 'GB',
      'specs[async_search_results][1][search_request_params][parameters][page]': str(pag_num),
      'specs[async_search_results][1][search_request_params][parameters][ref]': 'pagination',
      'specs[async_search_results][1][search_request_params][parameters][facet]': keys,
      'specs[async_search_results][1][search_request_params][parameters][referrer]': f'https://www.etsy.com/search/{keys}/dresses?q={val}&explicit=1&locationQuery=2635167&ship_to=GB&page={pag_num}&ref=pagination',
      'specs[async_search_results][1][search_request_params][user_id]': '',
      'specs[async_search_results][1][request_type]': 'pagination_preact',
      'specs[async_search_results][1][is_eligible_for_spa_reformulations]': 'false',
      'view_data_event_name': 'search_async_pagination_specview_rendered'
    }

        r = requests.post('https://www.etsy.com/api/v3/ajax/bespoke/member/neu/specs/async_search_results', headers=headers, data=data)
        print(r)