使用scrapy从动态网站复制数据

Question

我开始为这个网站写一个刮板来收集关于汽车的数据。事实证明，数据结构可能会改变，因为卖方没有填写所有字段，因为有哪些字段可以更改，而且在刮板过程中，由于csv文件的结果，值​​在不同的字段中。

页面示例：

https://www.olx.ua/obyavlenie/prodam-voikswagen-touran-2011-goda-IDBzxYq.html#87fcf09cbd

https://www.olx.ua/obyavlenie/fiat-500-1-4-IDBjdOc.html#87fcf09cbd

数据示例： 数据示例

一种方法是使用text () = "Category name"检查字段名，但我不确定如何正确地将结果写入正确的单元格。

此外，我使用内置的Google工具，在命令document.getElementsByClassName('margintop5')[0].innerText的帮助下，我获得了表的全部内容，但结果不是结构化的。

那么，如果输出可以是json格式的，那么它会解决我的问题吗？

innerText结果

此外，当我研究页面代码时，我遇到了一个javascript脚本，在这个脚本中，所有必需的数据都已经被结构化了，但是我不知道如何获得它们。

                 <script type="text/javascript">
                var GPT = GPT || {};
                GPT.targeting = {"cat_l0":"transport","cat_l1":"legkovye-avtomobili","cat_l2":"volkswagen","cat_l0_id":"1532","cat_l1_id":"108","cat_l2_id":"1109","ad_title":"volkswagen-jetta","ad_img":"https:\/\/img01-olxua.akamaized.net\/img-olxua\/676103437_1_644x461_volkswagen-jetta-kiev.jpg","offer_seek":"offer","private_business":"private","region":"ko","subregion":"kiev","city":"kiev","model":["jetta"],"modification":[],"motor_year":[2006],"car_body":["sedan"],"color":["6"],"fuel_type":["543"],"motor_engine_size":["1751-2000"],"transmission_type":["546"],"motor_mileage":["175001-200000"],"condition":["first-owner"],"car_option":["air_con","climate-control","cruise-control","electric_windows","heated-seats","leather-interior","light-sensor","luke","on-board-computer","park_assist","power-steering","rain-sensor"],"multimedia":["acoustics","aux","cd"],"safety":["abs","airbag","central-locking","esp","immobilizer","servorul"],"other":["glass-tinting"],"cleared_customs":["no"],"price":["3001-5000"],"ad_price":"4500","currency":"USD","safedealads":"","premium_ad":"0","imported":"0","importer_code":"","ad_type_view":"normal","dfp_user_id":"e3db0bed-c3c9-98e5-2476-1492de8f5969-ver2","segment":[],"dfp_segment_test":"76","dfp_segment_test_v2":"46","dfp_segment_test_v3":"46","dfp_segment_test_v4":"32","adx":["bda2p24","bda1p24","bdl2p24","bdl1p24"],"comp":["o12"],"lister_lifecycle":"0","last_pv_imps":"2","user-ad-fq":"2","ses_pv_seq":"1","user-ad-dens":"2","listingview_test":"1","env":"production","url_action":"ad","lang":"ru","con_inf":"transportxxlegkovye-avtomobilixx46"};

中的数据

如何使用python和scrapy从页面中获取数据？

Answer 1

您可以从<script>块中提取JS代码，使用regex只获取带有数据的JS对象，然后使用json模块加载它：

query = 'script:contains("GPT.targeting = ")::text'
js_code = response.css(query).re_first('targeting = ({.*});')
data = json.loads(js_code)

这样，data就是包含来自JS对象的数据的python。

有关re_first方法的更多信息如下：https://doc.scrapy.org/en/latest/topics/selectors.html#using-selectors-with-regular-expressions