如何将多页数据存储到json中？

Question

感谢您的关注，所以我写了一个小型项目，抓取新闻网站和存储主要文本从他们。我尝试了许多解决方案，在我的项目中添加json而没有consol.log，但总是在抓取它的显示后只显示一个主要文本。所以我向你展示了我的代码，这样你就可以帮助我如何使用json来处理所有这三个消息。

const { Cluster } = require('puppeteer-cluster');
const fs = require('fs');


const launchOptions = {
    headless: false,
    args: [
        '--disable-gpu',
        '--disable-dev-shm-usage',
        '--disable-web-security',
        '--disable-xss-auditor',
        '--disable-accelerated-2d-canvas',
        '--ignore-certifcate-errors',
        '--ignore-certifcate-errors-spki-list',
        '--no-zygote',
        '--no-sandbox',
        '--disable-setuid-sandbox',
        '--disable-webgl',
    ],
    ignoreHTTPSErrors: true,
    waitUntil: 'networkidle2',
};
(async() => {
    // Create a cluster with 2 workers
    const cluster = await Cluster.launch({
        monitor: true,
        concurrency: Cluster.CONCURRENCY_PAGE,
        maxConcurrency: 2,
        puppeteerOptions: launchOptions,
    });

    // Define a task (in this case: screenshot of page)
    await cluster.task(async({ page, data: url }) => {
        await page.setRequestInterception(true);
        page.on('request', (request) => {
            if (['stylesheet', 'font', 'image', 'styles','other', 'media'].indexOf(request.resourceType()) !== -1) {
                request.abort();
            } else {
                request.continue();
            }
        });
        await page.goto(url);
        const scrapedData = await page.$eval('div[class="entry-content clearfix"]', el => el.innerText)
        fs.writeFileSync('test.json', JSON.stringify(scrapedData, null, 2))
    });
    // Add some pages to queue
    cluster.queue('https://www.ettelaat.com/?p=526642');
    cluster.queue('https://www.ettelaat.com/?p=526640');
    cluster.queue('https://www.ettelaat.com/?p=526641');

    // Shutdown after everything is done
    await cluster.idle();
    await cluster.close();
})();

Answer 1

为了收集所有输出，我必须将fs放在cluster.close的底部

kanopyDB = []
.
.
.
        kanopyDB = kanopyDB.concat(name);
.
.
.
    await cluster.idle();
    await cluster.close();
    fs.writeFileSync('output.json', kanopyDB, 'utf8');