我必须导入大约20,000行的CSV文件。此CSV是从FTP服务器读取的,它将每30分钟更新一次。但是我写的代码已经花费了超过45分钟的时间来导入。这是非常慢的。有谁能帮帮我。
foreach ($readers as $key => $row) {
$totalRecords +=1;
$filterArray = $this->entityManager->getRepository(Article::class)->findBy(['id' => $row['id']]);
if (empty($filterArray)) {
$notFoundRecords +=1;
continue;
}
$foundRecords +=1;
$this->processPriceRow($row);
}
protected function processPriceRow($row)
{
$existingRecord = $this->entityManager
->getRepository(WareHouse::class)
->findBy(['id' => $row['product_id']]);
if (empty($existingRecord)) {
return $this->fillArticleWareHouse($row);
}
}
protected function fillArticleWareHouse($row, $i, $batchSize)
{
$newWareHouse = new WareHouse();
....
....
...
// Insert.
$this->entityManager->persist($newWareHouse);
$this->entityManager->flush();
}我正在考虑基于batchSize = 100来持久化数据。但是因为我在函数中有函数,所以我也不能实现它。
发布于 2020-07-20 14:36:57
您可以像这样实现批处理。
protected $batchSize = 100;
protected $i = 0;
protected function processPriceRow($row)
{
$existingRecord = $this->entityManager
->getRepository(WareHouse::class)
->findBy(['id' => $row['product_id']]);
if (empty($existingRecord)) {
return $this->fillArticleWareHouse($row);
}
$this->entityManager->flush();
}
protected function fillArticleWareHouse($row)
{
$newWareHouse = new WareHouse();
//....
$this->entityManager->persist($newWareHouse);
++$this->i;
if (($this->i % $this->batchSize) === 0) {
$this->entityManager->flush();
}
}同样,如果您用一次选择来选择所有的项目和WareHouse实体,并将它们保存到数组entityId =>实体中,效果会更好。
// articles
$rowIds = [];
foreach ($readers as $key => $row) {
$rowIds[] = $row['id'];
}
$articles = $this->entityManager->getRepository(Article::class)->findBy(['id' => $rowIds]);
$articleIdToArticle = [];
foreach ($articles as $article) {
$articleIdToArticle[$article->getId()] = $article;
}
foreach ($readers as $key => $row) {
$totalRecords +=1;
if(!key_exists($row['id'], $articleIdToArticle)) {
$notFoundRecords +=1;
continue;
}
$foundRecords +=1;
$this->processPriceRow($row);
}https://stackoverflow.com/questions/62989025
复制相似问题