Linux -优化shell脚本计数文件在几个文件夹中

Question

我有一个脚本，每天计数文件在~10个文件夹。当脚本运行时，需要很长时间才能完成。示例:3小时完成计数在3个文件夹，但当我使用命令

ll | grep "condition" | wc -l

在文件夹中需要10秒/文件夹:(

我认为我的脚本有问题，它没有优化。请分享你的经验，谢谢。

input_path_CDR.txt

/bgw_vol01/backup/cdr/folder1
/bgw_vol01/backup/cdr/folder2
/bgw_vol01/backup/cdr/folder3
...

script_count_CDR.sh

#!/bin/bash

file="/bgw_vol01/script/input_path_CDR.txt"

i=1
while read line; do
    FILES=$(find $line -type f)
    hostname=$HOSTNAME
    dat=$(date -d '-1 day' '+%g%m%d')
    
    for f in $FILES
    do
        FILECOUNT="$(find $line | grep "$dat" | wc -l)"
    done
    echo $HOSTNAME"|"$line"|"$dat"|"File count: $FILECOUNT >> output_`date -d '-1 day' '+%g%m%d'`.txt
    
    i=$((i+1))
done < $file

Answer 1

首先，请允许我指出我在您的脚本中看到的问题。

...
i=1   
while read line; do
    FILES=$(find $line -type f)    # you`re putting all found file names in this variable(!)
    hostname=$HOSTNAME    # is it expect to change? if not put it before loop
    dat=$(date -d '-1 day' '+%g%m%d')  # same thing
    
    for f in $FILES   # now you are looping over every single found file, which is unnecessary  
    do
        FILECOUNT="$(find $line | grep "$dat" | wc -l)"  # lets say you have 100 files, it will set this variable 100 times with same value(!)
    done
    echo $HOSTNAME"|"$line"|"$dat"|"File count: $FILECOUNT >> output_`date -d '-1 day' '+%g%m%d'`.txt    # it`s not really an issue but you have $dat variable, so why not to use it in out file name?
    
    i=$((i+1))   # if it`s not used, remove it
done < $file

现在，我建议把它变成这样的东西：

...
hostname=$HOSTNAME
dat=$(date -d '-1 day' '+%g%m%d')
while read line; do
    FILECOUNT=$(find $line -type f -name "*${dat}*" | wc -l)   # we are looking in $line dir, only for files having $dat in file name and count them
    echo "${HOSTNAME}|${line}|${dat}|File count: ${FILECOUNT}" >> output_${dat}.txt
done < $file