根据第一次/最后一次记录的最大值/分钟值对文件进行裁剪

Question

我对相当大的csv文件有一点问题。我能够编写简单的bash/awk脚本，但对于我有限的awk/bash编程经验来说，这个问题更难解决。

问题：

• 我所有的文件都在文件夹里。文件夹有一个偶数的csv文件，需要对对进行修剪(我将用这种方法解释)。文件名为: f1L、f1R、f2L、f2R、f3L、f3R、.、fnL、fnR。
• 文件需要成对阅读.f1L和f1R。f2L与f2R等
• 文件有两个逗号分隔的字段。f1L (文件开始/结束)和f1R，如

f1L (START) 1349971210，-0.984375 1349971211，-1.000000 f1R (START) 1349971206，-0.015625 1349971207，0.000000 f1L (END) 1350230398，0.500000 1350230399，0.515625 f1R (END) 1350230402，0.484375 1350230403，0.515625

关于awk，我想做的是：

1. 读取f1L的记录1、字段1 (即1349971210)，然后读取f1R的记录1、字段1(即1349971206)。然后取这两个值的最大值(即x1 = 1349971210)。
2. 读取最后一条记录，f1L的字段1(即1350230399)，最后一条记录，f1R字段1(即1350230403)。然后取最小值(即x2 = 1350230399)。
3. 然后以相同的名称提取和重新保存f1L和f1R中大于/等于x1和小于/等于x2之间的所有行。
4. 对我目录中的所有对重复处理。

想知道你们中是否有人建议使用bash/awk来完成任务。

Answer 1

一种在bash中实现这一目标的天真方法。这里根本不需要效率。没有错误检查(嗯，只有强制性的最小值)。

将此脚本命名为myscript。它将接受两个参数(文件fxL和fxR)。

#!/bin/bash

tmp=''

die() {
    echo >&2 "$@"
    exit 1
}

on_exit() {
    [[ -f $tmpL ]] && rm -- "$tmpL"
    [[ -f $tmpR ]] && rm -- "$tmpR"
}

last_non_blank_line() {
   sed -n -e $'/^$/ !h\n$ {x;p;}' "$1"
}

(($#==2)) || die "script takes two arguments"

fL=$1
fR=$2

[[ -r "$fL" && -w "$fL" ]] || die "problem with file \`$fL'"
[[ -r "$fR" && -w "$fR" ]] || die "problem with file \`$fR'"

# read record1, line1 of fL and fR
IFS=, read min _ < "$fL"
[[ $min =~ ^[[:digit:]]+$ ]] || die "first line of \`$fL' has a bad record"
IFS=, read t _ < "$fR"
[[ $t =~ ^[[:digit:]]+$ ]] || die "first line of \`$fR' has a bad record"
((t>min)) && ((min=t))

# read record1, last line of fL and fR
IFS=, read max _ < <( last_non_blank_line "$fL")
[[ $max =~ ^[[:digit:]]+$ ]] || die "last line of \`$fL' has a bad record"
IFS=, read t _ < <(last_non_blank_line "$fR")
[[ $t =~ ^[[:digit:]]+$ ]] || die "last line of \`$fR' has a bad record"
((t<max)) && ((max=t))

# create tmp files
tmpL=$(mktemp --tmpdir) || die "can't create tmp file"
tmpR=$(mktemp --tmpdir) || die "can't create tmp file"

trap 'on_exit' EXIT

# Read fL line by line, and only keep those
# the first record of which is between min and max
while IFS=, read a b; do
    [[ $a =~ ^[[:digit:]]+$ ]] && ((a<=max)) && ((a>=min)) && echo "$a,$b"
done < "$fL" > "$tmpL"
mv -- "$tmpL" "$fL"

# Same with fR:
while IFS=, read a b; do
    [[ $a =~ ^[[:digit:]]+$ ]] && ((a<=max)) && ((a>=min)) && echo "$a,$b"
done < "$fR" > "$tmpR"
mv -- "$tmpR" "$fR"

并称之为：

$ myscript f1L f1R

先在刮擦文件上使用它！没有保修！使用风险自负！

警告。由于脚本使用bash算法进行比较，因此假设每个文件中每一行的第一个记录是bash处理范围内的整数。

编辑.，因为您的第一条记录是浮动的，所以不能使用上面使用bash算法的方法。一个非常有趣的方法是让bash完成所有必要的操作(获取第一行、最后一行、打开文件、…)。算法部分采用bc。这样，您将完全不受数字大小的限制(bc使用任意精度)，并且欢迎浮动！例如：

#!/bin/bash

tmp=''

die() {
    echo >&2 "$@"
    exit 1
}

on_exit() {
    [[ -f $tmpL ]] && rm -- "$tmpL"
    [[ -f $tmpR ]] && rm -- "$tmpR"
}

last_non_blank_line() {
   sed -n -e $'/^$/ !h\n$ {x;p;}' "$1"
}

(($#==2)) || die "script takes two arguments"

fL=$1
fR=$2

[[ -r "$fL" && -w "$fL" ]] || die "problem with file \`$fL'"
[[ -r "$fR" && -w "$fR" ]] || die "problem with file \`$fR'"

# read record1, line1 of fL and fR
IFS=, read a _ < "$fL"
IFS=, read b _ < "$fR"
min=$(bc <<< "if($b>$a) { print \"$b\" } else { print \"$a\" }" 2> /dev/null)
[[ -z $min ]] && die "problem in first line of files \`$fL' or \`$fR'"

# read record1, last line of fL and fR
IFS=, read a _ < <( last_non_blank_line "$fL")
IFS=, read b _ < <(last_non_blank_line "$fR")
max=$(bc <<< "if($b<$a) { print \"$b\" } else { print \"$a\" }" 2> /dev/null)
[[ -z $max ]] && die "problem in last line of files \`$fL' or \`$fR'"

# create tmp files
tmpL=$(mktemp --tmpdir) || die "can't create tmp file"
tmpR=$(mktemp --tmpdir) || die "can't create tmp file"

trap 'on_exit' EXIT

# Read fL line by line, and only keep those
# the first record of which is between min and max
while read l; do
    [[ $l =~ ^[[:space:]]*$ ]] && continue
    r=${l%%,*}
    printf "if(%s>=$min && %s<=$max) { print \"%s\n\" }\n" "$r" "$r" "$l"
done < "$fL" | bc > "$tmpL" || die "Error in bc while doing file \`$fL'"

# Same with fR:
while read l; do
    [[ $l =~ ^[[:space:]]*$ ]] && continue
    r=${l%%,*}
    printf "if(%s>=$min && %s<=$max) { print \"%s\n\" }\n" "$r" "$r" "$l"
done < "$fR" | bc > "$tmpR" || die "Error in bc while doing file \`$fR'"

mv -- "$tmpL" "$fL"
mv -- "$tmpR" "$fR"

Answer 2

我试图包括所有必要的健全检查和最小化光盘I/O(假设您的文件足够大，读取它们是时间限制因素)。此外，这些文件不必作为一个整体在内存中读取(假设您的文件可能比可用的RAM还要大)。

然而，这只是尝试使用一个非常基本的虚拟输入-所以请测试它，并报告任何问题。

首先，我编写了一个修整一对的脚本(由f.L文件名标识)：

#!/bin/sh

#############    
# trim_pair #
#-----------#############################
# given fXL file path, trim fXL and fXR #
#########################################

#---------------# 
# sanity checks #
#---------------#

# error function
error(){
 echo >&2 "$@"
 exit 1
}

# argument given?
[[ $# -eq 1 ]] || \
 error "usage: $0 <file>"
LFILE="$1"

# argument format valid?
[[ `basename "$LFILE" | egrep '^f[[:digit:]]+L$'` ]] || \
 error "invalid file name: $LFILE (has to match /^f[[:digit:]]+L$/)"
RFILE="`echo $LFILE | sed s/L$/R/`" # is there a better POSIX compliant way?

# files exists?
[[ -e "$LFILE" ]] || \
 error "file does not exist: $LFILE"
[[ -e "$RFILE" ]] || \
 error "file does not exist: $RFILE"

# files readable?
[[ -r "$LFILE" ]] || \
 error "file not readable: $LFILE"
[[ -r "$RFILE" ]] || \
 error "file not readable: $RFILE"

# files writable?
[[ -w "$LFILE" ]] || \
 error "file not writable: $LFILE"
[[ -w "$RFILE" ]] || \
 error "file not writable: $RFILE"

#------------------#
# create tmp files #
# & ensure removal #
#------------------#

# cleanup function
cleanup(){
 [[ -e "$LTMP" ]] && rm -- "$LTMP"
 [[ -e "$RTMP" ]] && rm -- "$RTMP"
}

# cleanup on exit
trap 'cleanup' EXIT

#create tmp files
LTMP=`mktemp --tmpdir` || \
 error "tmp file creation failed"
RTMP=`mktemp --tmpdir` || \
 error "tmp file creation failed"

#----------------------#
# process both files   #
# prepended by their   #
# first and last lines #
#----------------------#

# extract first and last lines without reading the whole files twice
{
 head -q -n1 "$LFILE" "$RFILE"  # no need to read the whole files
 tail -q -n1 "$LFILE" "$RFILE"  # no need to read the whole files
} | awk -F, '
 NF!=2{
  print "incorrect file format: record "FNR" in file "FILENAME > "/dev/stderr"
  exit 1    
 }
 NR==1{                         # read record 1,
  x1=$1                         # field 1 of L,
  next                          # then read
 }
 NR==2{                         # record 1 of R,
  x1=$1>x1?$1:x1                # field 1 & take the max,
  next                          # then
 }
 NR==3{                         # read last record,
  x2=$1                         # field 1 of L,
  next                          # then
 }
 NR==4{                         # last record of R
  x2=$1>x2?$1:x2                # field 1 & take the max
  next
 }
 FILENAME!="-"&&NR<5{
  print "too few lines in input" > "/dev/stderr"
 }
 FNR==1{
  outfile=FILENAME~/L$/?"'"$LTMP"'":"'"$RTMP"'"
 }
 $1>=x1&&$1<=x2{
  print > outfile
 }
' - "$LFILE" "$RFILE" || \
 error "error while trimming"

#-----------------------#
# re-save trimmed files #
# under the same names  #
#-----------------------#

mv -- "$LTMP" "$LFILE" || \
 error "cannot re-save $LFILE"
mv -- "$RTMP" "$RFILE" || \
 error "cannot re-save $RFILE"

正如您所看到的，主要思想是在根据您的请求使用head和tail使用awk处理输入之前，使用重要的行对输入进行预置。

要对某个目录中的所有文件调用该脚本，您可以使用以下脚本(不像上面的那样工作，但我想您可以自己想出类似的方法)：

#!/bin/sh

############
# trim all #
#----------###################################
# find L files in current or given directory #
# and trim the corresponding file pairs      #
##############################################

TRIM_PAIR="trim_pair"   # path to the trim script for one pair

if [[ $# -eq 1 ]]
then
 WD="$1"
else
 WD="`pwd`"
fi

find "$WD"                         \
 -type f                           \
 -readable                         \
 -writable                         \
 -regextype posix-egrep            \
 -regex "^$WD/"'f[[:digit:]]+L'    \
 -exec "$TRIM_PAIR" "{}" \;

请注意，您必须将trim_pair脚本放在PATH上，或者调整trim_all脚本中的TRIM_PAIR变量。

Answer 3

使用珀尔

use warnings;
use strict;

my $dir = $ARGV[0];  # directory is argument
my @pairs;
for my $file (glob "$dir/f[0-9]*L") {
    my $n = ($file =~ /(\d+)/)[0];
    my ($fn1, $fn2) = ($file, "f${n}R");
    my ($dL, $dR) = (loadfile($fn1), loadfile($fn2));
    my ($min, $max) = (min($dL->[0][1], $dR->[0][1]),
                       max($dL->[-1][1], $dR->[-1][1]));    
    trimfile($fn1, $dL, $min, $max);
    trimfile($fn2, $dL, $min, $max);
}

sub loadfile {
    my ($fname, @d) = (shift);
    open(my $fh, "<", $fname) or die ("$!");
    chomp, push(@d, [ split(/[, ]+/, $_) ]) while <$fh>;
    close $fh;
    return \@d;
}

sub trimfile {
    my ($fname, $data, $min, $max) = @_;
    open(my $fh, ">", $fname) or die ("$!");
    print($fh $_->[0], " ", $_->[1], "\n") for @$data;
    close $fh;
}

sub min { my ($a,$b) = @_; return $a < $b ? $a : $b; }
sub max { my ($a,$b) = @_; return $a > $b ? $a : $b; }