如何在放电筒中增加批号

Question

在pytorch中，DataLoader会将数据集分割成集大小的批次，并增加洗牌等选项，然后可以循环使用这些选项。

但是，如果我需要增加批大小，比如前10批大小50，接下来5批大小100等等，那么最好的方法是什么呢？

我试着分裂张量，然后把它们连接起来：

#10x50 + 5*100
originalTensor = torch.randn(1000, 80)
split1=torch.split(originalTensor, 500, dim=0)
split2=torch.split(list(split1)[0], 100, dim=0)

在此之后，是否有一种将级联张量传递到dataLoader的方法，或者任何其他方法直接将级联张量转换为生成器(这可能会丢失洗牌和其他功能)？

Answer 1

我认为您可以通过简单地向您的batch_sampler提供一个非默认的DataLoader来做到这一点。

例如：

class VaryingSizeBatchSampler(Sampler):
    r"""Wraps another sampler to yield a varying-size mini-batch of indices.

    Args:
        sampler (Sampler): Base sampler.
        batch_size_fn (function): Size of current mini-batch.
        drop_last (bool): If ``True``, the sampler will drop the last batch if
            its size would be less than ``batch_size``
    """

    def __init__(self, sampler, batch_size_fn, drop_last):
        if not isinstance(sampler, Sampler):
            raise ValueError("sampler should be an instance of "
                             "torch.utils.data.Sampler, but got sampler={}"
                             .format(sampler))
        self.sampler = sampler
        self.batch_size_fn = batch_size_fn
        self.drop_last = drop_last
        self.batch_counter = 0

    def __iter__(self):
        batch = []
        cur_batch_size = self.batch_size_fn(self.batch_counter)  # get current batch size
        for idx in self.sampler:
            batch.append(idx)
            if len(batch) == cur_batch_size:
                yield batch
                self.batch_counter += 1
                cur_batch_size = self.batch_size_fn(self.batch_counter)  # get current batch size                
                batch = []
        if len(batch) > 0 and not self.drop_last:
            yield batch            

    def __len__(self):
        raise NotImplementedError('You need to implement it yourself!')