内存错误！使用mne数据集

Question

我正在尝试使用mne-python的'visual_92_categories‘数据集，但是当我想要过滤和提取纪元时，我得到了内存错误！我的内存是7G的。我想知道有没有人能帮我。python或jupyter笔记本有没有内存限制？谢谢

data_path = visual_92_categories.data_path()    
# Define stimulus - trigger mapping
fname = op.join(data_path, 'visual_stimuli.csv')
conds = read_csv(fname)
max_trigger = 92
conds = conds[:max_trigger]  
conditions = []
for c in conds.values:
    cond_tags = list(c[:2])
    cond_tags += [('not-' if i == 0 else '') + conds.columns[k]
                  for k, i in enumerate(c[2:], 2)]
    conditions.append('/'.join(map(str, cond_tags)))
print(conditions[24])
event_id = dict(zip(conditions, conds.trigger + 1))
n_runs = 4  # 4 for full data (use less to speed up computations)
fname = op.join(data_path, 'sample_subject_%i_tsss_mc.fif')
raws = [read_raw_fif(fname % block) for block in range(n_runs)]
raw = concatenate_raws(raws)    
events = mne.find_events(raw, min_duration=.002)    
events = events[events[:, 2] <= max_trigger]       
picks = mne.pick_types(raw.info, meg=True)
epochs = mne.Epochs(raw, events=events, event_id=event_id, baseline=None,
                    picks=picks, tmin=-.1, tmax=.500, preload=True)
y = epochs.events[:, 2]           
X1 = epochs.copy().get_data()

Answer 1

执行这段代码需要超过7 7Gb的内存。即使是X1阵列也只有4 4Gb。但是它的类型是float64，所以如果你不能获得更多的内存，试着将它保存为float32 (内存消耗将减半)。在大多数情况下，精度的下降是可以接受的。

此外，probalby还可以尝试逐块处理数据，将其作为numpy.array保存到磁盘上，完成后，上载并连接数组：

# leaving initial part intact
import pickle  # need to save a data

for block in range(n_runs):
    raw = mne.io.read_raw_fif(fname % block)
    # raw = concatenate_raws(raws)
    events = mne.find_events(raw, min_duration=.002)
    events = events[events[:, 2] <= max_trigger]
    picks = mne.pick_types(raw.info, meg=True)
    try:
        epochs = mne.Epochs(raw, events=events, event_id=event_id, base
    line=None,
    picks=picks, tmin=-.1, tmax=.500, preload=True)
    except ValueError:  # there's no correct data in some blocks, catch exception
        continue
       y = epochs.events[:, 2].astype('float32')
       X1 = epochs.copy().get_data().astype('float32')
       pickle.dump(y, open('y_block_{}.pkl'.format(block), 'wb'))  # use convenient names 
       pickle.dump(X1, open('x_block_{}.pkl'.format(block), 'wb'))

# remove unnecessary objects from memory
del y
del X1
del raw
del epochs

X1 = None  # strore x_arrays
y = None  # sore y_s
for block in range(n_runs):
    try:
        if X1 is None:
            X1 = pickle.load(open('x_block_{}.pkl'.format(block), 'rb'))
            y = pickle.load(open('y_block_{}.pkl'.format(block), 'rb'))
        else:
            X1 = np.concatenate((X1, pickle.load(open('x_block_{}.pkl'.format(block), 'rb'))))
             y = np.concatenate((y, pickle.load(open('y_block_{}.pkl'.format(block), 'rb'))))
     except FileNotFoundError:  # if no such block from the previous stage
         pass

因此，这段代码可以在不耗尽内存(即<7 Gb)的情况下工作，但我不确定mne是否独立地处理所有块，并且它是等价的代码。至少这段代码创建了一个没有约0.5%样本的数组。比我更有mne经验的人可能会修好它。