如何对大文件应用one-hot编码？

Question

我有3 Gb的训练文件和350 Mb的测试文件，6 Gb的RAM。该文件似乎不太大，无法将它们保存在内存中，但我甚至不能附加它们(发生内存错误)。

我目前的做法是这样的：

# reading files
      dtypes = {'Semana' : 'int32',
                          'Agencia_ID' :'int32',
                          'Canal_ID' : 'int32',
                          'Ruta_SAK' : 'int32',
                          'Cliente-ID' : 'int32',
                          'Producto_ID':'int32',
                          'Venta_hoy':'float32',
                          'Venta_uni_hoy': 'int32',
                          'Dev_uni_proxima':'int32',
                          'Dev_proxima':'float32',
                          'Demanda_uni_equil':'int32'}

train = pd.read_csv('train.csv', dtype  = dtypes, usecols=["Semana", "Agencia_ID", "Canal_ID", 'Ruta_SAK',  'Cliente_ID', 'Producto_ID','Demanda_uni_equil'])
test = pd.read_csv('test.csv',dtype  = dtypes)
ids = test['id']
test.drop(['id'], axis =1, inplace = True)

shapeTrain = train.shape[0]
shapeTest = test.shape[0]

train = train.append(test) # raises memory error

#applying one-hot encoding
train = pd.concat([train, pd.get_dummies(train['Semana'],sparse=True)], axis=1, join_axes=[train.index])
train.drop([11,'Semana'],axis=1, inplace = True)

train = pd.concat([train, pd.get_dummies(train['Producto_ID'],sparse=True)], axis=1, join_axes=[train.index])
train.drop([123,'Producto_ID'],axis=1, inplace = True)


train = pd.concat([train, pd.get_dummies(train['Canal_ID'],sparse=True)], axis=1, join_axes=[train.index])
train.drop([11,'Canal_ID'],axis=1, inplace = True)

# separating back to train and test
test = train[shapeTrain:shapeTrain+shapeTest]
train = train[0:shapeTrain]

test['id'] = ids

train.to_csv('train_1.csv', index=False)
test.to_csv('test_1.csv', index=False)

我认为有两种方法可以解决：

1)以某种方式遍历文件：

df_train = pd.read_csv('train.csv', chunksize=1500)
for chunk in df_train:
    #apply one-hot encoding

2)通过只对训练和测试应用one-hot编码来减少列数，但对于新的分类值，将所有列设置为0。

3)使用the hashing trick！

我该如何解决这个问题？