将不稳定表转换为DataFrame文件时出错

Question

我正在尝试从Gaia目录中获取一些数据，然后将这个不稳定的表转换为一个熊猫DataFrame，然后我想将它存储在一个hdf5文件中。我不能直接将astropy表(查询的结果)存储到hdf5文件中，因为我需要对它进行一些处理。

问题是，当我想将DataFrame存储到hdf文件中时，我会得到以下错误：

Traceback (most recent call last):
  File "C:/Users/Administrateur.UTILISA-D5U7HV7/Documents/MEGA/ipsa/cours/aero4/stage/working_directory/python/tests/stackoverflow_issue/1_panda_to_hdf/tohdf.py", line 8, in <module>
    pd_table.to_hdf("test.h5", key="test", format='table', data_columns=True, mode="w", encoding="utf-8")
  File "C:\Users\Administrateur.UTILISA-D5U7HV7\Documents\MEGA\ipsa\cours\aero4\stage\working_directory\python\venv\lib\site-packages\pandas\core\generic.py", line 2505, in to_hdf
    encoding=encoding,
  File "C:\Users\Administrateur.UTILISA-D5U7HV7\Documents\MEGA\ipsa\cours\aero4\stage\working_directory\python\venv\lib\site-packages\pandas\io\pytables.py", line 282, in to_hdf
    f(store)
  File "C:\Users\Administrateur.UTILISA-D5U7HV7\Documents\MEGA\ipsa\cours\aero4\stage\working_directory\python\venv\lib\site-packages\pandas\io\pytables.py", line 274, in <lambda>
    encoding=encoding,
  File "C:\Users\Administrateur.UTILISA-D5U7HV7\Documents\MEGA\ipsa\cours\aero4\stage\working_directory\python\venv\lib\site-packages\pandas\io\pytables.py", line 1042, in put
    errors=errors,
  File "C:\Users\Administrateur.UTILISA-D5U7HV7\Documents\MEGA\ipsa\cours\aero4\stage\working_directory\python\venv\lib\site-packages\pandas\io\pytables.py", line 1709, in _write_to_group
    data_columns=data_columns,
  File "C:\Users\Administrateur.UTILISA-D5U7HV7\Documents\MEGA\ipsa\cours\aero4\stage\working_directory\python\venv\lib\site-packages\pandas\io\pytables.py", line 4143, in write
    data_columns=data_columns,
  File "C:\Users\Administrateur.UTILISA-D5U7HV7\Documents\MEGA\ipsa\cours\aero4\stage\working_directory\python\venv\lib\site-packages\pandas\io\pytables.py", line 3813, in _create_axes
    errors=self.errors,
  File "C:\Users\Administrateur.UTILISA-D5U7HV7\Documents\MEGA\ipsa\cours\aero4\stage\working_directory\python\venv\lib\site-packages\pandas\io\pytables.py", line 4800, in _maybe_convert_for_string_atom
    for i in range(len(block.shape[0])):
TypeError: object of type 'int' has no len()

我最初认为是我的计算造成了一个问题，但即使没有它，我也会得到错误。

这是我的代码：

• 首先您需要通过运行：(查询可能需要几分钟)

来获取带有查询结果的文件

from astroquery.gaia import Gaia

job3 = Gaia.launch_job_async("SELECT * \
FROM gaiadr1.gaia_source \
WHERE CONTAINS(POINT('ICRS',gaiadr1.gaia_source.ra,gaiadr1.gaia_source.dec),CIRCLE('ICRS',56.75,24.1167,2))=1 \
AND abs(pmra_error/pmra)<0.10 \
AND abs(pmdec_error/pmdec)<0.10 \
AND pmra IS NOT NULL AND abs(pmra)>0 \
AND pmdec IS NOT NULL AND abs(pmdec)>0 \
AND pmra BETWEEN 15 AND 25 \
AND pmdec BETWEEN -55 AND -40;", dump_to_file=True)
print(job3)
p = job3.get_results()

然后，

• 可以运行以下代码，并显示上面的错误。只需注意Table.read()函数中文件的名称，因为查询不会给出与下面的示例相同的名称。

from astropy.table import Table
import pandas as pd

table = Table.read("async_20200611171019.vot", format='votable')

pd_table = table.to_pandas()
print(pd_table)
pd_table.to_hdf("test.h5", key="test", format='table', data_columns=True, mode="w", encoding="utf-8")

hdf_table = pd.DataFrame(pd.read_hdf("test.h5"))
print(hdf_table)

有没有人知道这个问题会从何而来？谢谢！

Answer 1

它看起来像phot_variable_flag列有对象dtype，也就是说，它是一个由对象组成的numpy数组。它还被蒙蔽：

In [30]: table['phot_variable_flag'].dtype                                                                                                                    
Out[30]: dtype('O')

In [31]: type(table['phot_variable_flag'])                                                                                                                    
Out[31]: astropy.table.column.MaskedColumn

当我删除那篇专栏的时候，它被熊猫成功地写成了HDF5。