在`index.search中使用FAISS `index.search()‘分割故障

Question

我正在尝试进行索引搜索，给定FAISS索引中的向量，该向量保存在磁盘上并被读取到内存中。这发生在使用FastAPI在端点中调用的函数中。

它正在调用的端点和函数如下：

# Endpoint
@app.get('/neighbours', response_model=Neighbours)
async def get_neighbours(case: str, ids: str, request: Request):
    """
       Retrieves a set of neighbors for a single id
    """
       id_list = ids.split(';')
       return Neighbours(result=get_neighbours_from_id(case, [int(id) for id in id_list]))

# Functions and objects
class Neighbour(BaseModel):
    id: int
    neighbours: List[int]


class Neighbours(BaseModel):
    result: List[Neighbour]


def get_neighbours_from_id(case, ids):
    
    case_path = validate_case_path(case)

    index = read_index(case_path.as_posix())
    # The above works well, the index is loaded properly.

    vec = index.reconstruct_n(0, 1)
    # This also works, I can see the vector being reconstructed.

    faulthandler.enable()
    faulthandler.dump_traceback()
    # Enabling faulthandler for the traceback

    d, i = index.search(vec, 1)
    # This is where the code crashes.

当我试图搜索给定重建向量的已加载索引时，代码会崩溃，并使用以下回溯：

Current thread 0x000000010781adc0 (most recent call first):
  File "/../api/app/vectors.py", line 37 in get_neighbours_from_id
  File ".../api/service.py", line 87 in get_neighbours
  File ".../pypoetry/virtualenvs/env1-py3.8/lib/python3.8/site-packages/fastapi/routing.py", line 160 in run_endpoint_function
  File ".../pypoetry/virtualenvs/env1-py3.8/lib/python3.8/site-packages/fastapi/routing.py", line 227 in app
  File ".../pypoetry/virtualenvs/env1-py3.8/lib/python3.8/site-packages/starlette/routing.py", line 61 in app
  File ".../pypoetry/virtualenvs/env1-py3.8/lib/python3.8/site-packages/starlette/routing.py", line 259 in handle
  File ".../pypoetry/virtualenvs/env1-py3.8/lib/python3.8/site-packages/starlette/routing.py", line 656 in __call__
  File ".../pypoetry/virtualenvs/env1-py3.8/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 18 in __call__
  File ".../pypoetry/virtualenvs/env1-py3.8/lib/python3.8/site-packages/starlette/exceptions.py", line 71 in __call__
  File ".../pypoetry/virtualenvs/env1-py3.8/lib/python3.8/site-packages/starlette/middleware/cors.py", line 84 in __call__
  File ".../pypoetry/virtualenvs/env1-py3.8/lib/python3.8/site-packages/starlette/middleware/base.py", line 34 in coro
  File ".../anaconda3/lib/python3.8/asyncio/events.py", line 81 in _run
  File ".../anaconda3/lib/python3.8/asyncio/base_events.py", line 1859 in _run_once
  File ".../anaconda3/lib/python3.8/asyncio/base_events.py", line 570 in run_forever
  File ".../anaconda3/lib/python3.8/asyncio/base_events.py", line 603 in run_until_complete
  File ".../anaconda3/lib/python3.8/asyncio/runners.py", line 43 in run
  File ".../run_service.py", line 19 in <module>
Fatal Python error: Segmentation faultFatal Python error: Fatal Python error: 

Segmentation faultFatal Python error: 
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

追溯使我认为这是与asyncio或FastAPI中的多线程有关的东西？

需要注意的是，完全相同的代码(索引加载、向量重建和搜索)在相同环境中的以下最小示例中都没有任何问题：

import faiss

index = faiss.read_index("my_index.index")

vec = index.reconstruct_n(0, 1)

d, i = index.search(vec, 1)

print(i)
# Works properly.

我看过本期，但它似乎是修正在较新版本的FAISS，这是我正在使用的。

非常感谢您的帮助！

Answer 1

分割故障通常是由于硬件/内存问题造成的。我用的是Macbook专业版，所以这对我来说是个mac问题。我可以选择以下两种方法之一来使其发挥作用：

• 我从Mac转到AWS EC2 / Linux
• 如果我不想切换到EC2 / Linux，我构建了一个映像并在其中运行它