使用requests / requests_futures实现流式HTTP的异步处理

Question

我了解如何通过HTTP请求流式传输数据：

import requests

r = requests.get(url, stream=True)
for line in r.iter_lines():
    print(line)

我了解如何异步处理请求：

from requests_futures.sessions import FuturesSession

session = FuturesSession()
def bg_cb(sess, resp):
    # parse the json storing the result on the response object
    resp.data = resp.json()

future = session.get(url, background_callback=bg_cb)
# do some other stuff, send some more requests while this one works
response = future.result()

但是我不知道如何将这两者结合起来，也就是说，当r.iter_lines()中的下一行可用时，如何将回调传递给我指定的函数

session = FuturesSession()
def bg_cb(sess, resp):
    # parse the json storing the result on the response object
    resp.data = resp.json()

future = session.get(url, background_callback=bg_cb, stream=True)

# do other stuff here

在任何数据实际可用之前，代码片段只调用我的bg_cb函数一次。相反，我希望只要有新的行可用就调用bg_cb。

Answer 1

您可以使用pycurl，就像它在docs中所说的那样：

一个例子可能是：

import pyrcurl

STREAM_URL = "whatever/url"

def on_receive (data):
    print(data)

'''We instantiate the Curl object and setoptions so that it gets executed. 
We use the on_receive function so that the stream of data is printed as it arrives the object, which is very cool! 

The perform() operation keeps executing as it is a real time stream of data, not executing the subsequent code UNTIL we set a TIMEOUT option or we sys.exit() it with CTRL+C
'''

conn = pycurl.Curl()
conn.setopt(pycurl.URL, STREAM_URL)
conn.setopt(pycurl.WRITEFUNCTION, on_receive)
# conn.setopt(pycurl.TIMEOUT, 15)
conn.perform()
conn.close()

每当有新数据被流式传输时，pycurl.WRITEFUNCTION都会执行on_receive函数。

pyrcul.TIMEOUT用于指定将执行perform()方法的秒数。

注意:生成的对象将是class <bytes>，因此您需要在应用程序中进行适当的处理。