在GAE中使用gcloud-python

Question

我有一堆小Raspberry Pis运行了一些python代码，它使用gcloud-python数据存储包直接保存到Datastore (跳过GAE)。这个很好用。现在我想使用通过网络和移动客户端来呈现数据。在我的MacBook上，我通过pip使用安装程序和gcloud安装了GAE。我可以编写一个简单的python脚本，并直接从终端执行它，它能够通过gcloud从数据存储区进行读写--这也很好。

但是，当我试图将相同的代码合并到GAE中时，它就失败了。根据我的研究，我希望这是一个路径问题，但经过几个小时的各种尝试，我无法解决这个问题。请提出建议。

我相信这篇文章和我的文章相似：，更改哪个python版本

以下是一些可能相关的信息：

Python版本

$ which python
/usr/bin/python

根据引用的堆栈溢出问题，我将GAE的首选项设置为Python路径为/usr/bin/python。我试过了

$PATH

$ echo $PATH
/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/bin:/Users/sheridangray/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin

Python解释器os.path

$ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path
<module 'posixpath' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>

gcloud安装

$ pip install --upgrade gcloud
Requirement already up-to-date: gcloud in /Library/Python/2.7/site-packages
...

Python2.7参考

$ sudo find / -name python2.7
Password:
/Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/include/python2.7
/Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Applications/Dropbox.app/Contents/Resources/include/python2.7
/Applications/Dropbox.app/Contents/Resources/lib/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/lib/python2.7
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/usr/bin/python2.7
/usr/lib/python2.7

相关代码

from gcloud import datastore

class MainHandler(webapp.RequestHandler):
  def get(self):
     dataset = datastore.get_dataset(dataset_id, email_address, private_key_file)

GAE日志文件

*** Running dev_appserver with the following flags:
--skip_sdk_update_check=yes --port=8080 --admin_port=8000
Python command: /usr/bin/python
INFO     2014-11-21 09:02:03,276 devappserver2.py:745] Skipping SDK update check.
INFO     2014-11-21 09:02:03,288 api_server.py:172] Starting API server at: http://localhost:49183
INFO     2014-11-21 09:02:03,292 dispatcher.py:185] Starting module "default" running at: http://localhost:8080
INFO     2014-11-21 09:02:03,294 admin_server.py:118] Starting admin server at: http://localhost:8000
INFO     2014-11-21 09:02:34,319 module.py:709] default: "GET / HTTP/1.1" 200 2
INFO     2014-11-21 09:02:34,455 module.py:709] default: "GET /favicon.ico HTTP/1.1" 200 8348
INFO     2014-11-21 09:02:44,359 module.py:387] Detected file changes:
  /Users/sheridangray/Projects/city-pulse-web/main.py
ERROR    2014-11-21 09:02:46,443 webapp2.py:1552] gcloud
Traceback (most recent call last):
      File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
  File "/Users/sheridangray/Projects/city-pulse-web/main.py", line 63, in get
dataset = datastore.get_dataset(dataset_id, email_address, private_key_file)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py", line 103, in get_dataset
connection = get_connection(client_email, private_key_path)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py", line 65, in get_connection
    from gcloud.datastore.connection import Connection
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/connection.py", line 3, in <module>
    from gcloud import connection
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py", line 8, in <module>
class Connection(object):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py", line 24, in Connection
USER_AGENT = "gcloud-python/{0}".format(get_distribution('gcloud').version)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 311, in get_distribution
if isinstance(dist,Requirement): dist = get_provider(dist)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 197, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 666, in require
needed = self.resolve(parse_requirements(requirements))
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 565, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
DistributionNotFound: gcloud
INFO     2014-11-21 09:02:46,456 module.py:709] default: "GET / HTTP/1.1" 500 5010
INFO     2014-11-21 09:02:46,514 module.py:709] default: "GET /favicon.ico HTTP/1.1" 304 -

Answer 1

您可以在App上运行gcloud-python，但是它需要一些额外的工作。看看我编写的一个框架项目，它可以实现这个功能。

要覆盖的主要基地是：

获取依赖项

在install_gcloud.sh中，pip用于安装gcloud及其在应用程序中的依赖项。

pip install --target="application/vendor/" gcloud

(正如其他答案中提到的，本地安装不会在部署时上传到App。)

修改依赖项

通过pip-installing和--target set，pkg_resources.get_distribution将按预期工作(在堆栈跟踪中失败)。

此外，默认情况下，pytz在App上有太多的读操作，因此将使用gae-pytz。因此，需要修改一些pytz导入。

此外，为了减少计算引擎检查开销(网络开销)，可以修改oauth2client.client模块。

这两种调整都可以在单个提交中找到。

使进口发挥作用

上面的脚本将所有依赖项放在一个名为vendor/的目录中，appengine_config.py通过达斯·本德尔将其添加到导入路径中。

import darth
darth.vendor('vendor')

此外，由于protobuf依赖项也是google包的一部分(就像所有dependency导入(例如google.appengine.ext.ndb)一样)，您需要修改与该包关联的__path__：

import os
import google

curr_dir = os.path.abspath(os.path.dirname(__file__))
vendor_dir = os.path.join(curr_dir, 'vendor')
google.__path__.append(os.path.join(vendor_dir, 'google'))

警告(截至2015年1月22日)

请注意，在App中使用gcloud-python将比使用本地App库db或ndb慢3-5倍。这是因为那些将RPC直接应用到API运行时，而gcloud-python将使用API外部的HTTP与Cloud对话。

注意到：我在最初的帖子之后更新了它，它引用了历史上以前的点。

Answer 2

Mac上的GAE无法访问安装在Mac上默认位置的Python包。你需要这样做：

ln -s /Library/Python/2.7/site-packages/.../gcloud /Users/sheridangray/Projects/city-pulse-web/gcloud

(需要替换.与相关的路径信息)