使用python2内核在GCP的datalab实例中安装apache出错

Question

我正在尝试在GCP上的datalab实例中安装apache beam，但遇到以下问题，我真的很喜欢解决这个问题的方法，目前我对pip或datalab都不是很精通，错误在底部：

        Collecting apache-beam[gcp]
Requirement already satisfied: dill<=0.2.8.2,>=0.2.6 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (0.2.6)
Requirement already satisfied: pydot<1.3,>=1.2.0 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (1.2.4)
Requirement already satisfied: oauth2client<5,>=2.0.1 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (2.2.0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (2.0.0)
Collecting typing<3.7.0,>=3.6.0 (from apache-beam[gcp])
  Using cached https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-none-any.whl
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (3.12)
Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam[gcp])
Requirement already satisfied: future<1.0.0,>=0.16.0 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (0.16.0)
Requirement already satisfied: avro<2.0.0,>=1.8.1 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (1.8.2)
Collecting fastavro==0.19.7 (from apache-beam[gcp])
Requirement already satisfied: crcmod<2.0,>=1.7 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (1.7)
Requirement already satisfied: httplib2<=0.11.3,>=0.8 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (0.11.3)
Requirement already satisfied: futures<4.0.0,>=3.1.1 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (3.2.0)
Requirement already satisfied: grpcio<2,>=1.8 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (1.13.0)
Requirement already satisfied: six<1.12,>=1.9 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (1.10.0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (3.5.2)
Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (2.1.0)
Collecting pytz<=2018.4,>=2018.3 (from apache-beam[gcp])
  Using cached https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl
Collecting google-apitools<=0.5.20,>=0.5.18; extra == "gcp" (from apache-beam[gcp])
  Downloading https://files.pythonhosted.org/packages/1d/0c/64f84f91643f775fdb64c6c10f4a4f0d827f8b0d98a2ba2b4bb9dc2f8646/google_apitools-0.5.20-py2-none-any.whl (330kB)
    100% |████████████████████████████████| 337kB 5.1MB/s ta 0:00:01
Collecting proto-google-cloud-pubsub-v1==0.15.4; extra == "gcp" (from apache-beam[gcp])
  Downloading https://files.pythonhosted.org/packages/c0/a2/2eeffa0069830f00016196dfdd69491cf562372b5353f2e8e378b3c2cb0a/proto-google-cloud-pubsub-v1-0.15.4.tar.gz
Collecting google-cloud-bigquery==0.25.0; extra == "gcp" (from apache-beam[gcp])
  Downloading https://files.pythonhosted.org/packages/76/67/6165c516ff6ceaa62eb61f11d8451e1b0acc4d3775e181630aba9652babb/google_cloud_bigquery-0.25.0-py2.py3-none-any.whl (41kB)
    100% |████████████████████████████████| 51kB 10.1MB/s ta 0:00:01
Collecting google-cloud-pubsub==0.26.0; extra == "gcp" (from apache-beam[gcp])
  Downloading https://files.pythonhosted.org/packages/37/92/c74a643126d58505daec9addf872dfaffea3305981b90cc435f4b9213cdd/google_cloud_pubsub-0.26.0-py2.py3-none-any.whl
Requirement already satisfied: googledatastore==7.0.1; extra == "gcp" in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (7.0.1)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0; extra == "gcp" in /usr/local/envs/py2env/lib/python2.7/site-packages (from apache-beam[gcp]) (0.90.4)
Requirement already satisfied: pyparsing>=2.1.4 in /usr/local/envs/py2env/lib/python2.7/site-packages (from pydot<1.3,>=1.2.0->apache-beam[gcp]) (2.2.0)
Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/envs/py2env/lib/python2.7/site-packages (from oauth2client<5,>=2.0.1->apache-beam[gcp]) (0.4.3)
Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/envs/py2env/lib/python2.7/site-packages (from oauth2client<5,>=2.0.1->apache-beam[gcp]) (0.2.2)
Requirement already satisfied: rsa>=3.1.4 in /usr/local/envs/py2env/lib/python2.7/site-packages (from oauth2client<5,>=2.0.1->apache-beam[gcp]) (3.4.2)
Requirement already satisfied: funcsigs>=1 in /usr/local/envs/py2env/lib/python2.7/site-packages (from mock<3.0.0,>=1.0.1->apache-beam[gcp]) (1.0.0)
Requirement already satisfied: pbr>=0.11 in /usr/local/envs/py2env/lib/python2.7/site-packages (from mock<3.0.0,>=1.0.1->apache-beam[gcp]) (4.0.4)
Requirement already satisfied: setuptools in /usr/local/envs/py2env/lib/python2.7/site-packages (from pyvcf<0.7.0,>=0.6.8->apache-beam[gcp]) (39.2.0)
Requirement already satisfied: enum34>=1.0.4 in /usr/local/envs/py2env/lib/python2.7/site-packages (from grpcio<2,>=1.8->apache-beam[gcp]) (1.1.6)
Requirement already satisfied: requests>=2.7.0 in /usr/local/envs/py2env/lib/python2.7/site-packages (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]) (2.18.4)
Requirement already satisfied: docopt in /usr/local/envs/py2env/lib/python2.7/site-packages (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]) (0.6.2)
Collecting fasteners>=0.14 (from google-apitools<=0.5.20,>=0.5.18; extra == "gcp"->apache-beam[gcp])
  Downloading https://files.pythonhosted.org/packages/14/3a/096c7ad18e102d4f219f5dd15951f9728ca5092a3385d2e8f79a7c1e1017/fasteners-0.14.1-py2.py3-none-any.whl
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in /usr/local/envs/py2env/lib/python2.7/site-packages (from proto-google-cloud-pubsub-v1==0.15.4; extra == "gcp"->apache-beam[gcp]) (1.5.3)
Collecting google-cloud-core<0.26dev,>=0.25.0 (from google-cloud-bigquery==0.25.0; extra == "gcp"->apache-beam[gcp])
  Downloading https://files.pythonhosted.org/packages/ef/dd/00e90bd1f6788f06ca5ea83a0ec8dd76350b38303bb8f09d2bf692eb1294/google_cloud_core-0.25.0-py2.py3-none-any.whl (52kB)
    100% |████████████████████████████████| 61kB 8.0MB/s ta 0:00:011
Collecting gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 (from google-cloud-pubsub==0.26.0; extra == "gcp"->apache-beam[gcp])
  Downloading https://files.pythonhosted.org/packages/bc/a7/0225bd7a95e037a0afa90b2dd9534d0c79cd62283a5bddb30a3197579cbc/gapic-google-cloud-pubsub-v1-0.15.4.tar.gz
Requirement already satisfied: ordereddict in /usr/local/envs/py2env/lib/python2.7/site-packages (from funcsigs>=1->mock<3.0.0,>=1.0.1->apache-beam[gcp]) (1.1)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/envs/py2env/lib/python2.7/site-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam[gcp]) (3.0.4)
Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/envs/py2env/lib/python2.7/site-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam[gcp]) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/envs/py2env/lib/python2.7/site-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam[gcp]) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/envs/py2env/lib/python2.7/site-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam[gcp]) (2018.4.16)
Collecting monotonic>=0.1 (from fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18; extra == "gcp"->apache-beam[gcp])
  Downloading https://files.pythonhosted.org/packages/ac/aa/063eca6a416f397bd99552c534c6d11d57f58f2e94c14780f3bbf818c4cf/monotonic-1.5-py2.py3-none-any.whl
Requirement already satisfied: google-auth-httplib2 in /usr/local/envs/py2env/lib/python2.7/site-packages (from google-cloud-core<0.26dev,>=0.25.0->google-cloud-bigquery==0.25.0; extra == "gcp"->apache-beam[gcp]) (0.0.3)
Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in /usr/local/envs/py2env/lib/python2.7/site-packages (from google-cloud-core<0.26dev,>=0.25.0->google-cloud-bigquery==0.25.0; extra == "gcp"->apache-beam[gcp]) (1.5.0)
Requirement already satisfied: google-gax<0.16dev,>=0.15.7 in /usr/local/envs/py2env/lib/python2.7/site-packages (from gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0; extra == "gcp"->apache-beam[gcp]) (0.15.16)
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in /usr/local/envs/py2env/lib/python2.7/site-packages (from gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0; extra == "gcp"->apache-beam[gcp]) (0.11.4)
Requirement already satisfied: cachetools>=2.0.0 in /usr/local/envs/py2env/lib/python2.7/site-packages (from google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-bigquery==0.25.0; extra == "gcp"->apache-beam[gcp]) (2.0.1)
Requirement already satisfied: ply==3.8 in /usr/local/envs/py2env/lib/python2.7/site-packages (from google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0; extra == "gcp"->apache-beam[gcp]) (3.8)
Building wheels for collected packages: proto-google-cloud-pubsub-v1, gapic-google-cloud-pubsub-v1
  Running setup.py bdist_wheel for proto-google-cloud-pubsub-v1 ... - done
  Stored in directory: /content/.cache/pip/wheels/d1/0d/9e/95e7192ab2625847ac40b2bc618800bf5b6c984cd572a83314
  Running setup.py bdist_wheel for gapic-google-cloud-pubsub-v1 ... - done
  Stored in directory: /content/.cache/pip/wheels/f4/2b/10/bdcbc9be2ae4e437232e118056f026025cf2cc46d6dcf0d69d
Successfully built proto-google-cloud-pubsub-v1 gapic-google-cloud-pubsub-v1
google-cloud-storage 1.6.0 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud-firestore 0.28.0 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud-bigtable 0.28.1 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud-dataflow 2.0.0 has requirement google-apitools==0.5.10, but you'll have google-apitools 0.5.20 which is incompatible.
google-cloud-dataflow 2.0.0 has requirement google-cloud-bigquery<0.24.0,>=0.23.0, but you'll have google-cloud-bigquery 0.25.0 which is incompatible.
google-cloud-dataflow 2.0.0 has requirement httplib2<0.10,>=0.8, but you'll have httplib2 0.11.3 which is incompatible.
google-cloud-dataflow 2.0.0 has requirement proto-google-cloud-datastore-v1==0.90.0, but you'll have proto-google-cloud-datastore-v1 0.90.4 which is incompatible.
google-cloud-dataflow 2.0.0 has requirement protobuf==3.2.0, but you'll have protobuf 3.5.2 which is incompatible.
pandas-gbq 0.3.0 has requirement google-cloud-bigquery>=0.28.0, but you'll have google-cloud-bigquery 0.25.0 which is incompatible.
google-cloud-monitoring 0.28.1 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud-error-reporting 0.28.0 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud-translate 1.3.1 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud-spanner 0.29.0 has requirement google-cloud-core[grpc]<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud-resource-manager 0.28.1 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud-speech 0.30.0 has requirement google-cloud-core[grpc]<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud-runtimeconfig 0.28.1 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud 0.33.1 has requirement google-cloud-bigquery<0.29dev,>=0.28.0, but you'll have google-cloud-bigquery 0.25.0 which is incompatible.
google-cloud 0.33.1 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud 0.33.1 has requirement google-cloud-pubsub<0.31dev,>=0.30.0, but you'll have google-cloud-pubsub 0.26.0 which is incompatible.
google-cloud-trace 0.17.0 has requirement google-cloud-core[grpc]<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud-logging 1.4.0 has requirement google-cloud-core[grpc]<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud-dns 0.28.0 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
google-cloud-datastore 1.4.0 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.25.0 which is incompatible.
Installing collected packages: typing, pyvcf, fastavro, pytz, monotonic, fasteners, google-apitools, proto-google-cloud-pubsub-v1, google-cloud-core, google-cloud-bigquery, gapic-google-cloud-pubsub-v1, google-cloud-pubsub, apache-beam
  Found existing installation: pytz 2016.7
Cannot uninstall 'pytz'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

我为这个全新的笔记本选择了python2内核。virtualenv也在版本16上。pip在版本18上。虚拟机是通过云shell使用datalab命令配置的，如果对此有任何帮助，将不胜感激。

Answer 1

问题是Datalab现在默认使用Python 3，您的pip安装是在Python 3中进行的，即使内核是Python 2，因为%%bash打开了一个新的shell，其中没有发生Python 2的conda激活。

要修复此问题，请更改apache-beamgcp的pip安装，如下所示：

%%bash
source activate py2env
conda install -y pytz
pip uninstall -y google-cloud-dataflow
pip install --upgrade apache-beam[gcp]