Python 3 upgrade (#3522)
authorOliver Bristow <evilumbrella+github@gmail.com>
Tue, 7 Jul 2020 00:24:27 +0000 (01:24 +0100)
committerGitHub <noreply@github.com>
Tue, 7 Jul 2020 00:24:27 +0000 (20:24 -0400)
commit4f7f90f2b823dc7e714ee2898b033cfc78f88cb0
tree66ccef5c984400eb282612fc032e5d36fd7b6fac
parent754416088bc040f2ea99e96f2c56580f807cd986
Python 3 upgrade (#3522)

* Initial Python 3 upgrade effort

* Fixes towards python3 support

 * update heronpy release scripts for python3
 * update dist Dockerfiles to only use python3
 * remove python2 from docker images
 * upgrade pylint for python3 support
 * upgrade PEX so transative dependencies are captured

Additionally:
 * fix Ubuntu 16.04 images
 * fix linting issues found by newer pylint

There is an issue with encapsulation in the builds where the global python3 environment is used
while PEX installs a nested transitive dependency of pylint: `pylint>astroid>wrapt`. This seems
to be because of logic in its setup.py which can be disabled with `WRAPT_INSTALL_EXTENSIONS=false`

* Fix new pylint issues

* update setuptools

* Make pex_pytest non-zip-safe

* Rough proto_library fix

The issue encountered was https://github.com/protocolbuffers/protobuf/issues/1491 which
may be fixed by a pending PR to protoc, or with a switch to the official protobuf rules
and the import_prefix parameter to proto_library.

* WIP: Fix python3 incompatibilities

 * bytes vs str issues
 * update kazoo
 * order of processes in executor test changed due to dict ordering?
 * some places needed / switched to // - may be more not caught by tests
 * add travis_wait as some stages going over 10 minutes without output in CI

TODO:
 * make sure the kazoo upgrade is correct, it was done only by updating package versoin

* Try fixing build time issue in travis

* Upgrade docker rules

* Upgrade to python3 in CI

* Fix python integration tests

* Fix more bytes vs str errors + update vagrant

* Update Travis to Python3.7 + fix Vagrant on mac

* Reduce requirement to python3.6 + py3 fixes

 * use universal_newline in popen instead of text in Popen for py3.6
 * fix bytes/str issues in deserialisation
 * fix file open modes
 * use set instead of sets.Set
 * fix __import__(level) default

* Update cloudpickle

* Fix python addressing of release.yaml

* Additions to get docker image builds working and tested

 * use new external pkg_* rules
 * add python to compile docker images until pkg_*
 * add --host_force_python=PY3 to other bazel.rc files

* WIP: Add CI for docker images/releases

 * use kind to create ephemeral clusters
 * start consolidating scripts with python

* Fix helm chart

* bytes vs str fix

* Mention Python 3.6 requirement in README.md

* updatedockerfile

Co-authored-by: Neng Lu <nlu@twitter.com>
Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
Co-authored-by: bed debug <huijunwu@users.noreply.github.com>
Co-authored-by: huijunwu <huijun.wu.2010@gmail.com>
336 files changed:
.travis.yml
README.md
WORKSPACE
bazel_configure.py
config/configure.ac
deploy/kubernetes/minikube/apiserver.yaml
deploy/kubernetes/minikube/bookkeeper.yaml
docker/base/Dockerfile.base.debian9
docker/compile/Dockerfile.centos7
docker/compile/Dockerfile.debian10
docker/compile/Dockerfile.debian9
docker/compile/Dockerfile.ubuntu14.04
docker/compile/Dockerfile.ubuntu16.04
docker/compile/Dockerfile.ubuntu18.04
docker/compile/Dockerfile.ubuntu20.04
docker/dist/Dockerfile.dist.centos7
docker/dist/Dockerfile.dist.debian10
docker/dist/Dockerfile.dist.debian9
docker/dist/Dockerfile.dist.ubuntu14.04
docker/dist/Dockerfile.dist.ubuntu16.04
docker/dist/Dockerfile.dist.ubuntu18.04
docker/dist/Dockerfile.dist.ubuntu20.04
docker/test/Dockerfile.centos7
docker/test/Dockerfile.ubuntu18.04
examples/src/python/bolt/consume_bolt.py
examples/src/python/bolt/count_bolt.py
examples/src/python/bolt/half_ack_bolt.py
examples/src/python/bolt/stateful_count_bolt.py
examples/src/python/bolt/stream_aggregate_bolt.py
examples/src/python/bolt/window_size_bolt.py
examples/src/python/custom_grouping_topology.py
examples/src/python/half_acking_topology.py
examples/src/python/join_streamlet_topology.py
examples/src/python/misc/test_task_hook.py
examples/src/python/multi_stream_topology.py
examples/src/python/pulsar_word_count_streamlet.py
examples/src/python/spout/multi_stream_spout.py
examples/src/python/spout/stateful_word_spout.py
examples/src/python/spout/word_spout.py
examples/src/python/stateful_word_count_topology.py
examples/src/python/window_size_topology.py
examples/src/python/word_count_streamlet.py
examples/src/python/word_count_topology.py
heron/common/src/python/pex_loader.py
heron/common/src/python/utils/log.py
heron/common/src/python/utils/proc.py
heron/common/tests/python/pex_loader/constants.py
heron/common/tests/python/pex_loader/pex_loader_unittest.py
heron/common/tests/python/pex_loader/testdata/src/sample.py
heron/executor/src/python/heron_executor.py
heron/executor/tests/python/heron_executor_unittest.py
heron/instance/src/python/basics/base_instance.py
heron/instance/src/python/basics/bolt_instance.py
heron/instance/src/python/basics/spout_instance.py
heron/instance/src/python/instance/st_heron_instance.py
heron/instance/src/python/network/event_looper.py
heron/instance/src/python/network/gateway_looper.py
heron/instance/src/python/network/heron_client.py
heron/instance/src/python/network/metricsmgr_client.py
heron/instance/src/python/network/protocol.py
heron/instance/src/python/network/socket_options.py
heron/instance/src/python/network/st_stmgr_client.py
heron/instance/src/python/utils/metrics/metrics_helper.py
heron/instance/src/python/utils/metrics/py_metrics.py
heron/instance/src/python/utils/misc/communicator.py
heron/instance/src/python/utils/misc/custom_grouping_helper.py
heron/instance/src/python/utils/misc/outgoing_tuple_helper.py
heron/instance/src/python/utils/misc/pplan_helper.py
heron/instance/src/python/utils/misc/serializer_helper.py
heron/instance/src/python/utils/system_config.py
heron/instance/src/python/utils/system_constants.py
heron/instance/src/python/utils/topology/topology_context_impl.py
heron/instance/src/python/utils/tuple.py
heron/instance/tests/python/mock_protobuf.py
heron/instance/tests/python/network/event_looper_unittest.py
heron/instance/tests/python/network/gateway_looper_unittest.py
heron/instance/tests/python/network/heron_client_unittest.py
heron/instance/tests/python/network/metricsmgr_client_unittest.py
heron/instance/tests/python/network/mock_generator.py
heron/instance/tests/python/network/mock_generator_client.py
heron/instance/tests/python/network/protocol_unittest.py
heron/instance/tests/python/network/st_stmgr_client_unittest.py
heron/instance/tests/python/utils/communicator_unittest.py
heron/instance/tests/python/utils/custom_grouping_unittest.py
heron/instance/tests/python/utils/global_metrics_unittest.py
heron/instance/tests/python/utils/log_unittest.py
heron/instance/tests/python/utils/metrics_helper_unittest.py
heron/instance/tests/python/utils/mock_generator.py
heron/instance/tests/python/utils/outgoing_tuple_helper_unittest.py
heron/instance/tests/python/utils/pplan_helper_unittest.py
heron/instance/tests/python/utils/py_metrics_unittest.py
heron/instance/tests/python/utils/topology_context_impl_unittest.py
heron/instance/tests/python/utils/tuple_helper_unittest.py
heron/proto/BUILD
heron/shell/src/python/handlers/browsehandler.py
heron/shell/src/python/handlers/downloadhandler.py
heron/shell/src/python/handlers/filedatahandler.py
heron/shell/src/python/handlers/filehandler.py
heron/shell/src/python/handlers/filestatshandler.py
heron/shell/src/python/handlers/healthhandler.py
heron/shell/src/python/handlers/jmaphandler.py
heron/shell/src/python/handlers/jstackhandler.py
heron/shell/src/python/handlers/killexecutorhandler.py
heron/shell/src/python/handlers/memoryhistogramhandler.py
heron/shell/src/python/handlers/pidhandler.py
heron/shell/src/python/handlers/pmaphandler.py
heron/shell/src/python/main.py
heron/shell/src/python/utils.py
heron/statemgrs/src/python/BUILD
heron/statemgrs/src/python/config.py
heron/statemgrs/src/python/configloader.py
heron/statemgrs/src/python/filestatemanager.py
heron/statemgrs/src/python/log.py
heron/statemgrs/src/python/stateexceptions.py
heron/statemgrs/src/python/statemanager.py
heron/statemgrs/src/python/statemanagerfactory.py
heron/statemgrs/src/python/zkstatemanager.py
heron/statemgrs/tests/python/configloader_unittest.py
heron/statemgrs/tests/python/statemanagerfactory_unittest.py
heron/statemgrs/tests/python/zkstatemanager_unittest.py
heron/tools/admin/src/python/main.py
heron/tools/admin/src/python/standalone.py
heron/tools/cli/src/python/activate.py
heron/tools/cli/src/python/args.py
heron/tools/cli/src/python/cdefs.py
heron/tools/cli/src/python/cli_helper.py
heron/tools/cli/src/python/cliconfig.py
heron/tools/cli/src/python/config.py
heron/tools/cli/src/python/deactivate.py
heron/tools/cli/src/python/execute.py
heron/tools/cli/src/python/help.py
heron/tools/cli/src/python/jars.py
heron/tools/cli/src/python/kill.py
heron/tools/cli/src/python/main.py
heron/tools/cli/src/python/opts.py
heron/tools/cli/src/python/rest.py
heron/tools/cli/src/python/restart.py
heron/tools/cli/src/python/result.py
heron/tools/cli/src/python/submit.py
heron/tools/cli/src/python/update.py
heron/tools/cli/src/python/version.py
heron/tools/cli/tests/python/client_command_unittest.py
heron/tools/cli/tests/python/opts_unittest.py
heron/tools/common/src/python/access/fetch.py
heron/tools/common/src/python/access/heron_api.py
heron/tools/common/src/python/access/query.py
heron/tools/common/src/python/access/tracker_access.py
heron/tools/common/src/python/utils/classpath.py
heron/tools/common/src/python/utils/config.py
heron/tools/common/src/python/utils/heronparser.py
heron/tools/explorer/src/python/args.py
heron/tools/explorer/src/python/clusters.py
heron/tools/explorer/src/python/help.py
heron/tools/explorer/src/python/logicalplan.py
heron/tools/explorer/src/python/main.py
heron/tools/explorer/src/python/opts.py
heron/tools/explorer/src/python/physicalplan.py
heron/tools/explorer/src/python/topologies.py
heron/tools/explorer/src/python/version.py
heron/tools/explorer/tests/python/explorer_unittest.py
heron/tools/tracker/src/python/config.py
heron/tools/tracker/src/python/constants.py
heron/tools/tracker/src/python/graph.py
heron/tools/tracker/src/python/handlers/basehandler.py
heron/tools/tracker/src/python/handlers/clustershandler.py
heron/tools/tracker/src/python/handlers/containerfilehandler.py
heron/tools/tracker/src/python/handlers/defaulthandler.py
heron/tools/tracker/src/python/handlers/exceptionhandler.py
heron/tools/tracker/src/python/handlers/exceptionsummaryhandler.py
heron/tools/tracker/src/python/handlers/executionstatehandler.py
heron/tools/tracker/src/python/handlers/jmaphandler.py
heron/tools/tracker/src/python/handlers/jstackhandler.py
heron/tools/tracker/src/python/handlers/logicalplanhandler.py
heron/tools/tracker/src/python/handlers/machineshandler.py
heron/tools/tracker/src/python/handlers/mainhandler.py
heron/tools/tracker/src/python/handlers/memoryhistogramhandler.py
heron/tools/tracker/src/python/handlers/metadatahandler.py
heron/tools/tracker/src/python/handlers/metricshandler.py
heron/tools/tracker/src/python/handlers/metricsqueryhandler.py
heron/tools/tracker/src/python/handlers/metricstimelinehandler.py
heron/tools/tracker/src/python/handlers/packingplanhandler.py
heron/tools/tracker/src/python/handlers/physicalplanhandler.py
heron/tools/tracker/src/python/handlers/pidhandler.py
heron/tools/tracker/src/python/handlers/runtimestatehandler.py
heron/tools/tracker/src/python/handlers/schedulerlocationhandler.py
heron/tools/tracker/src/python/handlers/stateshandler.py
heron/tools/tracker/src/python/handlers/topologieshandler.py
heron/tools/tracker/src/python/handlers/topologyconfighandler.py
heron/tools/tracker/src/python/handlers/topologyhandler.py
heron/tools/tracker/src/python/javaobj.py
heron/tools/tracker/src/python/main.py
heron/tools/tracker/src/python/metricstimeline.py
heron/tools/tracker/src/python/pyutils.py
heron/tools/tracker/src/python/query.py
heron/tools/tracker/src/python/query_operators.py
heron/tools/tracker/src/python/topology.py
heron/tools/tracker/src/python/tracker.py
heron/tools/tracker/src/python/utils.py
heron/tools/tracker/tests/python/mock_proto.py
heron/tools/ui/src/python/args.py
heron/tools/ui/src/python/consts.py
heron/tools/ui/src/python/handlers/api/metrics.py
heron/tools/ui/src/python/handlers/api/topology.py
heron/tools/ui/src/python/handlers/base.py
heron/tools/ui/src/python/handlers/common/consts.py
heron/tools/ui/src/python/handlers/common/utils.py
heron/tools/ui/src/python/handlers/mainhandler.py
heron/tools/ui/src/python/handlers/notfound.py
heron/tools/ui/src/python/handlers/ranges.py
heron/tools/ui/src/python/handlers/topology.py
heron/tools/ui/src/python/main.py
heronpy/api/api_constants.py
heronpy/api/bolt/bolt.py
heronpy/api/bolt/window_bolt.py
heronpy/api/cloudpickle.py
heronpy/api/component/base_component.py
heronpy/api/component/component_spec.py
heronpy/api/custom_grouping.py
heronpy/api/global_metrics.py
heronpy/api/metrics.py
heronpy/api/serializer.py
heronpy/api/spout/spout.py
heronpy/api/state/state.py
heronpy/api/state/stateful_component.py
heronpy/api/stream.py
heronpy/api/task_hook.py
heronpy/api/tests/python/component_unittest.py
heronpy/api/tests/python/metrics_unittest.py
heronpy/api/tests/python/serializer_unittest.py
heronpy/api/tests/python/stream_unittest.py
heronpy/api/tests/python/topology_unittest.py
heronpy/api/topology.py
heronpy/api/topology_context.py
heronpy/api/tuple.py
heronpy/connectors/mock/arraylooper.py
heronpy/connectors/pulsar/pulsarspout.py
heronpy/connectors/pulsar/pulsarstreamlet.py
heronpy/connectors/textfiles/textfilesgenerator.py
heronpy/proto/BUILD
heronpy/streamlet/builder.py
heronpy/streamlet/config.py
heronpy/streamlet/context.py
heronpy/streamlet/generator.py
heronpy/streamlet/impl/consumebolt.py
heronpy/streamlet/impl/contextimpl.py
heronpy/streamlet/impl/filterbolt.py
heronpy/streamlet/impl/flatmapbolt.py
heronpy/streamlet/impl/generatorspout.py
heronpy/streamlet/impl/joinbolt.py
heronpy/streamlet/impl/logbolt.py
heronpy/streamlet/impl/mapbolt.py
heronpy/streamlet/impl/reducebykeyandwindowbolt.py
heronpy/streamlet/impl/reducebywindowbolt.py
heronpy/streamlet/impl/repartitionbolt.py
heronpy/streamlet/impl/streamletboltbase.py
heronpy/streamlet/impl/streamletspoutbase.py
heronpy/streamlet/impl/supplierspout.py
heronpy/streamlet/impl/transformbolt.py
heronpy/streamlet/impl/unionbolt.py
heronpy/streamlet/keyedwindow.py
heronpy/streamlet/resources.py
heronpy/streamlet/runner.py
heronpy/streamlet/streamlet.py
heronpy/streamlet/transformoperator.py
heronpy/streamlet/window.py
heronpy/streamlet/windowconfig.py
integration_test/src/python/common/status.py
integration_test/src/python/integration_test/common/bolt/count_aggregator_bolt.py
integration_test/src/python/integration_test/common/bolt/double_tuples_bolt.py
integration_test/src/python/integration_test/common/bolt/identity_bolt.py
integration_test/src/python/integration_test/common/bolt/word_count_bolt.py
integration_test/src/python/integration_test/common/spout/ab_spout.py
integration_test/src/python/integration_test/core/aggregator_bolt.py
integration_test/src/python/integration_test/core/batch_bolt.py
integration_test/src/python/integration_test/core/constants.py
integration_test/src/python/integration_test/core/integration_test_bolt.py
integration_test/src/python/integration_test/core/integration_test_spout.py
integration_test/src/python/integration_test/core/terminal_bolt.py
integration_test/src/python/integration_test/core/test_runner.py
integration_test/src/python/integration_test/core/test_topology_builder.py
integration_test/src/python/integration_test/topology/all_grouping/all_grouping.py
integration_test/src/python/integration_test/topology/basic_one_task/basic_one_task.py
integration_test/src/python/integration_test/topology/bolt_double_emit_tuples/bolt_double_emit_tuples.py
integration_test/src/python/integration_test/topology/fields_grouping/fields_grouping.py
integration_test/src/python/integration_test/topology/global_grouping/global_grouping.py
integration_test/src/python/integration_test/topology/multi_spouts_multi_tasks/multi_spouts_multi_tasks.py
integration_test/src/python/integration_test/topology/none_grouping/none_grouping.py
integration_test/src/python/integration_test/topology/one_bolt_multi_tasks/one_bolt_multi_tasks.py
integration_test/src/python/integration_test/topology/one_spout_bolt_multi_tasks/one_spout_bolt_multi_tasks.py
integration_test/src/python/integration_test/topology/one_spout_multi_tasks/one_spout_multi_tasks.py
integration_test/src/python/integration_test/topology/one_spout_two_bolts/one_spout_two_bolts.py
integration_test/src/python/integration_test/topology/shuffle_grouping/shuffle_grouping.py
integration_test/src/python/integration_test/topology/streamlet/word_count_streamlet.py
integration_test/src/python/integration_test/topology/test_topology_main.py
integration_test/src/python/local_test_runner/main.py
integration_test/src/python/local_test_runner/test_kill_bolt.py
integration_test/src/python/local_test_runner/test_kill_metricsmgr.py
integration_test/src/python/local_test_runner/test_kill_stmgr.py
integration_test/src/python/local_test_runner/test_kill_stmgr_metricsmgr.py
integration_test/src/python/local_test_runner/test_kill_tmaster.py
integration_test/src/python/local_test_runner/test_scale_up.py
integration_test/src/python/local_test_runner/test_template.py
integration_test/src/python/test_runner/main.py
integration_test/src/python/topology_test_runner/main.py
scripts/applatix/build.sh
scripts/applatix/test.sh
scripts/applatix/testutils.sh
scripts/packages/BUILD
scripts/packages/heronpy/setup.py.template
scripts/release/docker-images [new file with mode: 0755]
scripts/setup-intellij.sh
scripts/shutils/common.sh
scripts/shutils/save-logs.py
scripts/travis/build.sh
scripts/travis/k8s.sh [new file with mode: 0755]
scripts/travis/k8s.sh.kind.yaml [new file with mode: 0644]
scripts/travis/test.sh
third_party/python/cpplint/BUILD
third_party/python/cpplint/cpplint.py
third_party/python/pylint/BUILD
third_party/python/pylint/main.py [deleted file]
tools/bazel.rc
tools/docker/bazel.rc
tools/rules/genproto.bzl
tools/rules/pex/BUILD
tools/rules/pex/pex_rules.bzl
tools/rules/pex/wrapper/pex_wrapper.py
tools/rules/pex/wrapper/setup.py
tools/rules/proto.bzl [deleted file]
tools/travis/bazel.rc
vagrant/README.md
vagrant/Vagrantfile
vagrant/init.sh
vagrant/local-ci.sh [new file with mode: 0755]
website2/docs/schedulers-nomad.md
website2/website/versioned_docs/version-0.20.0-incubating/compiling-docker.md