Python SDK for building, training, and deploying ML models

What this PR does / why we need it:

This PR contains implementation for fairing.backends.KubeflowAzureBackend which allows storing build context to Azure Storage and using Azure Container Registry as the Docker Registry.

Special notes for your reviewer:

1. An experimental CI is being worked on at https://dev.azure.com/kubeflow/kubeflow/_build with the intent of running Azure-specific integration tests there.

This change is 

Our current debug info is so simple,especially, the confuse time stamp, also don't show the target debug file's location , [I 200108 13:36:53 config:131] Using preprocessor: <kubeflow.fairing.preprocessors.base.BasePreProcessor object at 0x1017af9b0> after changed, it will be more meaningful, for example : INFO|2020-01-08 13:18:33|/Users/llhu/Library/Python/3.7/lib/python/site-packages/werkzeug/_internal.py|122| * Running on http://127.0.0.1:8080/

This change is 

What this PR does / why we need it: Add annotations param to TfJob for ability to pass it to Job.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes #

Special notes for your reviewer:

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

NONE

This change is 

/kind feature

Describe the solution you'd like Kubeflow Fairing should have baisc documents for Fairing APIs, so that the user can just reading the documents when calling APIs, instead of reading code.

Anything else you would like to add:

This adds a configMap builder and architechture to allow using configmaps for the code for notebooks during distributed run. This avoids having to build images and therefore can aid distributed training during early exploration.

This change is 

This patch enables kubeflow-fairing to run in AWS environment(EKS, S3, ECR...) like this.

import os
import fairing

DOCKER_REGISTRY = '{account_id}.dkr.ecr.{region}.amazonaws.com'
PY_VERSION = ".".join([str(x) for x in sys.version_info[0:3]])
BASE_IMAGE = 'python:{}'.format(PY_VERSION)

from fairing import TrainJob
from fairing.backends import KubeflowAWSBackend
train_job = TrainJob(HousingServe, BASE_IMAGE, input_files=['ames_dataset/train.csv', "requirements.txt"],
                     docker_registry=DOCKER_REGISTRY, backend=KubeflowAWSBackend())
train_job.submit()

Fixes #282

This change is 

What this PR does / why we need it: Fix bugs in the function predict in examples

Which issue(s) this PR fixes: Fixes #258

/assign @jlewi /assign @zhenghuiwang

This change is 

What this PR does / why we need it:

Because kfserving SDK version is changed to be consistent with kfserving release (0.2.1), to avoid confusing user, made decision by kfserving comminication, drop old version, and release new one to align with kfserving current release. so kfserving new release is 0.2.1.0, update fairing accordingly.

This change is 

I've encountered the issue similar to described in https://github.com/kubeflow/fairing/issues/452. It occurs when user uses function preprocessor + cloudpicke version containing cloudpickle_fast.py + python 3.8+.

The root cause is that newer versions of cloudpickle contain cloudpickle_fast.py and that file isn't copied by preprocessor during image preparation.

So, this PR aims to append mentioned above file along with the rest of cloudpickle package.

By the way, what's the reason for copying that package into image ? Wouldn't it be better to copy egg there f.e. and install it on start up ?

What this PR does / why we need it: Enhance kfserving integration, move to new version of kfserving, and the CRD has been changed to Inferenceservice, updated in Fairing accordingly.

Which issue(s) this PR fixes: Fixes #

Special notes for your reviewer:

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

This change is 

What this PR does / why we need it: Support KFServing in Fairing.

Which issue(s) this PR fixes : Fixes #290

Special notes for your reviewer:

1. The intergration tests for the kfserving are not added, since the automation platform should not have the kfserving installation. But I tested most of cases in the on-prem cluster, and we can add later once the kfserving is installed for test platform.

2. I updated the log() function in the fairing/kubernetes/manager.py to satify the implements, since there are two container in the pod, we need to point out which container logs will be got.

Release note:

Support KFServing to deploy model service

This change is 

I'm trying to run mnist_e2e_on_prem.ipynb to train a model on AWS, but in creating PV/PVC step, msrestazure is required to install, does this example only support azure? How should I set it to run on AWS or local environment?

[W 220617 06:49:11 append:81] Uploading registry.cn-hangzhou.aliyuncs.com/rory602/fairing-job:37DF66B9 [I 220617 06:49:12 docker_session_:284] Layer sha256:bcfb845ebce120ef8d7b6c3a1600b7b18de2cff8a66ddc57004034bc7f74ce80 pushed. [I 220617 06:49:39 docker_session_:284] Layer sha256:793b1b0c3ddfb6c3de731b9a027b6997bd786ff769648045a8f5a428a1406f5e pushed. [I 220617 06:49:40 docker_session_:284] Layer sha256:b62b8281eb2dcbcf3f4ac5199f7d729747bb256dd5104f44fd9d0fe5fe939353 pushed. [I 220617 06:49:40 docker_session_:284] Layer sha256:9b829c73b52b92b97d5c07a54fb0f3e921995a296c714b53a32ae67d19231fcd pushed. [I 220617 06:49:41 docker_session_:284] Layer sha256:700a07047b6b1612c1dc2a9ce93ec19913d129206585879749e8fe177f567fa9 pushed. [I 220617 06:49:46 docker_session_:284] Layer sha256:fcb6d5f7c98604476fda91fe5f61be5b56fdc398814fb15f7ea998f53023e774 pushed. [I 220617 06:50:25 docker_session_:284] Layer sha256:cb5b7ae361722f070eca53f35823ed21baa85d61d5d95cd5a95ab53d740cdd56 pushed. [I 220617 06:50:39 docker_session_:284] Layer sha256:35b0d149a82cb315c7d490f54f67bb122de76a15124c6524128fab47cba63bbd pushed. [I 220617 06:53:43 docker_session_:284] Layer sha256:6494e4811622b31c027ccac322ca463937fd805f569a93e6f15c01aade718793 pushed. [I 220617 06:53:57 docker_session_:284] Layer sha256:0e29546d541cdbd309281d21a73a9d1db78665c1b95b74f32b009e0b77a6e1e3 pushed. [I 220617 07:14:21 docker_session_:284] Layer sha256:6f9f74896dfa93fe0172f594faba85e0b4e8a0481a0fefd9112efc7e4d3c78f7 pushed. [E 220617 07:15:07 docker_session_:332] Error during upload of: registry.cn-hangzhou.aliyuncs.com/rory602/fairing-job:37DF66B9 ^CTraceback (most recent call last): File "/home/jovyan/.conda/envs/py38/lib/python3.8/runpy.py", line 192, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/jovyan/.conda/envs/py38/lib/python3.8/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/jovyan/.local/share/code-server/extensions/ms-python.python-2021.5.926500501/pythonFiles/lib/python/debugpy/main.py", line 45, in cli.main() File "/home/jovyan/.local/share/code-server/extensions/ms-python.python-2021.5.926500501/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main run() File "/home/jovyan/.local/share/code-server/extensions/ms-python.python-2021.5.926500501/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file runpy.run_path(target_as_str, run_name=compat.force_str("main")) File "/home/jovyan/.conda/envs/py38/lib/python3.8/runpy.py", line 262, in run_path return _run_module_code(code, init_globals, run_name, File "/home/jovyan/.conda/envs/py38/lib/python3.8/runpy.py", line 95, in _run_module_code _run_code(code, mod_globals, init_globals, File "/home/jovyan/.conda/envs/py38/lib/python3.8/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/jovyan/vol-1/fairing/helloworld.py", line 19, in remote_train() File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/kubeflow/fairing/config.py", line 163, in ret_fn self.run() File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/kubeflow/fairing/config.py", line 140, in run builder.build() File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/kubeflow/fairing/builders/append/append.py", line 58, in build self.timed_push(transport, src, new_img, dst) File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/kubeflow/fairing/builders/append/append.py", line 96, in timed_push self._push(transport, src, img, dst) File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/kubeflow/fairing/builders/append/append.py", line 82, in push session.upload(img) File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/containerregistry/client/v2_2/docker_session.py", line 321, in upload future.result() File "/home/jovyan/.conda/envs/py38/lib/python3.8/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/home/jovyan/.conda/envs/py38/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result raise self.exception File "/home/jovyan/.conda/envs/py38/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/containerregistry/client/v2_2/docker_session.py", line 283, in _upload_one self.put_blob(image, digest) File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/containerregistry/client/v2_2/docker_session.py", line 206, in _put_blob self.patch_upload(image, digest) File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/containerregistry/client/v2_2/docker_session.py", line 165, in _patch_upload resp, unused_content = self.transport.Request( File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/containerregistry/client/v2_2/docker_http.py", line 385, in Request resp, content = self.transport.request( File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/containerregistry/transport/transport_pool.py", line 61, in request return transport.request(*args, **kwargs) File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/httplib2/init.py", line 1701, in request (response, content) = self._request( File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/httplib2/init.py", line 1421, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/home/jovyan/.conda/envs/py38/lib/python3.8/site-packages/httplib2/init.py", line 1344, in _conn_request conn.request(method, request_uri, body, headers) File "/home/jovyan/.conda/envs/py38/lib/python3.8/http/client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "/home/jovyan/.conda/envs/py38/lib/python3.8/http/client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/home/jovyan/.conda/envs/py38/lib/python3.8/http/client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/home/jovyan/.conda/envs/py38/lib/python3.8/http/client.py", line 1043, in _send_output self.send(chunk) File "/home/jovyan/.conda/envs/py38/lib/python3.8/http/client.py", line 965, in send self.sock.sendall(data) File "/home/jovyan/.conda/envs/py38/lib/python3.8/ssl.py", line 1204, in sendall v = self.send(byte_view[count:]) File "/home/jovyan/.conda/envs/py38/lib/python3.8/ssl.py", line 1173, in send return self._sslobj.write(data) OverflowError: string longer than 2147483647 bytes

What this PR does / why we need it: When trying to pull an image built by append builder using podman, blob size mismatch error occurs. This is due to config size mismatch in manifest.

current append builder only appends a new layer and updates config digest in the manifest. so the config size of the newly built image manifest remains the same of that of the base image used to build a new image. This doesn't seem to be a problem when trying to pull an image using docker but when trying to pull an image using podman, size mismatch error occurs.

This PR added a code to add config size information in the manifest.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes #568

Special notes for your reviewer:

Release note:

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.]

I built a simple mnist ML model using an append builder. When trying to pull this image using podman I get a "Error: writing blob: blob size mismatch " error.

using docker pull works fine with this image.

What did you expect to happen: For the image to be pulled without any errorr using podman.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Fairing version: (use python -c "import kubeflow.fairing; print(kubeflow.fairing.__version__)"): 0.5.3
• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard):
• Minikube version:
• Kubernetes version: (use kubectl version): 1.19.4
• OS (e.g. from /etc/os-release):

NOTE: If you are using fair from master, please provide us the git commit hash.

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes #563 ，fixes #565

fixes the dependence is too old.

• use newest kfserving 0.6.1 , which same version like kubeflow-components-versions

• update to newest training job 1.3.0，kubeflow-tfjob and kubeflow-pytorchjob even never maintain.

/kind bug

What steps did you take and what happened: just use pip install kubeflow-fairing

keep installing for a long time and it try to install same package with different version.

finally I try this pip install kubeflow-fairing --use-deprecated=legacy-resolver

Then I run mnist e2e example py

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import uuid

import yaml
from kubeflow import fairing
from kubeflow.fairing.kubernetes.utils import mounting_pvc
from kubernetes import client as k8s_client
from kubernetes import config as k8s_config

DOCKER_REGISTRY = '10.19.64.203:8080'
my_namespace = 'kserve-test'

num_chief = 1  # number of Chief in TFJob
num_ps = 1  # number of PS in TFJob
num_workers = 2  # number of Worker in TFJob
model_dir = "/mnt"
export_path = "/mnt/export"
train_steps = "1000"
batch_size = "100"
learning_rate = "0.01"

pvc_name = 'mnist-pvc'
pvc_yaml = f'''
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {pvc_name}
  namespace: {my_namespace}
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 10Gi
'''

k8s_config.load_kube_config()

k8s_core_api = k8s_client.CoreV1Api()
# k8s_core_api.create_persistent_volume(yaml.safe_load(pv_yaml))
k8s_core_api.create_namespaced_persistent_volume_claim(my_namespace, yaml.safe_load(pvc_yaml))

tfjob_name = f'mnist-training-{uuid.uuid4().hex[:4]}'

output_map = {
    "Dockerfile": "Dockerfile",
    "mnist.py": "mnist.py"
}

command = ["python",
           "/opt/mnist.py",
           "--tf-model-dir=" + model_dir,
           "--tf-export-dir=" + export_path,
           "--tf-train-steps=" + train_steps,
           "--tf-batch-size=" + batch_size,
           "--tf-learning-rate=" + learning_rate]

fairing.config.set_preprocessor('python', command=command, path_prefix="/app", output_map=output_map)
fairing.config.set_builder(name='docker', registry=DOCKER_REGISTRY,
                           image_name="mnist", dockerfile_path="Dockerfile")

fairing.config.set_deployer(name='tfjob', namespace=my_namespace, stream_log=False, job_name=tfjob_name,
                            chief_count=num_chief, worker_count=num_workers, ps_count=num_ps,
                            pod_spec_mutators=[mounting_pvc(pvc_name=pvc_name, pvc_mount_path=model_dir)])
fairing.config.run()

What did you expect to happen:

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Fairing version: (use python -c "import kubeflow.fairing; print(kubeflow.fairing.__version__)"):

(.env) ➜  kubeflow git:(master) ✗ python -c "import kubeflow.fairing; print(kubeflow.fairing.__version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kubeflow/fairing/__init__.py", line 2, in <module>
    from kubeflow.fairing.ml_tasks.tasks import TrainJob, PredictionEndpoint
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kubeflow/fairing/ml_tasks/tasks.py", line 4, in <module>
    from kubeflow.fairing.backends import KubernetesBackend
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kubeflow/fairing/backends/__init__.py", line 1, in <module>
    from kubeflow.fairing.backends.backends import *
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kubeflow/fairing/backends/backends.py", line 8, in <module>
    from kubeflow.fairing.builders.cluster import gcs_context
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kubeflow/fairing/builders/cluster/gcs_context.py", line 6, in <module>
    from kubeflow.fairing.kubernetes.manager import client, KubeManager
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kubeflow/fairing/kubernetes/manager.py", line 6, in <module>
    from kfserving import KFServingClient
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kfserving/__init__.py", line 16, in <module>
    from kfserving.kfmodel import KFModel
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kfserving/kfmodel.py", line 24, in <module>
    from ray.serve.utils import ServeRequest
ImportError: cannot import name 'ServeRequest' from 'ray.serve.utils' (/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/ray/serve/utils.py)

(.env) ➜  kubeflow git:(master) ✗ pip list |grep kube                                                     
kubeflow-fairing               1.0.2
kubeflow-pytorchjob            0.1.3
kubeflow-tfjob                 0.1.3
kubernetes                     10.0.1

• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard):

dev_local

• Kubernetes version: (use kubectl version): k3s Kubernetes 1.19
• OS (e.g. from /etc/os-release): core run on osx
  k3s on centos7

NOTE: If you are using fair from master, please provide us the git commit hash.