Kubeflow Pipelines — End-to-End MLOps on Kubernetes with Experiments, Artifacts, and Serving

MLOps That Lives Where Your Cluster Already Is

Most machine-learning code starts life in a notebook and dies there. The gap between “it worked on my laptop” and “it runs every night, reproducibly, on shared infrastructure” is where MLOps lives — and it is mostly a Kubernetes problem before it is a modelling one.

Kubeflow is the answer that grew up inside the community that also gave us Kubernetes. It is not one tool but a family of subprojects — Pipelines, Katib, Notebooks, the Trainer, a Spark operator, a Model Registry, and a central dashboard — that together make the cluster a first-class ML platform.

This guide centres on Kubeflow Pipelines — the orchestration heart of the platform — and follows a single thread through it: author a pipeline in Python, pass typed artifacts between steps, log metrics, tune hyperparameters with Katib, and serve the result with KServe.

The Platform, Briefly

Before the code, a map. Kubeflow is deliberately modular, and you rarely deploy all of it. Each subproject solves one stage of the lifecycle and speaks to the cluster through custom resources.

• Pipelines. Build, run, and track multi-step ML workflows as DAGs. This is where most teams start.
• Katib. Hyperparameter tuning and neural architecture search, driven by an Experiment custom resource.
• Notebooks. Managed Jupyter environments running as pods, for interactive development.
• Trainer. Distributed training and LLM fine-tuning across many pods and GPUs.
• KServe.An ecosystem project for model serving — the natural deployment target for a pipeline’s output.

If you already run an MLflow-based ML pipeline, think of Kubeflow as the Kubernetes-native alternative that trades MLflow’s lightweight tracking server for a full cluster-resident platform. The two overlap, and many teams run them together.

Components — Type-Annotated Python Functions

A pipeline is a graph of components, and the smallest useful thing to know is that a component is just a Python function with type hints. The @dsl.component decorator turns that function into a self-contained, containerized step the backend can schedule.

from kfp import dsl

@dsl.component
def say_hello(name: str) -> str:
    hello_text = f'Hello, {name}!'
    print(hello_text)
    return hello_text

At compile time the SDK captures the function’s source, its type-annotated signature, and its dependencies. At run time the backend pulls a base image, installs any extra packages, and executes the function in a fresh container. Because dependencies install at run time, you iterate on the code without rebuilding an image for every change.

For anything beyond a toy, pin the base image and declare packages explicitly. Both are decorator arguments, and both are the difference between a reproducible run and a mysterious drift six months later.

@dsl.component(
    base_image="python:3.12-slim-bookworm",
    packages_to_install=["pandas>=2.2.3"],
)
def process_data(output_data: dsl.Output[dsl.Dataset]):
    import pandas as pd
    df = pd.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]})
    df.to_csv(output_data.path, index=False)

Artifacts — Passing Data, Not Just Values

KFP draws a sharp line between parameters and artifacts. Parameters are small typed values — an int, a string, a config — passed by value. Artifacts are larger objects — datasets, models, metrics — passed by reference, with the backend handling storage and lineage for you.

You declare an artifact by wrapping a type in dsl.Input[...] or dsl.Output[...]. The generic types are Dataset, Model, Metrics, and the base Artifact, each mapping to an ML Metadata schema so the platform can track provenance across runs.

from kfp.dsl import component, Input, Output, Dataset, Model

@component(packages_to_install=["scikit-learn>=1.4"])
def train_model(dataset: Input[Dataset], model: Output[Model]):
    import pandas as pd
    from sklearn.linear_model import LogisticRegression
    import joblib

    df = pd.read_csv(dataset.path)      # read the input artifact
    clf = LogisticRegression().fit(df[["x"]], df["y"])

    joblib.dump(clf, model.path)        # write to the output artifact
    model.metadata["framework"] = "scikit-learn"

Every artifact object exposes three things you actually use: .path, a local file path to read from or write to; .uri, the durable storage location the backend copies to and from; and .metadata, a free-form key-value dictionary for lineage. Write to .path and the backend moves the bytes to .uri automatically. Treat input artifacts as immutable.

Metrics and Visualizations

Two artifact types are purpose-built for evaluation output. Metrics holds scalar values through log_metric(name, value), and ClassificationMetrics renders confusion matrices and ROC curves in the Pipelines UI. Logging them is a one-liner inside a component.

from kfp.dsl import component, Output, Metrics, ClassificationMetrics

@component(packages_to_install=["scikit-learn>=1.4"])
def evaluate(metrics: Output[Metrics], cmatrix: Output[ClassificationMetrics]):
    from sklearn import datasets, model_selection
    from sklearn.linear_model import SGDClassifier
    from sklearn.metrics import confusion_matrix

    iris = datasets.load_iris()
    xtr, xte, ytr, yte = model_selection.train_test_split(
        iris["data"], iris["target"], test_size=0.3)
    clf = SGDClassifier().fit(xtr, ytr)

    metrics.log_metric("accuracy", clf.score(xte, yte))
    preds = model_selection.cross_val_predict(clf, xtr, ytr, cv=3)
    cmatrix.log_confusion_matrix(
        ["Setosa", "Versicolour", "Virginica"],
        confusion_matrix(ytr, preds).tolist(),  # numpy array -> list
    )

Note the .tolist() conversion — log_confusion_matrix expects plain Python lists, not NumPy arrays, and the category list length must match the matrix dimensions or the SDK raises a ValueError. The same ClassificationMetrics artifact also carries log_roc_curve(fpr, tpr, threshold) for ROC visualizations.

Assembling the Pipeline

A pipeline is another decorated Python function — @dsl.pipeline— that calls components and wires their outputs to the next step’s inputs. You do not build the DAG by hand; the SDK infers execution order from those data dependencies.

@dsl.pipeline(name="iris-training")
def iris_pipeline():
    prep = process_data()
    train = train_model(dataset=prep.outputs["output_data"])
    evaluate()

    # An explicit ordering edge with no data dependency:
    evaluate().after(train)

Passing prep.outputs["output_data"] into train_model creates the edge that forces process_data to finish first. When two steps share no data but still need ordering, .after() adds an explicit dependency. Control flow — conditionals, loops, and exit handlers — is expressed with dsl.If, dsl.ParallelFor, and dsl.ExitHandler.

Compile to IR YAML, Then Run

Here is the pattern that makes Kubeflow Pipelines reproducible. You never ship Python to the cluster directly. You compile the pipeline to an intermediate-representation YAML file — a portable, backend-agnostic artifact — and submit that.

from kfp import compiler

compiler.Compiler().compile(iris_pipeline, "pipeline.yaml")

The resulting pipeline.yaml is what you version in Git, review in a pull request, and hand to any KFP-conformant backend. Submitting a run uses the SDK Client, pointed at your Pipelines endpoint, with arguments supplied as a plain dictionary.

from kfp.client import Client

client = Client(host="${KFP_ENDPOINT}")
run = client.create_run_from_pipeline_package(
    "pipeline.yaml",
    arguments={},
)

Hyperparameter Tuning with Katib

Training once gives you one set of hyperparameters. Finding good ones is a search problem, and that is Katib. It runs many trials — each a training job with different parameters — and optimizes an objective metric using algorithms like random search, grid, or Bayesian optimization.

Katib is driven by an Experiment custom resource, but the Python SDK expresses the same thing more compactly. You define an objective function, a search space, and a trial budget, and Katib fans the trials out across the cluster as pods.

from kubeflow.katib import KatibClient, search

def objective(params):
    a, b = params["a"], params["b"]
    result = 4 * a - b            # bigger a, smaller b -> higher score
    print(f"result={result}")     # Katib parses this metric from stdout

KatibClient().tune(
    name="tune-experiment",
    objective=objective,
    parameters={
        "a": search.int(min=10, max=20),
        "b": search.double(min=0.1, max=0.2),
    },
    objective_metric_name="result",
    max_trial_count=12,
    resources_per_trial={"cpu": "2"},
)

The objective writes its metric to stdout in a name=value format that Katib’s metrics collector parses. Each trial becomes a pod; max_trial_count bounds the search and resources_per_trial caps what each one may consume. This is also where feature reuse pays off — a Feast feature store gives every trial the same consistent inputs so you are tuning the model, not chasing data drift between runs.

Serving the Model with KServe

A trained model is worthless until something can call it. KServe deploys models as an InferenceService custom resource — you declare the model format and a storage location, and KServe selects a serving runtime, provisions the pods, and exposes a prediction endpoint.

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
      resources:
        requests:
          cpu: "100m"
          memory: "512Mi"

Because the modelFormat is sklearn with autoselection, KServe matches it to the built-in scikit-learn ClusterServingRuntime — no container to build. Apply it with kubectl apply -f, watch it come up with kubectl get inferenceservice, then send it JSON.

# Deploy and check status
kubectl apply -f sklearn-iris.yaml
kubectl get inferenceservice sklearn-iris

# Query the prediction endpoint (V1 protocol)
curl -H "Host: sklearn-iris.default.example.com" \
  http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict \
  -d '{"instances": [[6.8, 2.8, 4.8, 1.4]]}'

KServe covers classic predictive models well, but LLM serving has its own memory and batching concerns. When the model is a large language model rather than a scikit-learn classifier, a dedicated inference engine like vLLM handles the KV-cache and continuous-batching problems that a generic runtime does not.

Production Checklist

• Compile in CI, not by hand. Treat pipeline.yaml as a build artifact — regenerate and validate it on every commit so the source and the deployed graph never diverge.
• Pin base images and packages. A component that installs latest is a time bomb. Pin versions and prefer prebuilt images for heavy dependencies.
• Right-size trial resources. Set resources_per_trial in Katib and requests/limits on serving pods so a search or a traffic spike cannot starve the cluster.
• Track lineage through artifacts. Write meaningful .metadataon every Model and Dataset so you can answer “which data trained this model?” months later.
• Plan GPU capacity. Distributed training and inference need nodes that scale; autoscaling with something like Karpenter keeps GPU cost bounded.

Kubeflow’s promise is that the same cluster running your services can run your ML lifecycle end to end — authoring, tuning, and serving — as reviewable, reproducible custom resources rather than notebooks and cron jobs. The cost is operational surface area: you are running a platform, not a library. Compile everything, pin everything, and the platform scales a long way.

Kubeflow is one layer of a full MLOps stack. It sits above data and features from a feature store, complements or replaces an MLflow tracking pipeline, and hands off to a serving layer that, for large models, means a purpose-built engine like vLLM.