Ingress-Nginx 使用 Opentelemetry

[TOC]

一、概要

• Ingress Nginx 使用 OpenTelemetry
• Ingress Nginx 版本 1.21.6

二、编译 otel_ngx_module

• 注意， 不同 otel_ngx_module.so 与 Nginx 版本需要匹配， 不能混用。 否则可能出现以下错误。

2.1、创建 Dockerfile 文件

cat <<'EOF'> Dockerfile
ARG IMAGE=nginx:1.21.6

## builder
FROM ${IMAGE} AS builder
WORKDIR /app

RUN sed -i s@/deb.debian.org/@/mirrors.aliyun.com/@g /etc/apt/sources.list \
  && sed -i s@/security.debian.org/@/mirrors.aliyun.com/@g /etc/apt/sources.list \
  && apt-get update \
  && DEBIAN_FRONTEND=noninteractive TZ="Europe/London" \
  && apt-get install --no-install-recommends --no-install-suggests -y \
      make cmake \
      build-essential autoconf libtool pkg-config  \
      curl apt-transport-https ca-certificates \
      software-properties-common \
      gcc g++ git lsb-release \
      gnupg2  \
      libcurl4-openssl-dev libpcre3-dev zlib1g-dev

RUN git clone --shallow-submodules --depth 1 --recurse-submodules -b v1.49.2 \
  https://github.com/grpc/grpc \
  && cd grpc \
  && mkdir -p cmake/build \
  && cd cmake/build \
  && cmake \
    -DgRPC_INSTALL=ON \
    -DgRPC_BUILD_TESTS=OFF \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_BUILD_TYPE=Release \
    -DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF \
    -DCMAKE_CXX_STANDARD=17 \
    ../.. \
  && make -j2 \
  && make install

RUN git clone --shallow-submodules --depth 1 --recurse-submodules -b v1.8.1 \
  https://github.com/open-telemetry/opentelemetry-cpp.git \
  && cd opentelemetry-cpp \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_PREFIX_PATH=/install \
    -DWITH_OTLP=ON \
    -DWITH_OTLP_GRPC=ON \
    -DWITH_OTLP_HTTP=OFF \
    -DBUILD_TESTING=OFF \
    -DWITH_EXAMPLES=OFF \
    -DCMAKE_CXX_STANDARD=17 \
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
    .. \
  && make -j2 \
  && make install

RUN git clone https://github.com/open-telemetry/opentelemetry-cpp-contrib.git \
  && cd opentelemetry-cpp-contrib/instrumentation/nginx \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DNGINX_BIN=/usr/sbin/nginx \
    -DCMAKE_PREFIX_PATH=/install \
    -DCMAKE_INSTALL_PREFIX=/usr/lib/nginx/modules/ \
    .. \
  && make -j2 \
  && make install

FROM bitnami/nginx-ingress-controller:1.8.1-debian-11-r24
COPY --from=builder /usr/lib/nginx/modules/otel_ngx_module.so /modules_mount/etc/nginx/modules/otel/otel_ngx_module.so
EOF

2.2、编译镜像

docker build -t nginx-ingress-controller:1.8.1-debian-11-r24 .

三、启用 opentelemetry

• 官方参考: https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/
• 在 Nginx Ingress Controller 中， 支持的第三方 OpenTelemetry 插件
• 虽然在模版配置文件 /etc/nginx/template/nginx.tmpl 是支持了， 但是 otel_ngx_module.so 需要额外挂载

{{ if (shouldLoadOpentelemetryModule $cfg $servers) }}
load_module /modules_mount/etc/nginx/modules/otel/otel_ngx_module.so;
{{ end }}

3.1、启用 opentelemetry

• 需要编辑 configmap 配置文件

---
apiVersion: v1
data:
  enable-opentelemetry: 'true'
  opentelemetry-config: /etc/nginx/opentelemetry.toml
  opentelemetry-operation-name: HTTP $request_method $service_name $uri $request_body
  opentelemetry-trust-incoming-span: 'true'
  otel-max-export-batch-size: '512'
  otel-max-queuesize: '2048'
  otel-sampler: AlwaysOn
  otel-sampler-parent-based: 'false'
  otel-sampler-ratio: '1.0'
  otel-schedule-delay-millis: '5000'
  otel-service-name: ingress-nginx
  otlp-collector-host: 10.199.250.141
  otlp-collector-port: '4317'
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: nginx-ingress
    meta.helm.sh/release-namespace: kube-system
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx-ingress-controller
    helm.sh/chart: nginx-ingress-controller-9.7.7
  name: nginx-ingress-nginx-ingress-controller
  namespace: kube-system