2025-06-06T00:20:12.6389166Z Current runner version: '2.325.0'
2025-06-06T00:20:12.6410864Z ##[group]Runner Image Provisioner
2025-06-06T00:20:12.6411983Z Hosted Compute Agent
2025-06-06T00:20:12.6412627Z Version: 20250508.323
2025-06-06T00:20:12.6413247Z Commit: 81b259f29879f73b4213d199e42d8c3465dae986
2025-06-06T00:20:12.6413895Z Build Date: 2025-05-08T19:40:08Z
2025-06-06T00:20:12.6414526Z ##[endgroup]
2025-06-06T00:20:12.6415100Z ##[group]Operating System
2025-06-06T00:20:12.6415678Z Ubuntu
2025-06-06T00:20:12.6416197Z 24.04.2
2025-06-06T00:20:12.6416637Z LTS
2025-06-06T00:20:12.6417103Z ##[endgroup]
2025-06-06T00:20:12.6417688Z ##[group]Runner Image
2025-06-06T00:20:12.6418250Z Image: ubuntu-24.04
2025-06-06T00:20:12.6418762Z Version: 20250511.1.0
2025-06-06T00:20:12.6419805Z Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250511.1/images/ubuntu/Ubuntu2404-Readme.md
2025-06-06T00:20:12.6421518Z Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250511.1
2025-06-06T00:20:12.6422667Z ##[endgroup]
2025-06-06T00:20:12.6424837Z ##[group]GITHUB_TOKEN Permissions
2025-06-06T00:20:12.6426763Z Actions: read
2025-06-06T00:20:12.6427333Z Attestations: read
2025-06-06T00:20:12.6427967Z Checks: read
2025-06-06T00:20:12.6428500Z Contents: read
2025-06-06T00:20:12.6428978Z Deployments: read
2025-06-06T00:20:12.6429538Z Discussions: read
2025-06-06T00:20:12.6430065Z Issues: read
2025-06-06T00:20:12.6430564Z Metadata: read
2025-06-06T00:20:12.6431305Z Models: read
2025-06-06T00:20:12.6431796Z Packages: read
2025-06-06T00:20:12.6432327Z Pages: read
2025-06-06T00:20:12.6432892Z PullRequests: read
2025-06-06T00:20:12.6433440Z RepositoryProjects: read
2025-06-06T00:20:12.6434029Z SecurityEvents: read
2025-06-06T00:20:12.6434719Z Statuses: read
2025-06-06T00:20:12.6435192Z ##[endgroup]
2025-06-06T00:20:12.6437881Z Secret source: None
2025-06-06T00:20:12.6438695Z Prepare workflow directory
2025-06-06T00:20:12.6833118Z Prepare all required actions
2025-06-06T00:20:12.6869189Z Getting action download info
2025-06-06T00:20:12.9049520Z Download action repository 'actions/setup-go@d35c59abb061a4a6fb18e82ac0862c26744d6ab5' (SHA:d35c59abb061a4a6fb18e82ac0862c26744d6ab5)
2025-06-06T00:20:13.4784729Z Download action repository 'actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683' (SHA:11bd71901bbe5b1630ceea73d27597364c9af683)
2025-06-06T00:20:13.5107566Z Download action repository 'actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093)
2025-06-06T00:20:14.0896647Z Complete job name: integration (integration_ruler)
2025-06-06T00:20:14.1724092Z ##[group]Run actions/setup-go@d35c59abb061a4a6fb18e82ac0862c26744d6ab5
2025-06-06T00:20:14.1725861Z with:
2025-06-06T00:20:14.1726653Z   go-version: 1.23.2
2025-06-06T00:20:14.1727547Z   check-latest: false
2025-06-06T00:20:14.1728702Z   token: ***
2025-06-06T00:20:14.1729507Z   cache: true
2025-06-06T00:20:14.1730525Z ##[endgroup]
2025-06-06T00:20:14.3374823Z Setup go version spec 1.23.2
2025-06-06T00:20:14.3400702Z Attempting to download 1.23.2...
2025-06-06T00:20:14.8932874Z matching 1.23.2...
2025-06-06T00:20:14.8976509Z Acquiring 1.23.2 from https://github.com/actions/go-versions/releases/download/1.23.2-11145922912/go-1.23.2-linux-x64.tar.gz
2025-06-06T00:20:15.9115589Z Extracting Go...
2025-06-06T00:20:15.9246801Z [command]/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/work/_temp/8d7e755f-6f08-4109-8bec-3841fc940f30 -f /home/runner/work/_temp/b532ea69-2db9-47f9-93aa-c7698e32aed7
2025-06-06T00:20:17.6414266Z Successfully extracted go to /home/runner/work/_temp/8d7e755f-6f08-4109-8bec-3841fc940f30
2025-06-06T00:20:17.6415178Z Adding to the cache ...
2025-06-06T00:20:21.7897706Z Successfully cached go to /opt/hostedtoolcache/go/1.23.2/x64
2025-06-06T00:20:21.7899804Z Added go to the path
2025-06-06T00:20:21.7903118Z Successfully set up Go version 1.23.2
2025-06-06T00:20:21.8096417Z [command]/opt/hostedtoolcache/go/1.23.2/x64/bin/go env GOMODCACHE
2025-06-06T00:20:21.8125084Z [command]/opt/hostedtoolcache/go/1.23.2/x64/bin/go env GOCACHE
2025-06-06T00:20:21.8154922Z /home/runner/go/pkg/mod
2025-06-06T00:20:21.8167914Z /home/runner/.cache/go-build
2025-06-06T00:20:21.8270269Z ##[warning]Restore cache failed: Dependencies file is not found in /home/runner/work/cortex/cortex. Supported file pattern: go.sum
2025-06-06T00:20:21.8361301Z go version go1.23.2 linux/amd64
2025-06-06T00:20:21.8361696Z 
2025-06-06T00:20:21.8362804Z ##[group]go env
2025-06-06T00:20:21.9164965Z GO111MODULE=''
2025-06-06T00:20:21.9165587Z GOARCH='amd64'
2025-06-06T00:20:21.9166106Z GOBIN=''
2025-06-06T00:20:21.9166530Z GOCACHE='/home/runner/.cache/go-build'
2025-06-06T00:20:21.9167070Z GOENV='/home/runner/.config/go/env'
2025-06-06T00:20:21.9167528Z GOEXE=''
2025-06-06T00:20:21.9167880Z GOEXPERIMENT=''
2025-06-06T00:20:21.9168258Z GOFLAGS=''
2025-06-06T00:20:21.9168634Z GOHOSTARCH='amd64'
2025-06-06T00:20:21.9169019Z GOHOSTOS='linux'
2025-06-06T00:20:21.9169414Z GOINSECURE=''
2025-06-06T00:20:21.9169878Z GOMODCACHE='/home/runner/go/pkg/mod'
2025-06-06T00:20:21.9170381Z GONOPROXY=''
2025-06-06T00:20:21.9170779Z GONOSUMDB=''
2025-06-06T00:20:21.9171384Z GOOS='linux'
2025-06-06T00:20:21.9171792Z GOPATH='/home/runner/go'
2025-06-06T00:20:21.9172213Z GOPRIVATE=''
2025-06-06T00:20:21.9172814Z GOPROXY='https://proxy.golang.org,direct'
2025-06-06T00:20:21.9173380Z GOROOT='/opt/hostedtoolcache/go/1.23.2/x64'
2025-06-06T00:20:21.9173892Z GOSUMDB='sum.golang.org'
2025-06-06T00:20:21.9174307Z GOTMPDIR=''
2025-06-06T00:20:21.9174655Z GOTOOLCHAIN='auto'
2025-06-06T00:20:21.9175248Z GOTOOLDIR='/opt/hostedtoolcache/go/1.23.2/x64/pkg/tool/linux_amd64'
2025-06-06T00:20:21.9175860Z GOVCS=''
2025-06-06T00:20:21.9176215Z GOVERSION='go1.23.2'
2025-06-06T00:20:21.9176589Z GODEBUG=''
2025-06-06T00:20:21.9176959Z GOTELEMETRY='local'
2025-06-06T00:20:21.9177431Z GOTELEMETRYDIR='/home/runner/.config/go/telemetry'
2025-06-06T00:20:21.9177992Z GCCGO='gccgo'
2025-06-06T00:20:21.9178353Z GOAMD64='v1'
2025-06-06T00:20:21.9178728Z AR='ar'
2025-06-06T00:20:21.9179065Z CC='gcc'
2025-06-06T00:20:21.9179378Z CXX='g++'
2025-06-06T00:20:21.9179713Z CGO_ENABLED='1'
2025-06-06T00:20:21.9180072Z GOMOD='/dev/null'
2025-06-06T00:20:21.9180419Z GOWORK=''
2025-06-06T00:20:21.9180762Z CGO_CFLAGS='-O2 -g'
2025-06-06T00:20:21.9181298Z CGO_CPPFLAGS=''
2025-06-06T00:20:21.9181669Z CGO_CXXFLAGS='-O2 -g'
2025-06-06T00:20:21.9182047Z CGO_FFLAGS='-O2 -g'
2025-06-06T00:20:21.9182403Z CGO_LDFLAGS='-O2 -g'
2025-06-06T00:20:21.9182789Z PKG_CONFIG='pkg-config'
2025-06-06T00:20:21.9184079Z GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1081115275=/tmp/go-build -gno-record-gcc-switches'
2025-06-06T00:20:21.9184972Z 
2025-06-06T00:20:21.9185622Z ##[endgroup]
2025-06-06T00:20:21.9443999Z ##[group]Run actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
2025-06-06T00:20:21.9444431Z with:
2025-06-06T00:20:21.9444678Z   repository: cortexproject/cortex
2025-06-06T00:20:21.9445112Z   token: ***
2025-06-06T00:20:21.9445350Z   ssh-strict: true
2025-06-06T00:20:21.9445596Z   ssh-user: git
2025-06-06T00:20:21.9445847Z   persist-credentials: true
2025-06-06T00:20:21.9446117Z   clean: true
2025-06-06T00:20:21.9446362Z   sparse-checkout-cone-mode: true
2025-06-06T00:20:21.9446638Z   fetch-depth: 1
2025-06-06T00:20:21.9446876Z   fetch-tags: false
2025-06-06T00:20:21.9447121Z   show-progress: true
2025-06-06T00:20:21.9447367Z   lfs: false
2025-06-06T00:20:21.9447591Z   submodules: false
2025-06-06T00:20:21.9447828Z   set-safe-directory: true
2025-06-06T00:20:21.9448089Z ##[endgroup]
2025-06-06T00:20:22.0365350Z Syncing repository: cortexproject/cortex
2025-06-06T00:20:22.0366382Z ##[group]Getting Git version info
2025-06-06T00:20:22.0367106Z Working directory is '/home/runner/work/cortex/cortex'
2025-06-06T00:20:22.0408243Z [command]/usr/bin/git version
2025-06-06T00:20:22.0485273Z git version 2.49.0
2025-06-06T00:20:22.0511412Z ##[endgroup]
2025-06-06T00:20:22.0523641Z Temporarily overriding HOME='/home/runner/work/_temp/3ea5c9e2-f80b-43df-a0dd-ac62ae05ffca' before making global git config changes
2025-06-06T00:20:22.0525301Z Adding repository directory to the temporary git global config as a safe directory
2025-06-06T00:20:22.0528343Z [command]/usr/bin/git config --global --add safe.directory /home/runner/work/cortex/cortex
2025-06-06T00:20:22.0558960Z Deleting the contents of '/home/runner/work/cortex/cortex'
2025-06-06T00:20:22.0562510Z ##[group]Initializing the repository
2025-06-06T00:20:22.0566192Z [command]/usr/bin/git init /home/runner/work/cortex/cortex
2025-06-06T00:20:22.0622947Z hint: Using 'master' as the name for the initial branch. This default branch name
2025-06-06T00:20:22.0623845Z hint: is subject to change. To configure the initial branch name to use in all
2025-06-06T00:20:22.0624472Z hint: of your new repositories, which will suppress this warning, call:
2025-06-06T00:20:22.0624867Z hint:
2025-06-06T00:20:22.0625182Z hint: 	git config --global init.defaultBranch <name>
2025-06-06T00:20:22.0625523Z hint:
2025-06-06T00:20:22.0625882Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2025-06-06T00:20:22.0626415Z hint: 'development'. The just-created branch can be renamed via this command:
2025-06-06T00:20:22.0627139Z hint:
2025-06-06T00:20:22.0627545Z hint: 	git branch -m <name>
2025-06-06T00:20:22.0628273Z Initialized empty Git repository in /home/runner/work/cortex/cortex/.git/
2025-06-06T00:20:22.0637969Z [command]/usr/bin/git remote add origin https://github.com/cortexproject/cortex
2025-06-06T00:20:22.0669499Z ##[endgroup]
2025-06-06T00:20:22.0670053Z ##[group]Disabling automatic garbage collection
2025-06-06T00:20:22.0673655Z [command]/usr/bin/git config --local gc.auto 0
2025-06-06T00:20:22.0700820Z ##[endgroup]
2025-06-06T00:20:22.0701482Z ##[group]Setting up auth
2025-06-06T00:20:22.0707581Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-06-06T00:20:22.0737829Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-06-06T00:20:22.1006885Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-06-06T00:20:22.1034338Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-06-06T00:20:22.1248797Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-06-06T00:20:22.1291826Z ##[endgroup]
2025-06-06T00:20:22.1292660Z ##[group]Fetching the repository
2025-06-06T00:20:22.1301000Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=1 origin +368026e2d7c0f060165d4092a3ec6a5f44540f82:refs/remotes/pull/6795/merge
2025-06-06T00:20:25.5356397Z From https://github.com/cortexproject/cortex
2025-06-06T00:20:25.5357160Z  * [new ref]         368026e2d7c0f060165d4092a3ec6a5f44540f82 -> pull/6795/merge
2025-06-06T00:20:25.5391495Z ##[endgroup]
2025-06-06T00:20:25.5392132Z ##[group]Determining the checkout info
2025-06-06T00:20:25.5393652Z ##[endgroup]
2025-06-06T00:20:25.5400021Z [command]/usr/bin/git sparse-checkout disable
2025-06-06T00:20:25.5438715Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig
2025-06-06T00:20:25.5465366Z ##[group]Checking out the ref
2025-06-06T00:20:25.5469656Z [command]/usr/bin/git checkout --progress --force refs/remotes/pull/6795/merge
2025-06-06T00:20:26.1864165Z Note: switching to 'refs/remotes/pull/6795/merge'.
2025-06-06T00:20:26.1864595Z 
2025-06-06T00:20:26.1865150Z You are in 'detached HEAD' state. You can look around, make experimental
2025-06-06T00:20:26.1866037Z changes and commit them, and you can discard any commits you make in this
2025-06-06T00:20:26.1866904Z state without impacting any branches by switching back to a branch.
2025-06-06T00:20:26.1867417Z 
2025-06-06T00:20:26.1868057Z If you want to create a new branch to retain commits you create, you may
2025-06-06T00:20:26.1868801Z do so (now or later) by using -c with the switch command. Example:
2025-06-06T00:20:26.1869290Z 
2025-06-06T00:20:26.1869935Z   git switch -c <new-branch-name>
2025-06-06T00:20:26.1870226Z 
2025-06-06T00:20:26.1870500Z Or undo this operation with:
2025-06-06T00:20:26.1870794Z 
2025-06-06T00:20:26.1870972Z   git switch -
2025-06-06T00:20:26.1871384Z 
2025-06-06T00:20:26.1872106Z Turn off this advice by setting config variable advice.detachedHead to false
2025-06-06T00:20:26.1872583Z 
2025-06-06T00:20:26.1873154Z HEAD is now at 368026e Merge 8566d87c441f37a86365e7f4e9d27e4998a3f03d into 844fa558726ca4a621b1265cc20b8f557d552da8
2025-06-06T00:20:26.1895237Z ##[endgroup]
2025-06-06T00:20:26.1937926Z [command]/usr/bin/git log -1 --format=%H
2025-06-06T00:20:26.1960853Z 368026e2d7c0f060165d4092a3ec6a5f44540f82
2025-06-06T00:20:26.2090725Z ##[group]Run sudo ./.github/workflows/scripts/install-docker.sh
2025-06-06T00:20:26.2091512Z sudo ./.github/workflows/scripts/install-docker.sh
2025-06-06T00:20:26.2233423Z shell: /usr/bin/bash -e {0}
2025-06-06T00:20:26.2233697Z ##[endgroup]
2025-06-06T00:20:26.2386777Z + VER=28.0.4
2025-06-06T00:20:26.2387545Z + curl -L -o /tmp/docker-28.0.4.tgz https://download.docker.com/linux/static/stable/x86_64/docker-28.0.4.tgz
2025-06-06T00:20:26.2440042Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2025-06-06T00:20:26.2441755Z                                  Dload  Upload   Total   Spent    Left  Speed
2025-06-06T00:20:26.2442146Z 
2025-06-06T00:20:26.5100429Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2025-06-06T00:20:26.6447202Z  34 75.1M   34 25.7M    0     0  96.6M      0 --:--:-- --:--:-- --:--:-- 96.3M
2025-06-06T00:20:26.6448078Z 100 75.1M  100 75.1M    0     0   187M      0 --:--:-- --:--:-- --:--:--  187M
2025-06-06T00:20:26.6472305Z + tar -xz -C /tmp -f /tmp/docker-28.0.4.tgz
2025-06-06T00:20:27.9969351Z + mkdir -vp /root/.docker/cli-plugins/
2025-06-06T00:20:27.9982356Z mkdir: created directory '/root/.docker'
2025-06-06T00:20:27.9982832Z mkdir: created directory '/root/.docker/cli-plugins/'
2025-06-06T00:20:27.9984687Z + curl --silent -L https://github.com/docker/buildx/releases/download/v0.3.0/buildx-v0.3.0.linux-amd64
2025-06-06T00:20:28.8151798Z + chmod a+x /root/.docker/cli-plugins/docker-buildx
2025-06-06T00:20:28.8164371Z + mv /tmp/docker/containerd /tmp/docker/containerd-shim-runc-v2 /tmp/docker/ctr /tmp/docker/docker /tmp/docker/docker-init /tmp/docker/docker-proxy /tmp/docker/dockerd /tmp/docker/runc /usr/bin
2025-06-06T00:20:28.9475953Z + docker run --privileged --rm tonistiigi/binfmt --install all
2025-06-06T00:20:29.0574330Z Unable to find image 'tonistiigi/binfmt:latest' locally
2025-06-06T00:20:30.0139377Z latest: Pulling from tonistiigi/binfmt
2025-06-06T00:20:30.3100751Z 3a6596812d61: Pulling fs layer
2025-06-06T00:20:30.3101411Z d9d64120404b: Pulling fs layer
2025-06-06T00:20:30.5824922Z d9d64120404b: Verifying Checksum
2025-06-06T00:20:30.5825539Z d9d64120404b: Download complete
2025-06-06T00:20:30.7355323Z 3a6596812d61: Verifying Checksum
2025-06-06T00:20:30.7355803Z 3a6596812d61: Download complete
2025-06-06T00:20:31.0729407Z 3a6596812d61: Pull complete
2025-06-06T00:20:32.8448190Z d9d64120404b: Pull complete
2025-06-06T00:20:32.8486361Z Digest: sha256:1b804311fe87047a4c96d38b4b3ef6f62fca8cd125265917a9e3dc3c996c39e6
2025-06-06T00:20:32.8499486Z Status: Downloaded newer image for tonistiigi/binfmt:latest
2025-06-06T00:20:32.9852544Z installing: arm64 OK
2025-06-06T00:20:32.9853104Z installing: arm OK
2025-06-06T00:20:32.9853360Z installing: s390x OK
2025-06-06T00:20:32.9853655Z installing: ppc64le OK
2025-06-06T00:20:32.9854004Z installing: riscv64 OK
2025-06-06T00:20:32.9854225Z installing: mips64le OK
2025-06-06T00:20:32.9854441Z installing: mips64 OK
2025-06-06T00:20:32.9854696Z installing: loong64 OK
2025-06-06T00:20:33.0084591Z {
2025-06-06T00:20:33.0085030Z   "supported": [
2025-06-06T00:20:33.0085971Z     "linux/amd64",
2025-06-06T00:20:33.0086418Z     "linux/amd64/v2",
2025-06-06T00:20:33.0086784Z     "linux/amd64/v3",
2025-06-06T00:20:33.0087132Z     "linux/arm64",
2025-06-06T00:20:33.0087468Z     "linux/riscv64",
2025-06-06T00:20:33.0087823Z     "linux/ppc64le",
2025-06-06T00:20:33.0088217Z     "linux/s390x",
2025-06-06T00:20:33.0088564Z     "linux/386",
2025-06-06T00:20:33.0088921Z     "linux/mips64le",
2025-06-06T00:20:33.0089296Z     "linux/mips64",
2025-06-06T00:20:33.0089658Z     "linux/loong64",
2025-06-06T00:20:33.0090013Z     "linux/arm/v7",
2025-06-06T00:20:33.0090351Z     "linux/arm/v6"
2025-06-06T00:20:33.0090647Z   ],
2025-06-06T00:20:33.0090945Z   "emulators": [
2025-06-06T00:20:33.0091563Z     "llvm-16-runtime.binfmt",
2025-06-06T00:20:33.0092007Z     "llvm-17-runtime.binfmt",
2025-06-06T00:20:33.0092443Z     "llvm-18-runtime.binfmt",
2025-06-06T00:20:33.0093269Z     "python3.12",
2025-06-06T00:20:33.0095838Z     "qemu-aarch64",
2025-06-06T00:20:33.0096221Z     "qemu-arm",
2025-06-06T00:20:33.0096623Z     "qemu-loongarch64",
2025-06-06T00:20:33.0097009Z     "qemu-mips64",
2025-06-06T00:20:33.0097386Z     "qemu-mips64el",
2025-06-06T00:20:33.0097774Z     "qemu-ppc64le",
2025-06-06T00:20:33.0098153Z     "qemu-riscv64",
2025-06-06T00:20:33.0098503Z     "qemu-s390x"
2025-06-06T00:20:33.0098819Z   ]
2025-06-06T00:20:33.0099108Z }
2025-06-06T00:20:33.0716013Z ##[group]Run sudo mkdir -p /go/src/github.com/cortexproject/cortex
2025-06-06T00:20:33.0716515Z sudo mkdir -p /go/src/github.com/cortexproject/cortex
2025-06-06T00:20:33.0716960Z sudo ln -s $GITHUB_WORKSPACE/* /go/src/github.com/cortexproject/cortex
2025-06-06T00:20:33.0767777Z shell: /usr/bin/bash -e {0}
2025-06-06T00:20:33.0768033Z ##[endgroup]
2025-06-06T00:20:33.1026714Z ##[group]Run actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093
2025-06-06T00:20:33.1027104Z with:
2025-06-06T00:20:33.1027293Z   name: Docker Images
2025-06-06T00:20:33.1027513Z   merge-multiple: false
2025-06-06T00:20:33.1027766Z   repository: cortexproject/cortex
2025-06-06T00:20:33.1028014Z   run-id: 15479868252
2025-06-06T00:20:33.1028214Z ##[endgroup]
2025-06-06T00:20:33.2936859Z Downloading single artifact
2025-06-06T00:20:33.5700343Z Preparing to download the following artifacts:
2025-06-06T00:20:33.5734203Z - Docker Images (ID: 3272208666, Size: 134664884, Expected Digest: sha256:3ad2cc26d598cd169ecb1e0231bbb5a18915ec3615317dc66ce2427ec0313a0b)
2025-06-06T00:20:33.7569397Z Redirecting to blob download url: https://productionresultssa15.blob.core.windows.net/actions-results/083446ea-bcf0-4d1e-965e-5687c068ff3c/workflow-job-run-ca395085-040a-526b-2ce8-bdc85f692774/artifacts/ec15abe1a6768860f1f1dd0ccf85d1eb16e773ce729b1905d66488829e85b4b2.zip
2025-06-06T00:20:33.7571377Z Starting download of artifact to: /home/runner/work/cortex/cortex
2025-06-06T00:20:34.0527979Z (node:2305) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
2025-06-06T00:20:34.0528884Z (Use `node --trace-deprecation ...` to show where the warning was created)
2025-06-06T00:20:39.1941531Z SHA256 digest of downloaded artifact is 3ad2cc26d598cd169ecb1e0231bbb5a18915ec3615317dc66ce2427ec0313a0b
2025-06-06T00:20:39.1942253Z Artifact download completed successfully.
2025-06-06T00:20:39.1942580Z Total of 1 artifact(s) downloaded
2025-06-06T00:20:39.1951706Z Download artifact has finished successfully
2025-06-06T00:20:39.2067843Z ##[group]Run tar -xvf images.tar -C /
2025-06-06T00:20:39.2068194Z tar -xvf images.tar -C /
2025-06-06T00:20:39.2118286Z shell: /usr/bin/bash -e {0}
2025-06-06T00:20:39.2118547Z ##[endgroup]
2025-06-06T00:20:39.2199412Z tmp/images/
2025-06-06T00:20:39.2199983Z tmp/images/quay.io_cortexproject_thanosconvert:HEAD-368026e-amd64
2025-06-06T00:20:39.2615391Z tmp/images/quay.io_cortexproject_query-tee:HEAD-368026e-arm64
2025-06-06T00:20:39.2810646Z tmp/images/quay.io_cortexproject_thanosconvert:HEAD-368026e-arm64
2025-06-06T00:20:39.3253403Z tmp/images/quay.io_cortexproject_query-tee:HEAD-368026e-amd64
2025-06-06T00:20:39.3452572Z tmp/images/quay.io_cortexproject_test-exporter:HEAD-368026e-arm64
2025-06-06T00:20:39.3676076Z tmp/images/quay.io_cortexproject_cortex:HEAD-368026e-amd64
2025-06-06T00:20:39.4484205Z tmp/images/quay.io_cortexproject_test-exporter:HEAD-368026e-amd64
2025-06-06T00:20:39.4709823Z tmp/images/quay.io_cortexproject_cortex:HEAD-368026e-arm64
2025-06-06T00:20:39.5471716Z ##[group]Run ln -s /tmp/images ./docker-images
2025-06-06T00:20:39.5472120Z ln -s /tmp/images ./docker-images
2025-06-06T00:20:39.5472453Z make BUILD_IN_CONTAINER=false load-images
2025-06-06T00:20:39.5522174Z shell: /usr/bin/bash -e {0}
2025-06-06T00:20:39.5522437Z ##[endgroup]
2025-06-06T00:20:39.7660856Z grep: warning: ? at start of expression
2025-06-06T00:20:39.9377388Z quay.io/cortexproject/build-image
2025-06-06T00:20:42.4696411Z Loaded image: quay.io/cortexproject/cortex:HEAD-368026e-amd64
2025-06-06T00:20:43.0657371Z Loaded image: quay.io/cortexproject/cortex:HEAD-368026e-arm64
2025-06-06T00:20:43.1851481Z Loaded image: quay.io/cortexproject/query-tee:HEAD-368026e-amd64
2025-06-06T00:20:43.2970126Z Loaded image: quay.io/cortexproject/query-tee:HEAD-368026e-arm64
2025-06-06T00:20:43.4127427Z Loaded image: quay.io/cortexproject/test-exporter:HEAD-368026e-amd64
2025-06-06T00:20:43.5366947Z Loaded image: quay.io/cortexproject/test-exporter:HEAD-368026e-arm64
2025-06-06T00:20:43.7597935Z Loaded image: quay.io/cortexproject/thanosconvert:HEAD-368026e-amd64
2025-06-06T00:20:43.9664146Z Loaded image: quay.io/cortexproject/thanosconvert:HEAD-368026e-arm64
2025-06-06T00:20:43.9796895Z ##[group]Run docker pull minio/minio:RELEASE.2024-05-28T17-19-04Z
2025-06-06T00:20:43.9797402Z docker pull minio/minio:RELEASE.2024-05-28T17-19-04Z
2025-06-06T00:20:43.9797742Z docker pull consul:1.8.4
2025-06-06T00:20:43.9798060Z docker pull gcr.io/etcd-development/etcd:v3.4.7
2025-06-06T00:20:43.9798471Z if [ "$TEST_TAGS" = "integration_backward_compatibility" ]; then
2025-06-06T00:20:43.9798877Z   docker pull quay.io/cortexproject/cortex:v1.13.1
2025-06-06T00:20:43.9799238Z   docker pull quay.io/cortexproject/cortex:v1.13.2
2025-06-06T00:20:43.9799596Z   docker pull quay.io/cortexproject/cortex:v1.14.0
2025-06-06T00:20:43.9799946Z   docker pull quay.io/cortexproject/cortex:v1.14.1
2025-06-06T00:20:43.9800302Z   docker pull quay.io/cortexproject/cortex:v1.15.0
2025-06-06T00:20:43.9800660Z   docker pull quay.io/cortexproject/cortex:v1.15.1
2025-06-06T00:20:43.9801037Z   docker pull quay.io/cortexproject/cortex:v1.15.2
2025-06-06T00:20:43.9801741Z   docker pull quay.io/cortexproject/cortex:v1.15.3
2025-06-06T00:20:43.9802120Z   docker pull quay.io/cortexproject/cortex:v1.16.0
2025-06-06T00:20:43.9802478Z   docker pull quay.io/cortexproject/cortex:v1.16.1
2025-06-06T00:20:43.9802833Z   docker pull quay.io/cortexproject/cortex:v1.17.0
2025-06-06T00:20:43.9803178Z   docker pull quay.io/cortexproject/cortex:v1.17.1
2025-06-06T00:20:43.9803527Z   docker pull quay.io/cortexproject/cortex:v1.18.0
2025-06-06T00:20:43.9803869Z   docker pull quay.io/cortexproject/cortex:v1.18.1
2025-06-06T00:20:43.9804227Z elif [ "$TEST_TAGS" = "integration_query_fuzz" ]; then
2025-06-06T00:20:43.9804593Z   docker pull quay.io/cortexproject/cortex:v1.18.1 
2025-06-06T00:20:43.9804953Z   docker pull quay.io/prometheus/prometheus:v2.51.0
2025-06-06T00:20:43.9805311Z   docker pull quay.io/prometheus/prometheus:v2.55.1
2025-06-06T00:20:43.9805616Z fi
2025-06-06T00:20:43.9805839Z docker pull memcached:1.6.1
2025-06-06T00:20:43.9806119Z docker pull redis:7.0.4-alpine
2025-06-06T00:20:43.9856087Z shell: /usr/bin/bash -e {0}
2025-06-06T00:20:43.9856339Z env:
2025-06-06T00:20:43.9856548Z   TEST_TAGS: integration_ruler
2025-06-06T00:20:43.9856979Z ##[endgroup]
2025-06-06T00:20:44.9594631Z RELEASE.2024-05-28T17-19-04Z: Pulling from minio/minio
2025-06-06T00:20:45.2159923Z 969cd2eb857f: Pulling fs layer
2025-06-06T00:20:45.2160439Z 91540efb70dc: Pulling fs layer
2025-06-06T00:20:45.2160927Z 7f90d5ce4604: Pulling fs layer
2025-06-06T00:20:45.2161590Z 16d8c7dde19e: Pulling fs layer
2025-06-06T00:20:45.2162061Z a48bc39d2f9c: Pulling fs layer
2025-06-06T00:20:45.2162528Z d34e36fb9d9b: Pulling fs layer
2025-06-06T00:20:45.2163054Z 91d55090a833: Pulling fs layer
2025-06-06T00:20:45.2163539Z 26aa18ab7212: Pulling fs layer
2025-06-06T00:20:45.2163916Z 16d8c7dde19e: Waiting
2025-06-06T00:20:45.2164332Z a48bc39d2f9c: Waiting
2025-06-06T00:20:45.2164733Z d34e36fb9d9b: Waiting
2025-06-06T00:20:45.2165065Z 91d55090a833: Waiting
2025-06-06T00:20:45.2165325Z 26aa18ab7212: Waiting
2025-06-06T00:20:45.4753961Z 91540efb70dc: Verifying Checksum
2025-06-06T00:20:45.4755885Z 91540efb70dc: Download complete
2025-06-06T00:20:45.5062736Z 969cd2eb857f: Verifying Checksum
2025-06-06T00:20:45.5063248Z 969cd2eb857f: Download complete
2025-06-06T00:20:45.7314199Z 7f90d5ce4604: Verifying Checksum
2025-06-06T00:20:45.7314799Z 7f90d5ce4604: Download complete
2025-06-06T00:20:45.7737250Z 969cd2eb857f: Pull complete
2025-06-06T00:20:45.7748117Z a48bc39d2f9c: Verifying Checksum
2025-06-06T00:20:45.7748621Z a48bc39d2f9c: Download complete
2025-06-06T00:20:45.8124112Z 91540efb70dc: Pull complete
2025-06-06T00:20:45.8353023Z 16d8c7dde19e: Verifying Checksum
2025-06-06T00:20:45.8353980Z 16d8c7dde19e: Download complete
2025-06-06T00:20:45.9974020Z d34e36fb9d9b: Verifying Checksum
2025-06-06T00:20:45.9974880Z d34e36fb9d9b: Download complete
2025-06-06T00:20:46.0286825Z 91d55090a833: Download complete
2025-06-06T00:20:46.0902097Z 26aa18ab7212: Download complete
2025-06-06T00:20:46.2568757Z 7f90d5ce4604: Pull complete
2025-06-06T00:20:46.3882863Z 16d8c7dde19e: Pull complete
2025-06-06T00:20:46.4238189Z a48bc39d2f9c: Pull complete
2025-06-06T00:20:46.4380611Z d34e36fb9d9b: Pull complete
2025-06-06T00:20:46.4468795Z 91d55090a833: Pull complete
2025-06-06T00:20:46.4547426Z 26aa18ab7212: Pull complete
2025-06-06T00:20:46.4579917Z Digest: sha256:391d1d45fdbe79944cb6de9337b073864bb9ee38c4c24280bfb39572e925af08
2025-06-06T00:20:46.4590939Z Status: Downloaded newer image for minio/minio:RELEASE.2024-05-28T17-19-04Z
2025-06-06T00:20:46.4599028Z docker.io/minio/minio:RELEASE.2024-05-28T17-19-04Z
2025-06-06T00:20:47.4362261Z 1.8.4: Pulling from library/consul
2025-06-06T00:20:47.6872298Z 188c0c94c7c5: Pulling fs layer
2025-06-06T00:20:47.6872816Z f79a37807a5b: Pulling fs layer
2025-06-06T00:20:47.6873278Z 5cf6ae53e9ed: Pulling fs layer
2025-06-06T00:20:47.6873710Z 36d6ebfedec3: Pulling fs layer
2025-06-06T00:20:47.6874129Z d747248c38e9: Pulling fs layer
2025-06-06T00:20:47.6874535Z c98d87c85712: Pulling fs layer
2025-06-06T00:20:47.6874927Z d747248c38e9: Waiting
2025-06-06T00:20:47.6875277Z c98d87c85712: Waiting
2025-06-06T00:20:47.6875627Z 36d6ebfedec3: Waiting
2025-06-06T00:20:47.9776923Z 188c0c94c7c5: Verifying Checksum
2025-06-06T00:20:47.9979339Z f79a37807a5b: Verifying Checksum
2025-06-06T00:20:47.9980081Z f79a37807a5b: Download complete
2025-06-06T00:20:48.0819706Z 188c0c94c7c5: Pull complete
2025-06-06T00:20:48.1101403Z f79a37807a5b: Pull complete
2025-06-06T00:20:48.2373934Z d747248c38e9: Download complete
2025-06-06T00:20:48.2417034Z 36d6ebfedec3: Verifying Checksum
2025-06-06T00:20:48.2417517Z 36d6ebfedec3: Download complete
2025-06-06T00:20:48.2507063Z 5cf6ae53e9ed: Verifying Checksum
2025-06-06T00:20:48.2507560Z 5cf6ae53e9ed: Download complete
2025-06-06T00:20:48.4842137Z c98d87c85712: Verifying Checksum
2025-06-06T00:20:48.4842726Z c98d87c85712: Download complete
2025-06-06T00:20:48.8009197Z 5cf6ae53e9ed: Pull complete
2025-06-06T00:20:48.8099663Z 36d6ebfedec3: Pull complete
2025-06-06T00:20:48.8194777Z d747248c38e9: Pull complete
2025-06-06T00:20:48.8282796Z c98d87c85712: Pull complete
2025-06-06T00:20:48.8355486Z Digest: sha256:4cc02f91a918f08655b39c4369b65929013525cc020f01dadee6b0ec4cd972f5
2025-06-06T00:20:48.8365519Z Status: Downloaded newer image for consul:1.8.4
2025-06-06T00:20:48.8373135Z docker.io/library/consul:1.8.4
2025-06-06T00:20:49.9924311Z v3.4.7: Pulling from etcd-development/etcd
2025-06-06T00:20:49.9924884Z 39fafc05754f: Pulling fs layer
2025-06-06T00:20:49.9925290Z 73bddc6689cc: Pulling fs layer
2025-06-06T00:20:49.9925710Z af998036f76b: Pulling fs layer
2025-06-06T00:20:49.9926156Z 0ddf7ea68954: Pulling fs layer
2025-06-06T00:20:49.9926599Z 52c0ffa8c335: Pulling fs layer
2025-06-06T00:20:49.9927033Z e36d0fda6376: Pulling fs layer
2025-06-06T00:20:49.9927431Z 0ddf7ea68954: Waiting
2025-06-06T00:20:49.9927686Z 52c0ffa8c335: Waiting
2025-06-06T00:20:49.9927909Z e36d0fda6376: Waiting
2025-06-06T00:20:50.4630751Z af998036f76b: Verifying Checksum
2025-06-06T00:20:50.4631498Z af998036f76b: Download complete
2025-06-06T00:20:50.5881011Z 73bddc6689cc: Verifying Checksum
2025-06-06T00:20:50.5882282Z 73bddc6689cc: Download complete
2025-06-06T00:20:50.6104806Z 39fafc05754f: Verifying Checksum
2025-06-06T00:20:50.6105329Z 39fafc05754f: Download complete
2025-06-06T00:20:50.9087070Z 52c0ffa8c335: Verifying Checksum
2025-06-06T00:20:50.9087617Z 52c0ffa8c335: Download complete
2025-06-06T00:20:50.9815068Z 0ddf7ea68954: Verifying Checksum
2025-06-06T00:20:50.9815501Z 0ddf7ea68954: Download complete
2025-06-06T00:20:51.1561737Z e36d0fda6376: Download complete
2025-06-06T00:20:51.1626331Z 39fafc05754f: Pull complete
2025-06-06T00:20:51.3169100Z 73bddc6689cc: Pull complete
2025-06-06T00:20:51.4074761Z af998036f76b: Pull complete
2025-06-06T00:20:51.4163784Z 0ddf7ea68954: Pull complete
2025-06-06T00:20:51.4250204Z 52c0ffa8c335: Pull complete
2025-06-06T00:20:51.4340669Z e36d0fda6376: Pull complete
2025-06-06T00:20:51.4367743Z Digest: sha256:c25955f76fa45e54c1362cd962dec70cd8836afc2659864a4302b26bb18492eb
2025-06-06T00:20:51.4378309Z Status: Downloaded newer image for gcr.io/etcd-development/etcd:v3.4.7
2025-06-06T00:20:51.4385155Z gcr.io/etcd-development/etcd:v3.4.7
2025-06-06T00:20:52.4400135Z 1.6.1: Pulling from library/memcached
2025-06-06T00:20:52.6804987Z 68ced04f60ab: Pulling fs layer
2025-06-06T00:20:52.6805508Z 7233c6ae94c5: Pulling fs layer
2025-06-06T00:20:52.6805825Z 3945a4bf251b: Pulling fs layer
2025-06-06T00:20:52.6806141Z a9e46f4b34ec: Pulling fs layer
2025-06-06T00:20:52.6806611Z ca1999a38cda: Pulling fs layer
2025-06-06T00:20:52.6807006Z d4f797df323b: Pulling fs layer
2025-06-06T00:20:52.6807307Z a9e46f4b34ec: Waiting
2025-06-06T00:20:52.6807720Z ca1999a38cda: Waiting
2025-06-06T00:20:52.6808105Z d4f797df323b: Waiting
2025-06-06T00:20:52.9332232Z 3945a4bf251b: Verifying Checksum
2025-06-06T00:20:52.9332881Z 3945a4bf251b: Download complete
2025-06-06T00:20:52.9359299Z 7233c6ae94c5: Download complete
2025-06-06T00:20:53.1734804Z 68ced04f60ab: Verifying Checksum
2025-06-06T00:20:53.1737661Z 68ced04f60ab: Download complete
2025-06-06T00:20:53.1923720Z ca1999a38cda: Verifying Checksum
2025-06-06T00:20:53.1924766Z ca1999a38cda: Download complete
2025-06-06T00:20:53.2993626Z a9e46f4b34ec: Verifying Checksum
2025-06-06T00:20:53.2995094Z a9e46f4b34ec: Download complete
2025-06-06T00:20:53.4393320Z d4f797df323b: Download complete
2025-06-06T00:20:54.2092037Z 68ced04f60ab: Pull complete
2025-06-06T00:20:54.2899744Z 7233c6ae94c5: Pull complete
2025-06-06T00:20:54.3466964Z 3945a4bf251b: Pull complete
2025-06-06T00:20:54.4043370Z a9e46f4b34ec: Pull complete
2025-06-06T00:20:54.4125068Z ca1999a38cda: Pull complete
2025-06-06T00:20:54.4194944Z d4f797df323b: Pull complete
2025-06-06T00:20:54.4224510Z Digest: sha256:96fa7f208b93c7f62f1febf8b5c38f8fd45230a708207b3e7eda98b4cff93bfc
2025-06-06T00:20:54.4234438Z Status: Downloaded newer image for memcached:1.6.1
2025-06-06T00:20:54.4241262Z docker.io/library/memcached:1.6.1
2025-06-06T00:20:55.4402287Z 7.0.4-alpine: Pulling from library/redis
2025-06-06T00:20:55.6887487Z 213ec9aee27d: Pulling fs layer
2025-06-06T00:20:55.6887962Z c99be1b28c7f: Pulling fs layer
2025-06-06T00:20:55.6888406Z 8ff0bb7e55e3: Pulling fs layer
2025-06-06T00:20:55.6889140Z 6d80de393db7: Pulling fs layer
2025-06-06T00:20:55.6889555Z 8dbffc478db1: Pulling fs layer
2025-06-06T00:20:55.6889961Z 7402bc4c98a0: Pulling fs layer
2025-06-06T00:20:55.6890351Z 8dbffc478db1: Waiting
2025-06-06T00:20:55.6890708Z 7402bc4c98a0: Waiting
2025-06-06T00:20:55.6891273Z 6d80de393db7: Waiting
2025-06-06T00:20:55.9391480Z c99be1b28c7f: Verifying Checksum
2025-06-06T00:20:55.9392134Z c99be1b28c7f: Download complete
2025-06-06T00:20:55.9555242Z 8ff0bb7e55e3: Verifying Checksum
2025-06-06T00:20:55.9555754Z 8ff0bb7e55e3: Download complete
2025-06-06T00:20:55.9589728Z 213ec9aee27d: Verifying Checksum
2025-06-06T00:20:55.9590089Z 213ec9aee27d: Download complete
2025-06-06T00:20:56.0461548Z 213ec9aee27d: Pull complete
2025-06-06T00:20:56.0639975Z c99be1b28c7f: Pull complete
2025-06-06T00:20:56.1967156Z 8dbffc478db1: Verifying Checksum
2025-06-06T00:20:56.1967677Z 8dbffc478db1: Download complete
2025-06-06T00:20:56.2310796Z 7402bc4c98a0: Verifying Checksum
2025-06-06T00:20:56.2312397Z 7402bc4c98a0: Download complete
2025-06-06T00:20:56.2607829Z 6d80de393db7: Verifying Checksum
2025-06-06T00:20:56.2608434Z 6d80de393db7: Download complete
2025-06-06T00:20:56.3208031Z 8ff0bb7e55e3: Pull complete
2025-06-06T00:20:56.4584715Z 6d80de393db7: Pull complete
2025-06-06T00:20:56.4686169Z 8dbffc478db1: Pull complete
2025-06-06T00:20:56.4806360Z 7402bc4c98a0: Pull complete
2025-06-06T00:20:56.4843961Z Digest: sha256:dc1b954f5a1db78e31b8870966294d2f93fa8a7fba5c1337a1ce4ec55f311bc3
2025-06-06T00:20:56.4853336Z Status: Downloaded newer image for redis:7.0.4-alpine
2025-06-06T00:20:56.4861412Z docker.io/library/redis:7.0.4-alpine
2025-06-06T00:20:56.4919375Z ##[group]Run export CORTEX_IMAGE_PREFIX="${IMAGE_PREFIX:-quay.io/cortexproject/}"
2025-06-06T00:20:56.4919958Z export CORTEX_IMAGE_PREFIX="${IMAGE_PREFIX:-quay.io/cortexproject/}"
2025-06-06T00:20:56.4920365Z export IMAGE_TAG=$(make image-tag)
2025-06-06T00:20:56.4920749Z export CORTEX_IMAGE="${CORTEX_IMAGE_PREFIX}cortex:$IMAGE_TAG-amd64"
2025-06-06T00:20:56.4921485Z export CORTEX_CHECKOUT_DIR="/go/src/github.com/cortexproject/cortex"
2025-06-06T00:20:56.4921966Z echo "Running integration tests with image: $CORTEX_IMAGE"
2025-06-06T00:20:56.4922499Z go test -tags=integration,integration_ruler -timeout 2400s -v -count=1 ./integration/...
2025-06-06T00:20:56.4972497Z shell: /usr/bin/bash -e {0}
2025-06-06T00:20:56.4972757Z env:
2025-06-06T00:20:56.4972948Z   IMAGE_PREFIX: 
2025-06-06T00:20:56.4973159Z ##[endgroup]
2025-06-06T00:20:56.5872676Z grep: warning: ? at start of expression
2025-06-06T00:20:56.6284507Z Running integration tests with image: quay.io/cortexproject/cortex:HEAD-368026e-amd64
2025-06-06T00:20:56.6310566Z go: downloading go1.24.0 (linux/amd64)
2025-06-06T00:26:09.6316424Z === RUN   TestRulerAPI
2025-06-06T00:26:09.6316958Z === RUN   TestRulerAPI/thanosEngine=false
2025-06-06T00:26:09.6317470Z 00:22:30 Starting consul
2025-06-06T00:26:09.6317932Z 00:22:31 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.6318494Z 00:22:31 consul: Version: '1.8.4'
2025-06-06T00:26:09.6319086Z 00:22:31 consul: Node ID: '74864b3d-b92d-7bfa-f6ec-6fc09f883c4a'
2025-06-06T00:26:09.6319561Z 00:22:31 consul: Node name: 'consul'
2025-06-06T00:26:09.6319885Z 00:22:31 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.6320388Z 00:22:31 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.6321349Z 00:22:31 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.6322156Z 00:22:31 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.6323086Z 00:22:31 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.6323920Z 00:22:31 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.6324467Z 00:22:31 consul: ==> Consul agent running!
2025-06-06T00:26:09.6325184Z 00:22:31 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32768]
2025-06-06T00:26:09.6326197Z 00:22:31 Starting minio-9000
2025-06-06T00:26:09.6326824Z 00:22:32 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32769]
2025-06-06T00:26:09.6327465Z 00:22:34 Starting ruler
2025-06-06T00:26:09.6328044Z 00:22:34 ruler: 2025/06/06 00:22:34 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6329423Z 00:22:34 ruler: ts=2025-06-06T00:22:34.238984775Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6330733Z 00:22:34 Ports for container: e2e-cortex-test-ruler Mapping: map[80:32770 9095:32771]
2025-06-06T00:26:09.6332320Z 00:22:40 ruler: ts=2025-06-06T00:22:40.245124343Z caller=notifier.go:64 level=error user=user-1 msg="error starting notifier discovery manager" err="context canceled"
2025-06-06T00:26:09.6333369Z 00:22:40 Killing ruler
2025-06-06T00:26:09.6333943Z 00:22:40 Error response from daemon: No such container: e2e-cortex-test-ruler
2025-06-06T00:26:09.6334435Z 
2025-06-06T00:26:09.6334605Z 00:22:40 Killing minio-9000
2025-06-06T00:26:09.6335229Z 00:22:41 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.6335727Z 
2025-06-06T00:26:09.6335880Z 00:22:41 Killing consul
2025-06-06T00:26:09.6336452Z 00:22:41 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.6336934Z 
2025-06-06T00:26:09.6337130Z === RUN   TestRulerAPI/thanosEngine=true
2025-06-06T00:26:09.6337581Z 00:22:41 Starting consul
2025-06-06T00:26:09.6337996Z 00:22:41 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.6338758Z 00:22:41 consul: Version: '1.8.4'
2025-06-06T00:26:09.6340103Z 00:22:41 consul: Node ID: '613e1a9f-2cb3-7178-1f66-6c1d62241767'
2025-06-06T00:26:09.6340687Z 00:22:41 consul: Node name: 'consul'
2025-06-06T00:26:09.6341398Z 00:22:41 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.6341960Z 00:22:41 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.6342647Z 00:22:41 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.6343387Z 00:22:41 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.6344272Z 00:22:41 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.6345148Z 00:22:41 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.6345699Z 00:22:41 consul: ==> Consul agent running!
2025-06-06T00:26:09.6346350Z 00:22:41 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32772]
2025-06-06T00:26:09.6346979Z 00:22:41 Starting minio-9000
2025-06-06T00:26:09.6347860Z 00:22:42 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32773]
2025-06-06T00:26:09.6348496Z 00:22:43 Starting ruler
2025-06-06T00:26:09.6349074Z 00:22:43 ruler: 2025/06/06 00:22:43 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6350394Z 00:22:43 ruler: ts=2025-06-06T00:22:43.886678621Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6352435Z 00:22:44 Ports for container: e2e-cortex-test-ruler Mapping: map[80:32774 9095:32775]
2025-06-06T00:26:09.6353755Z 00:22:49 ruler: ts=2025-06-06T00:22:49.893214215Z caller=notifier.go:64 level=error user=user-1 msg="error starting notifier discovery manager" err="context canceled"
2025-06-06T00:26:09.6354784Z 00:22:50 Killing ruler
2025-06-06T00:26:09.6355360Z 00:22:50 Error response from daemon: No such container: e2e-cortex-test-ruler
2025-06-06T00:26:09.6355856Z 
2025-06-06T00:26:09.6356023Z 00:22:50 Killing minio-9000
2025-06-06T00:26:09.6356650Z 00:22:50 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.6357155Z 
2025-06-06T00:26:09.6357309Z 00:22:50 Killing consul
2025-06-06T00:26:09.6357880Z 00:22:50 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.6358359Z 
2025-06-06T00:26:09.6358768Z --- PASS: TestRulerAPI (20.19s)
2025-06-06T00:26:09.6359369Z     --- PASS: TestRulerAPI/thanosEngine=false (10.84s)
2025-06-06T00:26:09.6360038Z     --- PASS: TestRulerAPI/thanosEngine=true (9.35s)
2025-06-06T00:26:09.6360577Z === RUN   TestRulerAPISingleBinary
2025-06-06T00:26:09.6361020Z 00:22:50 Starting cortex
2025-06-06T00:26:09.6361772Z 00:22:50 cortex: 2025/06/06 00:22:50 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6363261Z 00:22:50 cortex: ts=2025-06-06T00:22:50.877809235Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/shared/rule_tmp err="open /shared/rule_tmp: no such file or directory"
2025-06-06T00:26:09.6365784Z 00:22:50 cortex: ts=2025-06-06T00:22:50.88093762Z caller=modules.go:370 level=warn msg="Worker address is empty in single binary mode.  Attempting automatic worker configuration.  If queries are unresponsive consider configuring the worker explicitly." address=127.0.0.1:9095
2025-06-06T00:26:09.6367804Z 00:22:51 Ports for container: e2e-cortex-test-cortex Mapping: map[9009:32776 9095:32777]
2025-06-06T00:26:09.6368490Z 00:22:51 Stopping cortex
2025-06-06T00:26:09.6369511Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374563387Z caller=module_service.go:94 level=warn msg="module failed with error" module=compactor err="context canceled"
2025-06-06T00:26:09.6371032Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374612659Z caller=cortex.go:460 level=error msg="module failed" module=compactor err="context canceled"
2025-06-06T00:26:09.6373042Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374619529Z caller=notifier.go:64 level=error user=fake msg="error starting notifier discovery manager" err="context canceled"
2025-06-06T00:26:09.6374959Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374645489Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6377027Z 00:22:51 cortex: ts=2025-06-06T00:22:51.37468297Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6379098Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374700933Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6381407Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374717384Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6383496Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374734146Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6385563Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374756568Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6387628Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374755738Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6389680Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374776335Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6391917Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374781145Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6394002Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374793487Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6396055Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374802675Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6398295Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374809767Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6400357Z 00:22:51 cortex: ts=2025-06-06T00:22:51.37482733Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6402560Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374836369Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6404631Z 00:22:51 cortex: ts=2025-06-06T00:22:51.37485382Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6406687Z 00:22:51 cortex: ts=2025-06-06T00:22:51.374872234Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6408751Z 00:22:51 cortex: ts=2025-06-06T00:22:51.37491281Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6411201Z 00:22:51 cortex: ts=2025-06-06T00:22:51.37492833Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6413333Z 00:22:51 cortex: ts=2025-06-06T00:22:51.37496044Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6415396Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375028409Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
2025-06-06T00:26:09.6417414Z 00:22:51 cortex: ts=2025-06-06T00:22:51.37513599Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.251034ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6419218Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375198174Z caller=grpc_logging.go:74 level=warn duration=492.358322ms method=/frontend.Frontend/Process err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6421025Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375199888Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.306105ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6422985Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375217421Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.399781ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6424795Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375176475Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.370836ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6426594Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375237649Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.411352ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6428387Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375239762Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.542151ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6430212Z 00:22:51 cortex: ts=2025-06-06T00:22:51.37532273Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.461028ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6432188Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375328299Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.563218ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6434393Z 00:22:51 cortex: ts=2025-06-06T00:22:51.37538717Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.671004ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6436183Z 00:22:51 cortex: ts=2025-06-06T00:22:51.37545063Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.544634ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6437978Z 00:22:51 cortex: ts=2025-06-06T00:22:51.37547804Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.694358ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6439775Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375270048Z caller=grpc_logging.go:74 level=warn duration=492.540217ms method=/frontend.Frontend/Process err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6441744Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375368495Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.790067ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6443553Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375409332Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.647652ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6445349Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375431244Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.557298ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6447307Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375176373Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.427877ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6449098Z 00:22:51 cortex: ts=2025-06-06T00:22:51.37550407Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.731369ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6450885Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375253678Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.405049ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6452842Z 00:22:51 cortex: ts=2025-06-06T00:22:51.375539303Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=492.748072ms err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.6456078Z 00:22:52 cortex: ts=2025-06-06T00:22:52.87648078Z caller=log.go:125 level=error msg="error running cortex" err="failed services\ngithub.com/cortexproject/cortex/pkg/cortex.(*Cortex).Run\n\t/__w/cortex/cortex/pkg/cortex/cortex.go:495\nmain.main\n\t/__w/cortex/cortex/cmd/cortex/main.go:202\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:283\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"
2025-06-06T00:26:09.6458685Z 00:22:53 Error response from daemon: No such container: e2e-cortex-test-cortex
2025-06-06T00:26:09.6459174Z 
2025-06-06T00:26:09.6459360Z 00:22:53 Starting cortex-restarted
2025-06-06T00:26:09.6460079Z 00:22:53 cortex-restarted: 2025/06/06 00:22:53 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6462305Z 00:22:53 cortex-restarted: ts=2025-06-06T00:22:53.208811941Z caller=modules.go:370 level=warn msg="Worker address is empty in single binary mode.  Attempting automatic worker configuration.  If queries are unresponsive consider configuring the worker explicitly." address=127.0.0.1:9095
2025-06-06T00:26:09.6464383Z 00:22:53 Ports for container: e2e-cortex-test-cortex-restarted Mapping: map[9009:32778 9095:32779]
2025-06-06T00:26:09.6465152Z 00:22:53 Killing cortex-restarted
2025-06-06T00:26:09.6465846Z 00:22:53 Error response from daemon: No such container: e2e-cortex-test-cortex-restarted
2025-06-06T00:26:09.6466390Z 
2025-06-06T00:26:09.6467840Z 00:22:53 error while removing sharedDir /home/runner/work/cortex/cortex/integration/e2e_integration_test913768499 err: unlinkat /home/runner/work/cortex/cortex/integration/e2e_integration_test913768499/rule_tmp/fake/ns: permission denied
2025-06-06T00:26:09.6469568Z --- PASS: TestRulerAPISingleBinary (3.29s)
2025-06-06T00:26:09.6470248Z === RUN   TestRulerSharding
2025-06-06T00:26:09.6470656Z 00:22:53 Starting consul
2025-06-06T00:26:09.6471260Z 00:22:54 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.6471757Z 00:22:54 consul: Version: '1.8.4'
2025-06-06T00:26:09.6472402Z 00:22:54 consul: Node ID: '0ec39d22-136b-568d-ce46-53b8b6e91b72'
2025-06-06T00:26:09.6472972Z 00:22:54 consul: Node name: 'consul'
2025-06-06T00:26:09.6473469Z 00:22:54 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.6474016Z 00:22:54 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.6474710Z 00:22:54 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.6475437Z 00:22:54 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.6476318Z 00:22:54 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.6477291Z 00:22:54 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.6477885Z 00:22:54 consul: ==> Consul agent running!
2025-06-06T00:26:09.6478558Z 00:22:54 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32780]
2025-06-06T00:26:09.6479194Z 00:22:54 Starting minio-9000
2025-06-06T00:26:09.6479812Z 00:22:54 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32781]
2025-06-06T00:26:09.6480457Z 00:22:56 Starting ruler-1
2025-06-06T00:26:09.6481220Z 00:22:56 ruler-1: 2025/06/06 00:22:56 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6482741Z 00:22:56 ruler-1: ts=2025-06-06T00:22:56.854866972Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6484095Z 00:22:57 Ports for container: e2e-cortex-test-ruler-1 Mapping: map[80:32782 9095:32783]
2025-06-06T00:26:09.6484772Z 00:22:57 Starting ruler-2
2025-06-06T00:26:09.6485366Z 00:22:57 ruler-2: 2025/06/06 00:22:57 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6486689Z 00:22:57 ruler-2: ts=2025-06-06T00:22:57.27046291Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6488015Z 00:22:57 Ports for container: e2e-cortex-test-ruler-2 Mapping: map[80:32784 9095:32785]
2025-06-06T00:26:09.6488686Z 00:22:59 Killing ruler-2
2025-06-06T00:26:09.6489273Z 00:22:59 Error response from daemon: No such container: e2e-cortex-test-ruler-2
2025-06-06T00:26:09.6489799Z 
2025-06-06T00:26:09.6489948Z 00:22:59 Killing ruler-1
2025-06-06T00:26:09.6490551Z 00:22:59 Error response from daemon: No such container: e2e-cortex-test-ruler-1
2025-06-06T00:26:09.6491200Z 
2025-06-06T00:26:09.6491368Z 00:22:59 Killing minio-9000
2025-06-06T00:26:09.6491996Z 00:22:59 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.6492509Z 
2025-06-06T00:26:09.6492659Z 00:22:59 Killing consul
2025-06-06T00:26:09.6493236Z 00:22:59 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.6493740Z 
2025-06-06T00:26:09.6493923Z --- PASS: TestRulerSharding (6.13s)
2025-06-06T00:26:09.6494394Z === RUN   TestRulerAPISharding
2025-06-06T00:26:09.6494827Z 00:23:00 Starting consul
2025-06-06T00:26:09.6495253Z 00:23:00 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.6495734Z 00:23:00 consul: Version: '1.8.4'
2025-06-06T00:26:09.6496266Z 00:23:00 consul: Node ID: '6d859d86-d82b-77fa-5c66-7784968fa6e9'
2025-06-06T00:26:09.6496821Z 00:23:00 consul: Node name: 'consul'
2025-06-06T00:26:09.6497318Z 00:23:00 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.6497876Z 00:23:00 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.6498553Z 00:23:00 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.6499278Z 00:23:00 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.6500161Z 00:23:00 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.6501374Z 00:23:00 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.6501943Z 00:23:00 consul: ==> Consul agent running!
2025-06-06T00:26:09.6502594Z 00:23:00 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32786]
2025-06-06T00:26:09.6503230Z 00:23:00 Starting minio-9000
2025-06-06T00:26:09.6503842Z 00:23:01 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32787]
2025-06-06T00:26:09.6504487Z 00:23:02 Starting ruler-1
2025-06-06T00:26:09.6505085Z 00:23:02 ruler-1: 2025/06/06 00:23:02 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6506434Z 00:23:02 ruler-1: ts=2025-06-06T00:23:02.721626677Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6507753Z 00:23:03 Ports for container: e2e-cortex-test-ruler-1 Mapping: map[80:32788 9095:32789]
2025-06-06T00:26:09.6508428Z 00:23:03 Starting ruler-2
2025-06-06T00:26:09.6509015Z 00:23:03 ruler-2: 2025/06/06 00:23:03 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6510340Z 00:23:03 ruler-2: ts=2025-06-06T00:23:03.205087805Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6511810Z 00:23:03 Ports for container: e2e-cortex-test-ruler-2 Mapping: map[80:32790 9095:32791]
2025-06-06T00:26:09.6512474Z 00:23:03 Starting ruler-3
2025-06-06T00:26:09.6513058Z 00:23:03 ruler-3: 2025/06/06 00:23:03 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6514548Z 00:23:03 ruler-3: ts=2025-06-06T00:23:03.805888196Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6515872Z 00:23:04 Ports for container: e2e-cortex-test-ruler-3 Mapping: map[80:32792 9095:32793]
2025-06-06T00:26:09.6516623Z === RUN   TestRulerAPISharding/Filter_for_Alert_Rules
2025-06-06T00:26:09.6517234Z === RUN   TestRulerAPISharding/Filter_for_Recording_Rules
2025-06-06T00:26:09.6517863Z === RUN   TestRulerAPISharding/Filter_by_Namespace_Name
2025-06-06T00:26:09.6518561Z === RUN   TestRulerAPISharding/Filter_by_Namespace_Name_and_Alert_Rules
2025-06-06T00:26:09.6519246Z === RUN   TestRulerAPISharding/Filter_by_Rule_Names
2025-06-06T00:26:09.6519798Z === RUN   TestRulerAPISharding/Exclude_Alerts
2025-06-06T00:26:09.6520330Z === RUN   TestRulerAPISharding/Include_Alerts
2025-06-06T00:26:09.6521002Z === RUN   TestRulerAPISharding/Filter_Rules_and_verify_Group_Labels_exist
2025-06-06T00:26:09.6521779Z 00:23:08 Killing ruler-3
2025-06-06T00:26:09.6522375Z 00:23:08 Error response from daemon: No such container: e2e-cortex-test-ruler-3
2025-06-06T00:26:09.6522882Z 
2025-06-06T00:26:09.6523030Z 00:23:08 Killing ruler-2
2025-06-06T00:26:09.6523607Z 00:23:08 Error response from daemon: No such container: e2e-cortex-test-ruler-2
2025-06-06T00:26:09.6524092Z 
2025-06-06T00:26:09.6524240Z 00:23:08 Killing ruler-1
2025-06-06T00:26:09.6524817Z 00:23:08 Error response from daemon: No such container: e2e-cortex-test-ruler-1
2025-06-06T00:26:09.6525301Z 
2025-06-06T00:26:09.6525463Z 00:23:08 Killing minio-9000
2025-06-06T00:26:09.6526079Z 00:23:08 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.6526579Z 
2025-06-06T00:26:09.6526731Z 00:23:08 Killing consul
2025-06-06T00:26:09.6527301Z 00:23:09 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.6527781Z 
2025-06-06T00:26:09.6527981Z --- PASS: TestRulerAPISharding (9.21s)
2025-06-06T00:26:09.6528685Z     --- PASS: TestRulerAPISharding/Filter_for_Alert_Rules (0.00s)
2025-06-06T00:26:09.6529517Z     --- PASS: TestRulerAPISharding/Filter_for_Recording_Rules (0.00s)
2025-06-06T00:26:09.6530353Z     --- PASS: TestRulerAPISharding/Filter_by_Namespace_Name (0.00s)
2025-06-06T00:26:09.6531435Z     --- PASS: TestRulerAPISharding/Filter_by_Namespace_Name_and_Alert_Rules (0.00s)
2025-06-06T00:26:09.6532344Z     --- PASS: TestRulerAPISharding/Filter_by_Rule_Names (0.00s)
2025-06-06T00:26:09.6533265Z     --- PASS: TestRulerAPISharding/Exclude_Alerts (0.00s)
2025-06-06T00:26:09.6533960Z     --- PASS: TestRulerAPISharding/Include_Alerts (0.00s)
2025-06-06T00:26:09.6534869Z     --- PASS: TestRulerAPISharding/Filter_Rules_and_verify_Group_Labels_exist (0.00s)
2025-06-06T00:26:09.6535701Z === RUN   TestRulerAPIShardingWithAPIRulesBackupEnabled
2025-06-06T00:26:09.6536258Z 00:23:09 Starting consul
2025-06-06T00:26:09.6536680Z 00:23:09 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.6537161Z 00:23:09 consul: Version: '1.8.4'
2025-06-06T00:26:09.6537698Z 00:23:09 consul: Node ID: '94b9fba5-2868-24ef-7e94-3a30101c087f'
2025-06-06T00:26:09.6538251Z 00:23:09 consul: Node name: 'consul'
2025-06-06T00:26:09.6538748Z 00:23:09 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.6539293Z 00:23:09 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.6539974Z 00:23:09 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.6540706Z 00:23:09 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.6541793Z 00:23:09 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.6542662Z 00:23:09 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.6543228Z 00:23:09 consul: ==> Consul agent running!
2025-06-06T00:26:09.6543885Z 00:23:09 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32794]
2025-06-06T00:26:09.6544520Z 00:23:09 Starting minio-9000
2025-06-06T00:26:09.6545304Z 00:23:10 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32795]
2025-06-06T00:26:09.6545967Z 00:23:11 Starting ruler-1
2025-06-06T00:26:09.6546555Z 00:23:11 ruler-1: 2025/06/06 00:23:11 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6547917Z 00:23:12 ruler-1: ts=2025-06-06T00:23:12.018233795Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6549256Z 00:23:12 Ports for container: e2e-cortex-test-ruler-1 Mapping: map[80:32796 9095:32797]
2025-06-06T00:26:09.6549939Z 00:23:12 Starting ruler-2
2025-06-06T00:26:09.6550533Z 00:23:12 ruler-2: 2025/06/06 00:23:12 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6552053Z 00:23:12 ruler-2: ts=2025-06-06T00:23:12.405813297Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6553367Z 00:23:12 Ports for container: e2e-cortex-test-ruler-2 Mapping: map[80:32798 9095:32799]
2025-06-06T00:26:09.6554032Z 00:23:12 Starting ruler-3
2025-06-06T00:26:09.6554606Z 00:23:12 ruler-3: 2025/06/06 00:23:12 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6555917Z 00:23:12 ruler-3: ts=2025-06-06T00:23:12.893774435Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6557223Z 00:23:13 Ports for container: e2e-cortex-test-ruler-3 Mapping: map[80:32800 9095:32801]
2025-06-06T00:26:09.6559109Z 00:23:18 ruler-1: ts=2025-06-06T00:23:18.331584764Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:18.33157722Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_41 name=rule_41 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6562061Z 00:23:18 ruler-1: ts=2025-06-06T00:23:18.463588687Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:18.463580061Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_88 name=rule_88 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6564887Z 00:23:18 ruler-1: ts=2025-06-06T00:23:18.478822259Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:18.478813472Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_23 name=rule_23 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6566668Z 00:23:18 Killing ruler-2
2025-06-06T00:26:09.6568302Z 00:23:18 ruler-1: ts=2025-06-06T00:23:18.573476875Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:18.573468069Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_54 name=rule_54 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6571220Z 00:23:18 ruler-1: ts=2025-06-06T00:23:18.584544804Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:18.584537921Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_9 name=rule_9 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6573996Z 00:23:18 ruler-1: ts=2025-06-06T00:23:18.620859789Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:18.620846373Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_40 name=rule_40 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6575796Z 00:23:18 Error response from daemon: No such container: e2e-cortex-test-ruler-2
2025-06-06T00:26:09.6576295Z 
2025-06-06T00:26:09.6576740Z === RUN   TestRulerAPIShardingWithAPIRulesBackupEnabled/Filter_by_Namespace_Name
2025-06-06T00:26:09.6578689Z 00:23:18 ruler-1: ts=2025-06-06T00:23:18.662097049Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:18.662090056Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_49 name=rule_49 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6581827Z 00:23:18 ruler-1: ts=2025-06-06T00:23:18.667231622Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:18.66722531Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_4 name=rule_4 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6584616Z 00:23:18 ruler-1: ts=2025-06-06T00:23:18.667326609Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:18.667316951Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_76 name=rule_76 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6587402Z 00:23:18 ruler-1: ts=2025-06-06T00:23:18.979716148Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:18.979708834Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_97 name=rule_97 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6590193Z 00:23:18 ruler-1: ts=2025-06-06T00:23:18.987819632Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:18.987811036Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_10 name=rule_10 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6593137Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.013972643Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.013967133Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_37 name=rule_37 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6595917Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.016071452Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.016066312Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_78 name=rule_78 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6598695Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.019240613Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.019235584Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_71 name=rule_71 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6601642Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.330822499Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.330815676Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_41 name=rule_41 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6604724Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.464224551Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.464217228Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_88 name=rule_88 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6607529Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.479257042Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.479251411Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_23 name=rule_23 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6610310Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.573486095Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.573479663Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_54 name=rule_54 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6613214Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.58460005Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.584594851Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_9 name=rule_9 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6615999Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.620781327Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.620776177Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_40 name=rule_40 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6618940Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.662084084Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.662076029Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_49 name=rule_49 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6621874Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.667185701Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.667178898Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_4 name=rule_4 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6624650Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.66720369Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.667196015Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_76 name=rule_76 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6627445Z 00:23:19 ruler-3: ts=2025-06-06T00:23:19.870623858Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.870617156Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_36 name=rule_36 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6630229Z 00:23:19 ruler-3: ts=2025-06-06T00:23:19.884783826Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.884778335Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_57 name=rule_57 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6633213Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.97967648Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.979670549Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_97 name=rule_97 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6635990Z 00:23:19 ruler-1: ts=2025-06-06T00:23:19.987814465Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:19.987807652Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_10 name=rule_10 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6638754Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.007444234Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.007439084Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_77 name=rule_77 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6641655Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.013930262Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.013925043Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_37 name=rule_37 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6644627Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.0160644Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.016059721Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_78 name=rule_78 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6647389Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.019256776Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.019250985Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_71 name=rule_71 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6650168Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.024550043Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.024544863Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_79 name=rule_79 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6653093Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.044338842Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.044333712Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_33 name=rule_33 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6656037Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.11149892Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.111493239Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_35 name=rule_35 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6658818Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.129658272Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.129653002Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_86 name=rule_86 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6661733Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.144793332Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.144788343Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_1 name=rule_1 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6664561Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.189051279Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.189045067Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_47 name=rule_47 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6667333Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.331196284Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.33118881Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_41 name=rule_41 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6670111Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.463521947Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.463507931Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_88 name=rule_88 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6673086Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.479536843Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.479531564Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_23 name=rule_23 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6675879Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.516535308Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.516517194Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_45 name=rule_45 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6678656Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.547654961Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.547649571Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_17 name=rule_17 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6681823Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.573750973Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.573745813Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_54 name=rule_54 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6684627Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.584930364Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.584924353Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_9 name=rule_9 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6687400Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.618844602Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.618839693Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_72 name=rule_72 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6690186Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.620076265Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.620070725Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_40 name=rule_40 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6693283Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.632029249Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.632022337Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6696072Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.655090856Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.655085166Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_70 name=rule_70 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6698858Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.662220886Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.662214564Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_49 name=rule_49 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6701780Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.666378569Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.666370754Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_4 name=rule_4 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6704555Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.666465069Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.66645485Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_76 name=rule_76 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6707316Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.831937339Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.831931728Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_73 name=rule_73 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6710113Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.871126586Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.87110255Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_36 name=rule_36 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6712498Z 00:23:20 ruler-3: ts=2025-06-06T00:23:20.88423085Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.884224247Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_57 name=rule_57 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6714089Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.978837984Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.978829938Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_97 name=rule_97 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6715786Z 00:23:20 ruler-1: ts=2025-06-06T00:23:20.987978522Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:20.987973212Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_10 name=rule_10 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6717346Z 00:23:21 ruler-3: ts=2025-06-06T00:23:21.007909856Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.007899287Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_77 name=rule_77 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6718982Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.014192066Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.014186406Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_37 name=rule_37 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6720532Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.016283583Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.016278463Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_78 name=rule_78 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6722189Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.019449466Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.019443946Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_71 name=rule_71 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6723847Z 00:23:21 ruler-3: ts=2025-06-06T00:23:21.023990421Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.023983468Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_79 name=rule_79 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6725424Z 00:23:21 ruler-3: ts=2025-06-06T00:23:21.04517867Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.045171817Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_33 name=rule_33 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6726967Z 00:23:21 ruler-3: ts=2025-06-06T00:23:21.11137968Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.11137475Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_35 name=rule_35 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6728507Z 00:23:21 ruler-3: ts=2025-06-06T00:23:21.129547527Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.129542057Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_86 name=rule_86 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6730054Z 00:23:21 ruler-3: ts=2025-06-06T00:23:21.144676446Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.144670725Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_1 name=rule_1 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6731707Z 00:23:21 ruler-3: ts=2025-06-06T00:23:21.188891518Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.188886439Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_47 name=rule_47 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6733250Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.331322974Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.331317564Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_41 name=rule_41 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6734788Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.463813509Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.463806576Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_88 name=rule_88 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6736445Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.47892166Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.478916199Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_23 name=rule_23 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6737986Z 00:23:21 ruler-3: ts=2025-06-06T00:23:21.516445566Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.516438212Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_45 name=rule_45 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6739536Z 00:23:21 ruler-3: ts=2025-06-06T00:23:21.547606087Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.547600647Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_17 name=rule_17 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6741177Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.573232436Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.573225252Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_54 name=rule_54 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6742733Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.585349481Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.585344431Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_9 name=rule_9 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6744583Z 00:23:21 ruler-3: ts=2025-06-06T00:23:21.618736281Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.618731321Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_72 name=rule_72 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6746148Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.620449361Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.620444883Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_40 name=rule_40 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6747709Z 00:23:21 ruler-3: ts=2025-06-06T00:23:21.63187668Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.631872232Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6749254Z 00:23:21 ruler-3: ts=2025-06-06T00:23:21.656097613Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.656092243Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_70 name=rule_70 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6750809Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.661609767Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.661604988Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_49 name=rule_49 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6752597Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.666762993Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.666757462Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_4 name=rule_4 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6754161Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.666984831Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:21.666977878Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_76 name=rule_76 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6755781Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.738335809Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.6757075Z === RUN   TestRulerAPIShardingWithAPIRulesBackupEnabled/Filter_by_Namespace_Name_and_Alert_Rules
2025-06-06T00:26:09.6758274Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.740681144Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.6759340Z === RUN   TestRulerAPIShardingWithAPIRulesBackupEnabled/Filter_by_Rule_Names
2025-06-06T00:26:09.6760461Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.74214517Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.6761613Z === RUN   TestRulerAPIShardingWithAPIRulesBackupEnabled/Exclude_Alerts
2025-06-06T00:26:09.6762714Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.743360019Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.6763755Z === RUN   TestRulerAPIShardingWithAPIRulesBackupEnabled/Include_Alerts
2025-06-06T00:26:09.6764980Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.748281333Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.6766123Z === RUN   TestRulerAPIShardingWithAPIRulesBackupEnabled/Filter_Rules_and_verify_Group_Labels_exist
2025-06-06T00:26:09.6767312Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.751139426Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.6768379Z === RUN   TestRulerAPIShardingWithAPIRulesBackupEnabled/Filter_for_Alert_Rules
2025-06-06T00:26:09.6769505Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.754201847Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.6770591Z === RUN   TestRulerAPIShardingWithAPIRulesBackupEnabled/Filter_for_Recording_Rules
2025-06-06T00:26:09.6771845Z 00:23:21 ruler-1: ts=2025-06-06T00:23:21.756368607Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.6772776Z 00:23:21 Killing ruler-3
2025-06-06T00:26:09.6773132Z 00:23:21 Error response from daemon: No such container: e2e-cortex-test-ruler-3
2025-06-06T00:26:09.6773417Z 
2025-06-06T00:26:09.6773507Z 00:23:21 Killing ruler-1
2025-06-06T00:26:09.6773735Z 00:23:22 Killing minio-9000
2025-06-06T00:26:09.6774105Z 00:23:22 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.6774401Z 
2025-06-06T00:26:09.6774491Z 00:23:22 Killing consul
2025-06-06T00:26:09.6774835Z 00:23:22 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.6775112Z 
2025-06-06T00:26:09.6775317Z --- PASS: TestRulerAPIShardingWithAPIRulesBackupEnabled (13.18s)
2025-06-06T00:26:09.6775956Z     --- PASS: TestRulerAPIShardingWithAPIRulesBackupEnabled/Filter_by_Namespace_Name (3.09s)
2025-06-06T00:26:09.6776749Z     --- PASS: TestRulerAPIShardingWithAPIRulesBackupEnabled/Filter_by_Namespace_Name_and_Alert_Rules (0.00s)
2025-06-06T00:26:09.6777622Z     --- PASS: TestRulerAPIShardingWithAPIRulesBackupEnabled/Filter_by_Rule_Names (0.00s)
2025-06-06T00:26:09.6778270Z     --- PASS: TestRulerAPIShardingWithAPIRulesBackupEnabled/Exclude_Alerts (0.00s)
2025-06-06T00:26:09.6778888Z     --- PASS: TestRulerAPIShardingWithAPIRulesBackupEnabled/Include_Alerts (0.00s)
2025-06-06T00:26:09.6779627Z     --- PASS: TestRulerAPIShardingWithAPIRulesBackupEnabled/Filter_Rules_and_verify_Group_Labels_exist (0.00s)
2025-06-06T00:26:09.6780387Z     --- PASS: TestRulerAPIShardingWithAPIRulesBackupEnabled/Filter_for_Alert_Rules (0.00s)
2025-06-06T00:26:09.6781207Z     --- PASS: TestRulerAPIShardingWithAPIRulesBackupEnabled/Filter_for_Recording_Rules (0.00s)
2025-06-06T00:26:09.6781695Z === RUN   TestRulesPaginationAPISharding
2025-06-06T00:26:09.6781985Z 00:23:22 Starting consul
2025-06-06T00:26:09.6782240Z 00:23:22 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.6782527Z 00:23:22 consul: Version: '1.8.4'
2025-06-06T00:26:09.6782851Z 00:23:22 consul: Node ID: '89752a0b-a67e-cf3a-23c2-6517d0bb41d0'
2025-06-06T00:26:09.6783178Z 00:23:22 consul: Node name: 'consul'
2025-06-06T00:26:09.6783474Z 00:23:22 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.6783797Z 00:23:22 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.6784197Z 00:23:22 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.6784620Z 00:23:22 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.6785251Z 00:23:22 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.6785757Z 00:23:22 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.6786085Z 00:23:22 consul: ==> Consul agent running!
2025-06-06T00:26:09.6786466Z 00:23:23 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32802]
2025-06-06T00:26:09.6786838Z 00:23:23 Starting minio-9000
2025-06-06T00:26:09.6787200Z 00:23:23 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32803]
2025-06-06T00:26:09.6787579Z 00:23:25 Starting ruler-1
2025-06-06T00:26:09.6787933Z 00:23:25 ruler-1: 2025/06/06 00:23:25 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6788697Z 00:23:25 ruler-1: ts=2025-06-06T00:23:25.445060803Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6789445Z 00:23:25 Ports for container: e2e-cortex-test-ruler-1 Mapping: map[80:32804 9095:32805]
2025-06-06T00:26:09.6789833Z 00:23:25 Starting ruler-2
2025-06-06T00:26:09.6790188Z 00:23:25 ruler-2: 2025/06/06 00:23:25 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6790941Z 00:23:25 ruler-2: ts=2025-06-06T00:23:25.890443918Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6791803Z 00:23:26 Ports for container: e2e-cortex-test-ruler-2 Mapping: map[80:32806 9095:32807]
2025-06-06T00:26:09.6792202Z 00:23:26 Starting ruler-3
2025-06-06T00:26:09.6792549Z 00:23:26 ruler-3: 2025/06/06 00:23:26 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6793290Z 00:23:26 ruler-3: ts=2025-06-06T00:23:26.406320406Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6794025Z 00:23:26 Ports for container: e2e-cortex-test-ruler-3 Mapping: map[80:32808 9095:32809]
2025-06-06T00:26:09.6795097Z 00:23:30 ruler-1: ts=2025-06-06T00:23:30.429364203Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.429354655Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_75 name=rule_75 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6796674Z 00:23:30 ruler-1: ts=2025-06-06T00:23:30.429364185Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.429354667Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_85 name=rule_85 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6798362Z 00:23:30 ruler-1: ts=2025-06-06T00:23:30.450474426Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.450465269Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_98 name=rule_98 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6799918Z 00:23:30 ruler-1: ts=2025-06-06T00:23:30.501908308Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.501899081Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_93 name=rule_93 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6801564Z 00:23:30 ruler-1: ts=2025-06-06T00:23:30.562568793Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.562559335Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_61 name=rule_61 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6803118Z 00:23:30 ruler-1: ts=2025-06-06T00:23:30.572042318Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.572036397Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_54 name=rule_54 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6804778Z 00:23:30 ruler-1: ts=2025-06-06T00:23:30.661598886Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.661591463Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_32 name=rule_32 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6806326Z 00:23:30 ruler-1: ts=2025-06-06T00:23:30.783720472Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.783711044Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_8 name=rule_8 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6807871Z 00:23:30 ruler-1: ts=2025-06-06T00:23:30.788845114Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.78883776Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_96 name=rule_96 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6809441Z 00:23:30 ruler-1: ts=2025-06-06T00:23:30.813563778Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.813553008Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_19 name=rule_19 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6811001Z 00:23:30 ruler-1: ts=2025-06-06T00:23:30.836325489Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.836316832Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_99 name=rule_99 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6812668Z 00:23:30 ruler-1: ts=2025-06-06T00:23:30.950939803Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.950932159Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_40 name=rule_40 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6814217Z 00:23:30 ruler-2: ts=2025-06-06T00:23:30.989529997Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:30.989518405Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_73 name=rule_73 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6815774Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.021673351Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.021665767Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_46 name=rule_46 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6817318Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.051325345Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.051307742Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_88 name=rule_88 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6819038Z 00:23:31 ruler-2: ts=2025-06-06T00:23:31.134262913Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.134252593Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_72 name=rule_72 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6820601Z 00:23:31 ruler-2: ts=2025-06-06T00:23:31.200854171Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.200829174Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_29 name=rule_29 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6822260Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.264218403Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.264208063Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_39 name=rule_39 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6823810Z 00:23:31 ruler-2: ts=2025-06-06T00:23:31.372295365Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.37228715Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_56 name=rule_56 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6825356Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.430859228Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.430828971Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_75 name=rule_75 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6827009Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.431079571Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.431070313Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_85 name=rule_85 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6828574Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.451385036Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.451373033Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_98 name=rule_98 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6830129Z 00:23:31 ruler-3: ts=2025-06-06T00:23:31.457868169Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.45785806Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_86 name=rule_86 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6831802Z 00:23:31 ruler-2: ts=2025-06-06T00:23:31.472085881Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.472077886Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_69 name=rule_69 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6832910Z === RUN   TestRulesPaginationAPISharding/List_Rule_Groups_-_Equal_number_of_rule_groups_per_page
2025-06-06T00:26:09.6834036Z 00:23:31 ruler-2: ts=2025-06-06T00:23:31.488364344Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.488354796Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_63 name=rule_63 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6835049Z === RUN   TestRulesPaginationAPISharding/List_Rule_Groups_-_Last_page_unequal
2025-06-06T00:26:09.6835523Z === RUN   TestRulesPaginationAPISharding/List_all_rule_groups
2025-06-06T00:26:09.6836532Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.502186737Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.502178392Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_93 name=rule_93 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6837579Z === RUN   TestRulesPaginationAPISharding/List_all_rule_groups_-_Max_Rule_Groups_>_Actual
2025-06-06T00:26:09.6838056Z 00:23:31 Killing ruler-3
2025-06-06T00:26:09.6838959Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.563297286Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.563288319Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_61 name=rule_61 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6840662Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.572326271Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.572319689Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_54 name=rule_54 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6842367Z 00:23:31 ruler-2: ts=2025-06-06T00:23:31.655472324Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.655460613Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_70 name=rule_70 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6843919Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.662078957Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.662069098Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_32 name=rule_32 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6844923Z 00:23:31 Error response from daemon: No such container: e2e-cortex-test-ruler-3
2025-06-06T00:26:09.6845203Z 
2025-06-06T00:26:09.6845303Z 00:23:31 Killing ruler-2
2025-06-06T00:26:09.6846203Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.783949762Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.783941417Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_8 name=rule_8 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6847867Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.789050078Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.789042113Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_96 name=rule_96 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6849431Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.814607761Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.814596229Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_19 name=rule_19 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6850435Z 00:23:31 Error response from daemon: No such container: e2e-cortex-test-ruler-2
2025-06-06T00:26:09.6850718Z 
2025-06-06T00:26:09.6850806Z 00:23:31 Killing ruler-1
2025-06-06T00:26:09.6851981Z 00:23:31 ruler-1: ts=2025-06-06T00:23:31.837013062Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:31.836984058Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_99 name=rule_99 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6853111Z 00:23:31 Error response from daemon: No such container: e2e-cortex-test-ruler-1
2025-06-06T00:26:09.6853422Z 
2025-06-06T00:26:09.6853589Z 00:23:31 Killing minio-9000
2025-06-06T00:26:09.6854124Z 00:23:32 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.6854502Z 
2025-06-06T00:26:09.6871419Z 00:23:32 Killing consul
2025-06-06T00:26:09.6871878Z 00:23:32 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.6872187Z 
2025-06-06T00:26:09.6872351Z --- PASS: TestRulesPaginationAPISharding (9.96s)
2025-06-06T00:26:09.6873020Z     --- PASS: TestRulesPaginationAPISharding/List_Rule_Groups_-_Equal_number_of_rule_groups_per_page (0.01s)
2025-06-06T00:26:09.6873755Z     --- PASS: TestRulesPaginationAPISharding/List_Rule_Groups_-_Last_page_unequal (0.00s)
2025-06-06T00:26:09.6874363Z     --- PASS: TestRulesPaginationAPISharding/List_all_rule_groups (0.00s)
2025-06-06T00:26:09.6875013Z     --- PASS: TestRulesPaginationAPISharding/List_all_rule_groups_-_Max_Rule_Groups_>_Actual (0.00s)
2025-06-06T00:26:09.6875583Z === RUN   TestRulesPaginationAPIShardingWithAPIRulesBackupEnabled
2025-06-06T00:26:09.6875953Z 00:23:32 Starting consul
2025-06-06T00:26:09.6876210Z 00:23:32 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.6876691Z 00:23:32 consul: Version: '1.8.4'
2025-06-06T00:26:09.6877024Z 00:23:32 consul: Node ID: 'c42dcf9f-60b6-95d2-788d-4cd618a06288'
2025-06-06T00:26:09.6877359Z 00:23:32 consul: Node name: 'consul'
2025-06-06T00:26:09.6877678Z 00:23:32 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.6878018Z 00:23:32 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.6878438Z 00:23:32 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.6878870Z 00:23:32 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.6879411Z 00:23:32 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.6879929Z 00:23:32 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.6880266Z 00:23:32 consul: ==> Consul agent running!
2025-06-06T00:26:09.6880661Z 00:23:32 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32810]
2025-06-06T00:26:09.6881040Z 00:23:32 Starting minio-9000
2025-06-06T00:26:09.6881530Z 00:23:33 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32811]
2025-06-06T00:26:09.6881911Z 00:23:35 Starting ruler-1
2025-06-06T00:26:09.6882269Z 00:23:35 ruler-1: 2025/06/06 00:23:35 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6883059Z 00:23:35 ruler-1: ts=2025-06-06T00:23:35.297163125Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6883828Z 00:23:35 Ports for container: e2e-cortex-test-ruler-1 Mapping: map[80:32812 9095:32813]
2025-06-06T00:26:09.6884363Z 00:23:35 Starting ruler-2
2025-06-06T00:26:09.6884733Z 00:23:35 ruler-2: 2025/06/06 00:23:35 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6885495Z 00:23:35 ruler-2: ts=2025-06-06T00:23:35.65383137Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6886244Z 00:23:35 Ports for container: e2e-cortex-test-ruler-2 Mapping: map[80:32814 9095:32815]
2025-06-06T00:26:09.6886633Z 00:23:35 Starting ruler-3
2025-06-06T00:26:09.6886980Z 00:23:36 ruler-3: 2025/06/06 00:23:36 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.6887730Z 00:23:36 ruler-3: ts=2025-06-06T00:23:36.097595283Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.6888467Z 00:23:36 Ports for container: e2e-cortex-test-ruler-3 Mapping: map[80:32816 9095:32817]
2025-06-06T00:26:09.6889546Z 00:23:39 ruler-1: ts=2025-06-06T00:23:39.458467454Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:39.458456273Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_22 name=rule_22 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6891239Z 00:23:39 ruler-1: ts=2025-06-06T00:23:39.513038666Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:39.513029809Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_67 name=rule_67 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6892801Z 00:23:39 ruler-1: ts=2025-06-06T00:23:39.60522076Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:39.605205952Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_74 name=rule_74 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6894351Z 00:23:39 ruler-1: ts=2025-06-06T00:23:39.630240413Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:39.630229843Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6895894Z 00:23:39 ruler-1: ts=2025-06-06T00:23:39.729689312Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:39.729681297Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_17 name=rule_17 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6897600Z 00:23:39 ruler-1: ts=2025-06-06T00:23:39.730697572Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:39.730691911Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_77 name=rule_77 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6899156Z 00:23:39 ruler-1: ts=2025-06-06T00:23:39.759894697Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:39.759887363Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_82 name=rule_82 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6901176Z 00:23:39 ruler-1: ts=2025-06-06T00:23:39.812986023Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:39.812979581Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_24 name=rule_24 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6902915Z 00:23:39 ruler-1: ts=2025-06-06T00:23:39.884539697Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:39.884528536Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_57 name=rule_57 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6904614Z 00:23:39 ruler-1: ts=2025-06-06T00:23:39.934349302Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:39.934339664Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_78 name=rule_78 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6906182Z 00:23:39 ruler-1: ts=2025-06-06T00:23:39.982235618Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:39.982226601Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_37 name=rule_37 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6907736Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.014259393Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.014249965Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_58 name=rule_58 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6909289Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.023596945Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.023587959Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_97 name=rule_97 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6910833Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.05110151Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.051093454Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_88 name=rule_88 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6912534Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.135795815Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.135788551Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_92 name=rule_92 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6914088Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.311752448Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.311743091Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_34 name=rule_34 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6915638Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.400876714Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.400866765Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_14 name=rule_14 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6917180Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.402839513Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.402830215Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_0 name=rule_0 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6918854Z 00:23:40 ruler-2: ts=2025-06-06T00:23:40.454776873Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.454767255Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_61 name=rule_61 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6920407Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.458918739Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.458910223Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_22 name=rule_22 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6922068Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.513385037Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.513375008Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_67 name=rule_67 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6923628Z 00:23:40 ruler-2: ts=2025-06-06T00:23:40.549747975Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.549739569Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_95 name=rule_95 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6925176Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.605393365Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.605382975Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_74 name=rule_74 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6926829Z 00:23:40 ruler-2: ts=2025-06-06T00:23:40.60904057Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.609017707Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_30 name=rule_30 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6928383Z 00:23:40 ruler-2: ts=2025-06-06T00:23:40.609031351Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.609017736Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_68 name=rule_68 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6929933Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.630668107Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.630660803Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6931578Z 00:23:40 ruler-2: ts=2025-06-06T00:23:40.709722263Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.709712504Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_53 name=rule_53 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6933126Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.73071599Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.730707143Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_17 name=rule_17 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6934676Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.730887992Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.730879637Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_77 name=rule_77 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6936228Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.760490308Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.760481812Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_82 name=rule_82 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6937776Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.813344895Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.813335498Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_24 name=rule_24 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6939447Z 00:23:40 ruler-2: ts=2025-06-06T00:23:40.830124086Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.830098759Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_87 name=rule_87 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6940985Z 00:23:40 ruler-2: ts=2025-06-06T00:23:40.851595317Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.85158606Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_65 name=rule_65 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6942623Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.884842252Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.8848361Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_57 name=rule_57 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6944160Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.934718011Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.934711048Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_78 name=rule_78 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6945711Z 00:23:40 ruler-2: ts=2025-06-06T00:23:40.950684586Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.950674467Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_91 name=rule_91 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6947365Z 00:23:40 ruler-2: ts=2025-06-06T00:23:40.950683853Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.950673414Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_40 name=rule_40 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6948918Z 00:23:40 ruler-2: ts=2025-06-06T00:23:40.957736311Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.957728276Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_59 name=rule_59 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6950480Z 00:23:40 ruler-2: ts=2025-06-06T00:23:40.967632351Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.967624326Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_99 name=rule_99 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6952147Z 00:23:40 ruler-1: ts=2025-06-06T00:23:40.982597241Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:40.982586891Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_37 name=rule_37 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6953691Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.014671373Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.014664931Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_58 name=rule_58 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6955238Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.024385978Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.024373063Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_97 name=rule_97 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6956784Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.051428425Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.051418607Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_88 name=rule_88 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6958320Z 00:23:41 ruler-2: ts=2025-06-06T00:23:41.069828228Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.0698189Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_80 name=rule_80 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6959852Z 00:23:41 ruler-2: ts=2025-06-06T00:23:41.084144245Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.084137713Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_47 name=rule_47 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6961605Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.136634833Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.136625034Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_92 name=rule_92 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6963157Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.143851571Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.143844087Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_52 name=rule_52 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6964728Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.14589461Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.145889059Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_4 name=rule_4 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6965610Z 00:23:41 Killing ruler-2
2025-06-06T00:26:09.6966508Z 00:23:41 ruler-2: ts=2025-06-06T00:23:41.175910943Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.17590405Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_76 name=rule_76 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6968158Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.22614699Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.226139156Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_62 name=rule_62 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6969703Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.311871708Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.311860076Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_34 name=rule_34 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6970710Z 00:23:41 Error response from daemon: No such container: e2e-cortex-test-ruler-2
2025-06-06T00:26:09.6970997Z 
2025-06-06T00:26:09.6971534Z === RUN   TestRulesPaginationAPIShardingWithAPIRulesBackupEnabled/List_Rule_Groups_-_Equal_number_of_rule_groups_per_page
2025-06-06T00:26:09.6972783Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.352885421Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.352877987Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_69 name=rule_69 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6974330Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.400915067Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.400907142Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_14 name=rule_14 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6975880Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.403012226Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.403005433Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_0 name=rule_0 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6977422Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.455327127Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.455314744Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_71 name=rule_71 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6978974Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.459328757Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.459320641Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_22 name=rule_22 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6980524Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.459364345Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.459355138Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_18 name=rule_18 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6982491Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.490487237Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.490482108Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_63 name=rule_63 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6984057Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.513640059Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.513633637Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_67 name=rule_67 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6985598Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.51745891Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.517450484Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_38 name=rule_38 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6987144Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.524594991Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.524588389Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_48 name=rule_48 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6988816Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.567810849Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.567803926Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_56 name=rule_56 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6990369Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.586820112Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.58681342Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_31 name=rule_31 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6992052Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.590933128Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.590925503Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_39 name=rule_39 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6993601Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.60607773Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.60607245Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_74 name=rule_74 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6995138Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.630181029Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.630175609Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6996681Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.633318226Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.633312656Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_15 name=rule_15 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6998228Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.643408451Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.643401838Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_32 name=rule_32 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.6999769Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.65954692Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.659542251Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_49 name=rule_49 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7001394Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.672455632Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.672450563Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_1 name=rule_1 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7003067Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.687707647Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.687700814Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_79 name=rule_79 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7004618Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.713879863Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.713874362Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_64 name=rule_64 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7006168Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.730685738Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.730678665Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_17 name=rule_17 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7007717Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.730848613Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.730838885Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_77 name=rule_77 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7009351Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.743914644Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.74390731Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_36 name=rule_36 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7010898Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.759010967Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.759004374Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_41 name=rule_41 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7012539Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.76005801Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.760052921Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_82 name=rule_82 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7014078Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.804281104Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.804274492Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_12 name=rule_12 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7015621Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.81341824Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.813410145Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_24 name=rule_24 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7017160Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.867636407Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.867629995Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_21 name=rule_21 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7018710Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.884762309Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.884757029Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_57 name=rule_57 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7020252Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.934770534Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.934764663Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_78 name=rule_78 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7022008Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.936638856Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.936632905Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_28 name=rule_28 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7023736Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.969899794Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.969892921Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_25 name=rule_25 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7025279Z 00:23:41 ruler-1: ts=2025-06-06T00:23:41.983057024Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.983050763Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_37 name=rule_37 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7026819Z 00:23:41 ruler-3: ts=2025-06-06T00:23:41.993891401Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:41.993884287Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_90 name=rule_90 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7028363Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.015200548Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.015195468Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_58 name=rule_58 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7030015Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.024352795Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.024346484Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_97 name=rule_97 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7031709Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.032500052Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.032494131Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_93 name=rule_93 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7033257Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.051209071Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.051202128Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_89 name=rule_89 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7034800Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.051634196Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.051629537Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_88 name=rule_88 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7036350Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.062352096Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.062347026Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_45 name=rule_45 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7037891Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.1359067Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.135898214Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_92 name=rule_92 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7039433Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.143936455Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.143929723Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_52 name=rule_52 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7040984Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.146045385Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.146039024Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_4 name=rule_4 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7042616Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.201042593Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.201034608Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_29 name=rule_29 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7044297Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.226367108Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.226359293Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_62 name=rule_62 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7045841Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.257265959Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.257255659Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_44 name=rule_44 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7047383Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.312683546Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.312676994Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_34 name=rule_34 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7048934Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.316788879Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.316783449Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_66 name=rule_66 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7050479Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.353811101Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.353801704Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_69 name=rule_69 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7052278Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.401069943Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.401063291Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_14 name=rule_14 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7053840Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.403178103Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.403171481Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_0 name=rule_0 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7055378Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.405310311Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.405305642Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_9 name=rule_9 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7056918Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.456320187Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.456313364Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_71 name=rule_71 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7058460Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.458511933Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.458505972Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_22 name=rule_22 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7060000Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.458511894Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.458505883Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_18 name=rule_18 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7061769Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.489740722Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.489733418Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_63 name=rule_63 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7063324Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.513921153Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.513912587Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_67 name=rule_67 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7065031Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.518432229Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.518425607Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_38 name=rule_38 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7066577Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.525558778Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.525552907Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_48 name=rule_48 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7068118Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.568192311Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.568184336Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_56 name=rule_56 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7069649Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.587806064Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.587799392Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_31 name=rule_31 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7071311Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.591270462Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.59126387Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_39 name=rule_39 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7072958Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.60538581Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.6053805Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_74 name=rule_74 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7074499Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.630525949Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.630520529Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7076039Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.632671579Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.632666739Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_15 name=rule_15 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7077582Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.64426464Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.644258458Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_32 name=rule_32 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7079118Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.660397676Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.660393418Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_49 name=rule_49 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7080659Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.671851061Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.671844699Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_1 name=rule_1 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7082292Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.688551667Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.688546618Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_79 name=rule_79 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7083833Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.714767086Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.714761065Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_64 name=rule_64 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7085373Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.730013023Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.73000613Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_17 name=rule_17 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7087033Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.731073308Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.731067678Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_77 name=rule_77 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7088579Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.744233275Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.744228075Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_36 name=rule_36 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7090121Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.75839208Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.758386099Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_41 name=rule_41 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7091756Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.760521733Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.760516373Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_82 name=rule_82 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7093409Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.804722114Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.804715572Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_12 name=rule_12 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7094957Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.813850254Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.813842449Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_24 name=rule_24 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7096500Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.867027239Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.867020686Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_21 name=rule_21 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7098054Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.885185383Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.885177669Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_57 name=rule_57 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7099593Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.934509392Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.934503331Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_78 name=rule_78 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7101273Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.936627206Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.936621496Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_28 name=rule_28 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7102900Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.970648611Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.97064246Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_25 name=rule_25 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7104442Z 00:23:42 ruler-1: ts=2025-06-06T00:23:42.982798696Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.982792424Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_37 name=rule_37 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7105981Z 00:23:42 ruler-3: ts=2025-06-06T00:23:42.994903923Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:42.994898513Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_90 name=rule_90 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7107655Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.014989537Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.014984387Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_58 name=rule_58 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7109194Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.024180945Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.024175585Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_97 name=rule_97 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7110746Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.032307756Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.032302767Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_93 name=rule_93 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7112393Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.051445356Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.051439996Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_88 name=rule_88 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7114050Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.052196001Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.052191163Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_89 name=rule_89 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7115589Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.06334197Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.06333649Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_45 name=rule_45 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7117130Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.136770566Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.136757391Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_92 name=rule_92 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7118667Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.144313667Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.144307576Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_52 name=rule_52 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7120209Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.146424107Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.146417435Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_4 name=rule_4 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7121846Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.201056265Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.201046647Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_29 name=rule_29 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7123400Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.226654584Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.226648312Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_62 name=rule_62 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7124944Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.257251394Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.257243569Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_44 name=rule_44 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7126480Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.289289293Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.289281208Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_43 name=rule_43 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7128135Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.312775933Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.312769421Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_34 name=rule_34 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7129677Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.316857046Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.316852227Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_66 name=rule_66 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7131433Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.317384332Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.31737813Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_26 name=rule_26 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7133005Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.353563126Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.353556093Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_69 name=rule_69 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7134668Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.369746671Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.369740068Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_27 name=rule_27 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7136225Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.374849482Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.37484346Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_98 name=rule_98 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7137755Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.401103901Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.401097068Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_14 name=rule_14 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7139300Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.403214027Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.403208687Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_0 name=rule_0 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7140851Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.405356932Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.405352374Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_9 name=rule_9 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7142674Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.456144134Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.456136279Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_71 name=rule_71 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7144223Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.458546585Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.458540133Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_22 name=rule_22 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7145764Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.458553326Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.458547665Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_18 name=rule_18 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7147299Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.489850167Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.489844817Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_63 name=rule_63 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7148976Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.512963624Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.512956491Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_67 name=rule_67 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7150528Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.518229352Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.518222929Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_38 name=rule_38 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7152194Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.525349712Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.5253433Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_48 name=rule_48 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7153740Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.568148366Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.568143047Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_56 name=rule_56 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7155280Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.587522764Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.587514288Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_31 name=rule_31 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7156939Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.591272884Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.591267384Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_39 name=rule_39 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7158494Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.605425594Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.605419843Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_74 name=rule_74 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7160041Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.606649007Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.606643787Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_8 name=rule_8 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7161702Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.608791567Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.608787139Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_55 name=rule_55 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7163253Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.608829313Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.608820787Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_2 name=rule_2 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7164823Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.61795096Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.617946191Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_72 name=rule_72 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7166362Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.630579203Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.630573823Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7167903Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.632703207Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.632698007Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_15 name=rule_15 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7169566Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.644102171Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.644096921Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_32 name=rule_32 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7171284Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.66024572Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.660240861Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_49 name=rule_49 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7172840Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.671872867Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.671866094Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_1 name=rule_1 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7174540Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.688455722Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.688450292Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_79 name=rule_79 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7176101Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.714604349Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.714597356Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_64 name=rule_64 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7177809Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.730053177Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.730047025Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_17 name=rule_17 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7179363Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.731126916Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.731098162Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_77 name=rule_77 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7180911Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.744257927Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.744253288Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_36 name=rule_36 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7182681Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.75840987Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.75840481Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_41 name=rule_41 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7184216Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.760540294Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.760535836Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_82 name=rule_82 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7185761Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.803841875Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.803834731Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_70 name=rule_70 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7187302Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.804726187Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.804720095Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_12 name=rule_12 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7188842Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.813874125Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.81386627Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_24 name=rule_24 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7190383Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.85192964Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.851923208Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_20 name=rule_20 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7192284Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.867044265Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.867038274Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_21 name=rule_21 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7193842Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.885193084Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.885186662Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_57 name=rule_57 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7195383Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.905092172Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.905086031Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_19 name=rule_19 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7196930Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.934382915Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.934372756Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_78 name=rule_78 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7198597Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.937260251Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.937252537Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_28 name=rule_28 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7200147Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.970516682Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.970510811Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_25 name=rule_25 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7201819Z 00:23:43 ruler-1: ts=2025-06-06T00:23:43.982655301Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.98264948Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_37 name=rule_37 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7203371Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.987430675Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.987423321Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_10 name=rule_10 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7204914Z 00:23:43 ruler-3: ts=2025-06-06T00:23:43.994546059Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:43.994540558Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_90 name=rule_90 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7206453Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.014799488Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.014794138Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_58 name=rule_58 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7207991Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.023956951Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.023951861Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_97 name=rule_97 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7209532Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.032103944Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.032099235Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_93 name=rule_93 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7211169Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.044679722Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.044674512Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_81 name=rule_81 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7212874Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.051249583Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.051243893Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_88 name=rule_88 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7214428Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.051817639Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.051813251Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_89 name=rule_89 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7215968Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.062962636Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.062957537Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_45 name=rule_45 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7217509Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.093201163Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.093186936Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_54 name=rule_54 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7219150Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.116984936Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.116978404Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_7 name=rule_7 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7220686Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.136466853Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.136460852Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_92 name=rule_92 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7222455Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.144132731Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.144125467Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_52 name=rule_52 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7223998Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.146266066Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.146260806Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_4 name=rule_4 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7225538Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.200773778Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.200764822Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_29 name=rule_29 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7227069Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.226541386Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.226534202Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_62 name=rule_62 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7228610Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.257004622Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.256997749Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_44 name=rule_44 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7230147Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.289742698Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.289733401Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_43 name=rule_43 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7231776Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.312587214Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.31257959Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_34 name=rule_34 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7233456Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.316700846Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.316694894Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_66 name=rule_66 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7234993Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.316899798Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.31689Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_26 name=rule_26 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7236530Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.353133485Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.35310379Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_69 name=rule_69 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7238076Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.369211072Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.369205021Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_27 name=rule_27 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7239727Z 00:23:44 ruler-3: ts=2025-06-06T00:23:44.374307731Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.374303002Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_98 name=rule_98 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7241471Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.394383809Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.7243126Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.396188882Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.7244769Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.398428273Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.7246412Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.400768819Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.7248014Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.400816796Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.400807949Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_14 name=rule_14 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7249561Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.402771212Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.402763318Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_0 name=rule_0 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7251267Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.403302949Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.7252861Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.404768007Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.404759812Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_9 name=rule_9 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7254113Z === RUN   TestRulesPaginationAPIShardingWithAPIRulesBackupEnabled/List_Rule_Groups_-_Last_page_unequal
2025-06-06T00:26:09.7255341Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.405545614Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.7256974Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.408746512Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.7258086Z === RUN   TestRulesPaginationAPIShardingWithAPIRulesBackupEnabled/List_all_rule_groups
2025-06-06T00:26:09.7259251Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.410832651Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.7260456Z === RUN   TestRulesPaginationAPIShardingWithAPIRulesBackupEnabled/List_all_rule_groups_-_Max_Rule_Groups_>_Actual
2025-06-06T00:26:09.7262040Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.413333721Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.5:9095: connect: no route to host\""
2025-06-06T00:26:09.7262978Z 00:23:44 Killing ruler-3
2025-06-06T00:26:09.7263889Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.458513417Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.458502707Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_22 name=rule_22 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7265466Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.45851344Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.4585027Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_18 name=rule_18 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7267007Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.49060268Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.490596008Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_63 name=rule_63 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7268546Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.513739713Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.513730626Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_67 name=rule_67 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7270098Z 00:23:44 ruler-1: ts=2025-06-06T00:23:44.568092662Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:44.568081021Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_56 name=rule_56 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7271195Z 00:23:44 Error response from daemon: No such container: e2e-cortex-test-ruler-3
2025-06-06T00:26:09.7271481Z 
2025-06-06T00:26:09.7271583Z 00:23:44 Killing ruler-1
2025-06-06T00:26:09.7271925Z 00:23:44 Error response from daemon: No such container: e2e-cortex-test-ruler-1
2025-06-06T00:26:09.7272204Z 
2025-06-06T00:26:09.7272305Z 00:23:44 Killing minio-9000
2025-06-06T00:26:09.7272676Z 00:23:44 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.7272959Z 
2025-06-06T00:26:09.7273175Z 00:23:44 Killing consul
2025-06-06T00:26:09.7273518Z 00:23:44 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.7273790Z 
2025-06-06T00:26:09.7274045Z --- PASS: TestRulesPaginationAPIShardingWithAPIRulesBackupEnabled (12.72s)
2025-06-06T00:26:09.7274920Z     --- PASS: TestRulesPaginationAPIShardingWithAPIRulesBackupEnabled/List_Rule_Groups_-_Equal_number_of_rule_groups_per_page (3.08s)
2025-06-06T00:26:09.7275901Z     --- PASS: TestRulesPaginationAPIShardingWithAPIRulesBackupEnabled/List_Rule_Groups_-_Last_page_unequal (0.01s)
2025-06-06T00:26:09.7276741Z     --- PASS: TestRulesPaginationAPIShardingWithAPIRulesBackupEnabled/List_all_rule_groups (0.00s)
2025-06-06T00:26:09.7277631Z     --- PASS: TestRulesPaginationAPIShardingWithAPIRulesBackupEnabled/List_all_rule_groups_-_Max_Rule_Groups_>_Actual (0.00s)
2025-06-06T00:26:09.7278273Z === RUN   TestRulesPaginationAPIWithShardingAndNextToken
2025-06-06T00:26:09.7278596Z 00:23:45 Starting consul
2025-06-06T00:26:09.7278852Z 00:23:45 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.7279141Z 00:23:45 consul: Version: '1.8.4'
2025-06-06T00:26:09.7279458Z 00:23:45 consul: Node ID: '39830187-1f24-bbc3-790f-18f22c1276c9'
2025-06-06T00:26:09.7279787Z 00:23:45 consul: Node name: 'consul'
2025-06-06T00:26:09.7280083Z 00:23:45 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.7280407Z 00:23:45 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.7280803Z 00:23:45 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.7281358Z 00:23:45 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.7281989Z 00:23:45 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.7282493Z 00:23:45 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.7282825Z 00:23:45 consul: ==> Consul agent running!
2025-06-06T00:26:09.7283210Z 00:23:45 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32818]
2025-06-06T00:26:09.7283584Z 00:23:45 Starting minio-9000
2025-06-06T00:26:09.7283949Z 00:23:46 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32819]
2025-06-06T00:26:09.7284322Z 00:23:47 Starting ruler-1
2025-06-06T00:26:09.7284679Z 00:23:47 ruler-1: 2025/06/06 00:23:47 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7285443Z 00:23:47 ruler-1: ts=2025-06-06T00:23:47.877748547Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7286195Z 00:23:48 Ports for container: e2e-cortex-test-ruler-1 Mapping: map[80:32820 9095:32821]
2025-06-06T00:26:09.7286587Z 00:23:48 Starting ruler-2
2025-06-06T00:26:09.7286934Z 00:23:48 ruler-2: 2025/06/06 00:23:48 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7287680Z 00:23:48 ruler-2: ts=2025-06-06T00:23:48.274411006Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7288429Z 00:23:48 Ports for container: e2e-cortex-test-ruler-2 Mapping: map[80:32822 9095:32823]
2025-06-06T00:26:09.7288813Z 00:23:48 Starting ruler-3
2025-06-06T00:26:09.7289155Z 00:23:48 ruler-3: 2025/06/06 00:23:48 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7289892Z 00:23:48 ruler-3: ts=2025-06-06T00:23:48.651826172Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7290629Z 00:23:49 Ports for container: e2e-cortex-test-ruler-3 Mapping: map[80:32824 9095:32825]
2025-06-06T00:26:09.7291829Z 00:23:52 ruler-1: ts=2025-06-06T00:23:52.729405393Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:52.729394051Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_17 name=rule_17 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7293385Z 00:23:52 ruler-1: ts=2025-06-06T00:23:52.786917932Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:52.786908063Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_13 name=rule_13 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7295057Z 00:23:52 ruler-1: ts=2025-06-06T00:23:52.829571396Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:52.82956345Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_42 name=rule_42 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7296599Z 00:23:52 ruler-1: ts=2025-06-06T00:23:52.842293503Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:52.842283845Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_4 name=rule_4 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7298142Z 00:23:52 ruler-1: ts=2025-06-06T00:23:52.861645502Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:52.861635143Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_46 name=rule_46 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7299682Z 00:23:52 ruler-1: ts=2025-06-06T00:23:52.963437027Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:52.963423Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_85 name=rule_85 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7301574Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.007851234Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.007838921Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_97 name=rule_97 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7303162Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.031155249Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.031146131Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_87 name=rule_87 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7304720Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.167784557Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.167776922Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_51 name=rule_51 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7306263Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.230647714Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.230634429Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7307804Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.289742668Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.289731768Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_62 name=rule_62 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7309354Z 00:23:53 ruler-3: ts=2025-06-06T00:23:53.292588014Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.292576152Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_31 name=rule_31 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7310888Z 00:23:53 ruler-3: ts=2025-06-06T00:23:53.315662983Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.315652463Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_66 name=rule_66 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7312543Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.31687962Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.316870964Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_94 name=rule_94 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7314101Z 00:23:53 ruler-3: ts=2025-06-06T00:23:53.348312319Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.3483021Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_91 name=rule_91 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7315375Z === RUN   TestRulesPaginationAPIWithShardingAndNextToken/List_Rule_Groups_-_Equal_number_of_rule_groups_per_page
2025-06-06T00:26:09.7316113Z === RUN   TestRulesPaginationAPIWithShardingAndNextToken/List_Rule_Groups_-_Retrieve_page_2_and_3
2025-06-06T00:26:09.7316579Z 00:23:53 Killing ruler-3
2025-06-06T00:26:09.7317492Z 00:23:53 ruler-3: ts=2025-06-06T00:23:53.376692656Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.376681755Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_74 name=rule_74 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7319033Z 00:23:53 ruler-2: ts=2025-06-06T00:23:53.411693697Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.41168435Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_15 name=rule_15 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7320585Z 00:23:53 ruler-2: ts=2025-06-06T00:23:53.414914985Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.414903984Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_24 name=rule_24 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7322383Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.455157179Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.455148833Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_71 name=rule_71 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7323935Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.512316162Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.512305802Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_67 name=rule_67 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7325486Z 00:23:53 ruler-2: ts=2025-06-06T00:23:53.536130009Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.536100704Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_96 name=rule_96 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7326483Z 00:23:53 Error response from daemon: No such container: e2e-cortex-test-ruler-3
2025-06-06T00:26:09.7326767Z 
2025-06-06T00:26:09.7326857Z 00:23:53 Killing ruler-2
2025-06-06T00:26:09.7327761Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.545777686Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.545764471Z msg="Rule sample appending failed" file=/rules/user-1/test3 group=test_16 name=rule_16 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7329307Z 00:23:53 ruler-2: ts=2025-06-06T00:23:53.549203298Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.549195103Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_95 name=rule_95 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7330845Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.585079271Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.585070243Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_9 name=rule_9 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7332514Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.609142329Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.609136068Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_30 name=rule_30 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7334052Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.632383349Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.632377028Z msg="Rule sample appending failed" file=/rules/user-1/test4 group=test_41 name=rule_41 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7335717Z 00:23:53 ruler-1: ts=2025-06-06T00:23:53.633461171Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:23:53.633453366Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_98 name=rule_98 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7336708Z 00:23:53 Error response from daemon: No such container: e2e-cortex-test-ruler-2
2025-06-06T00:26:09.7336988Z 
2025-06-06T00:26:09.7337075Z 00:23:53 Killing ruler-1
2025-06-06T00:26:09.7337411Z 00:23:53 Error response from daemon: No such container: e2e-cortex-test-ruler-1
2025-06-06T00:26:09.7337686Z 
2025-06-06T00:26:09.7337779Z 00:23:53 Killing minio-9000
2025-06-06T00:26:09.7338142Z 00:23:53 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.7338434Z 
2025-06-06T00:26:09.7338525Z 00:23:53 Killing consul
2025-06-06T00:26:09.7338860Z 00:23:54 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.7339138Z 
2025-06-06T00:26:09.7339349Z --- PASS: TestRulesPaginationAPIWithShardingAndNextToken (9.12s)
2025-06-06T00:26:09.7340128Z     --- PASS: TestRulesPaginationAPIWithShardingAndNextToken/List_Rule_Groups_-_Equal_number_of_rule_groups_per_page (0.01s)
2025-06-06T00:26:09.7341017Z     --- PASS: TestRulesPaginationAPIWithShardingAndNextToken/List_Rule_Groups_-_Retrieve_page_2_and_3 (0.00s)
2025-06-06T00:26:09.7341766Z === RUN   TestRulesAPIWithNoRules
2025-06-06T00:26:09.7342034Z 00:23:54 Starting consul
2025-06-06T00:26:09.7342288Z 00:23:54 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.7342742Z 00:23:54 consul: Version: '1.8.4'
2025-06-06T00:26:09.7343063Z 00:23:54 consul: Node ID: 'd7d7c2e8-3e10-4bf1-b764-7752664f7414'
2025-06-06T00:26:09.7343389Z 00:23:54 consul: Node name: 'consul'
2025-06-06T00:26:09.7343685Z 00:23:54 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.7344008Z 00:23:54 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.7344408Z 00:23:54 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.7344826Z 00:23:54 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.7345339Z 00:23:54 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.7345841Z 00:23:54 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.7346169Z 00:23:54 consul: ==> Consul agent running!
2025-06-06T00:26:09.7346557Z 00:23:54 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32826]
2025-06-06T00:26:09.7346932Z 00:23:54 Starting minio-9000
2025-06-06T00:26:09.7347294Z 00:23:55 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32827]
2025-06-06T00:26:09.7347668Z 00:23:56 Starting ruler-1
2025-06-06T00:26:09.7348018Z 00:23:56 ruler-1: 2025/06/06 00:23:56 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7348777Z 00:23:56 ruler-1: ts=2025-06-06T00:23:56.834166664Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7349537Z 00:23:57 Ports for container: e2e-cortex-test-ruler-1 Mapping: map[80:32828 9095:32829]
2025-06-06T00:26:09.7349924Z 00:23:57 Starting ruler-2
2025-06-06T00:26:09.7350275Z 00:23:57 ruler-2: 2025/06/06 00:23:57 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7351022Z 00:23:57 ruler-2: ts=2025-06-06T00:23:57.462211156Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7351880Z 00:23:57 Ports for container: e2e-cortex-test-ruler-2 Mapping: map[80:32830 9095:32831]
2025-06-06T00:26:09.7352264Z 00:23:57 Starting ruler-3
2025-06-06T00:26:09.7352606Z 00:23:58 ruler-3: 2025/06/06 00:23:58 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7353350Z 00:23:58 ruler-3: ts=2025-06-06T00:23:58.066936938Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7354210Z 00:23:58 Ports for container: e2e-cortex-test-ruler-3 Mapping: map[80:32832 9095:32833]
2025-06-06T00:26:09.7354674Z === RUN   TestRulesAPIWithNoRules/List_Rule_Groups_With_Filter
2025-06-06T00:26:09.7355102Z === RUN   TestRulesAPIWithNoRules/List_All_Rule_Groups_With_No_Filter
2025-06-06T00:26:09.7355454Z 00:24:03 Killing ruler-3
2025-06-06T00:26:09.7355798Z 00:24:03 Error response from daemon: No such container: e2e-cortex-test-ruler-3
2025-06-06T00:26:09.7356075Z 
2025-06-06T00:26:09.7356173Z 00:24:03 Killing ruler-2
2025-06-06T00:26:09.7356512Z 00:24:03 Error response from daemon: No such container: e2e-cortex-test-ruler-2
2025-06-06T00:26:09.7356782Z 
2025-06-06T00:26:09.7356873Z 00:24:03 Killing ruler-1
2025-06-06T00:26:09.7357204Z 00:24:03 Error response from daemon: No such container: e2e-cortex-test-ruler-1
2025-06-06T00:26:09.7357475Z 
2025-06-06T00:26:09.7357571Z 00:24:03 Killing minio-9000
2025-06-06T00:26:09.7357935Z 00:24:04 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.7358220Z 
2025-06-06T00:26:09.7358312Z 00:24:04 Killing consul
2025-06-06T00:26:09.7358648Z 00:24:04 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.7358922Z 
2025-06-06T00:26:09.7359052Z --- PASS: TestRulesAPIWithNoRules (10.14s)
2025-06-06T00:26:09.7359505Z     --- PASS: TestRulesAPIWithNoRules/List_Rule_Groups_With_Filter (0.00s)
2025-06-06T00:26:09.7360072Z     --- PASS: TestRulesAPIWithNoRules/List_All_Rule_Groups_With_No_Filter (0.00s)
2025-06-06T00:26:09.7360589Z === RUN   TestRulerAlertmanager
2025-06-06T00:26:09.7360841Z 00:24:04 Starting consul
2025-06-06T00:26:09.7361218Z 00:24:04 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.7361507Z 00:24:04 consul: Version: '1.8.4'
2025-06-06T00:26:09.7361821Z 00:24:04 consul: Node ID: '7dc0a5fa-1d5a-9812-ce40-119b89ddf21b'
2025-06-06T00:26:09.7362147Z 00:24:04 consul: Node name: 'consul'
2025-06-06T00:26:09.7362448Z 00:24:04 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.7362771Z 00:24:04 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.7363164Z 00:24:04 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.7363584Z 00:24:04 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.7364094Z 00:24:04 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.7364619Z 00:24:04 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.7364954Z 00:24:04 consul: ==> Consul agent running!
2025-06-06T00:26:09.7365339Z 00:24:05 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32834]
2025-06-06T00:26:09.7365705Z 00:24:05 Starting minio-9000
2025-06-06T00:26:09.7366073Z 00:24:05 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32835]
2025-06-06T00:26:09.7366448Z 00:24:06 Starting alertmanager1
2025-06-06T00:26:09.7366839Z 00:24:07 alertmanager1: 2025/06/06 00:24:07 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7367441Z 00:24:07 Ports for container: e2e-cortex-test-alertmanager1 Mapping: map[80:32836 9094:32837 9095:32838]
2025-06-06T00:26:09.7367891Z 00:24:07 Starting alertmanager2
2025-06-06T00:26:09.7368271Z 00:24:07 alertmanager2: 2025/06/06 00:24:07 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7368862Z 00:24:07 Ports for container: e2e-cortex-test-alertmanager2 Mapping: map[80:32839 9094:32840 9095:32841]
2025-06-06T00:26:09.7369303Z 00:24:07 Starting ruler
2025-06-06T00:26:09.7369640Z 00:24:08 ruler: 2025/06/06 00:24:08 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7370394Z 00:24:08 ruler: ts=2025-06-06T00:24:08.166857463Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7371255Z 00:24:08 Ports for container: e2e-cortex-test-ruler Mapping: map[80:32842 9095:32843]
2025-06-06T00:26:09.7371763Z 00:24:15 Killing ruler
2025-06-06T00:26:09.7372100Z 00:24:15 Error response from daemon: No such container: e2e-cortex-test-ruler
2025-06-06T00:26:09.7372375Z 
2025-06-06T00:26:09.7372475Z 00:24:15 Killing alertmanager2
2025-06-06T00:26:09.7372853Z 00:24:15 Error response from daemon: No such container: e2e-cortex-test-alertmanager2
2025-06-06T00:26:09.7373147Z 
2025-06-06T00:26:09.7373248Z 00:24:15 Killing alertmanager1
2025-06-06T00:26:09.7373617Z 00:24:16 Error response from daemon: No such container: e2e-cortex-test-alertmanager1
2025-06-06T00:26:09.7373909Z 
2025-06-06T00:26:09.7374006Z 00:24:16 Killing minio-9000
2025-06-06T00:26:09.7374365Z 00:24:16 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.7374649Z 
2025-06-06T00:26:09.7374742Z 00:24:16 Killing consul
2025-06-06T00:26:09.7375079Z 00:24:16 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.7375355Z 
2025-06-06T00:26:09.7375478Z --- PASS: TestRulerAlertmanager (12.01s)
2025-06-06T00:26:09.7375777Z === RUN   TestRulerAlertmanagerTLS
2025-06-06T00:26:09.7376033Z 00:24:16 Starting consul
2025-06-06T00:26:09.7376284Z 00:24:16 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.7376568Z 00:24:16 consul: Version: '1.8.4'
2025-06-06T00:26:09.7376883Z 00:24:16 consul: Node ID: '1ea3139d-cbac-5e03-4472-f11653bc6009'
2025-06-06T00:26:09.7377214Z 00:24:16 consul: Node name: 'consul'
2025-06-06T00:26:09.7377509Z 00:24:16 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.7377946Z 00:24:16 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.7378345Z 00:24:16 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.7378765Z 00:24:16 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.7379272Z 00:24:16 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.7379774Z 00:24:16 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.7380104Z 00:24:16 consul: ==> Consul agent running!
2025-06-06T00:26:09.7380485Z 00:24:16 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32844]
2025-06-06T00:26:09.7380861Z 00:24:16 Starting minio-9000
2025-06-06T00:26:09.7381425Z 00:24:17 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32845]
2025-06-06T00:26:09.7381817Z 00:24:18 Starting alertmanager1
2025-06-06T00:26:09.7382204Z 00:24:19 alertmanager1: 2025/06/06 00:24:19 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7382807Z 00:24:19 Ports for container: e2e-cortex-test-alertmanager1 Mapping: map[80:32846 9094:32847 9095:32848]
2025-06-06T00:26:09.7383252Z 00:24:19 Starting ruler
2025-06-06T00:26:09.7383616Z 00:24:19 alertmanager1: 2025/06/06 00:24:19 http: TLS handshake error from 172.18.0.1:34836: EOF
2025-06-06T00:26:09.7384114Z 00:24:19 ruler: 2025/06/06 00:24:19 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7384871Z 00:24:19 ruler: ts=2025-06-06T00:24:19.525419058Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7385601Z 00:24:19 Ports for container: e2e-cortex-test-ruler Mapping: map[80:32849 9095:32850]
2025-06-06T00:26:09.7386888Z 00:24:21 ruler: ts=2025-06-06T00:24:21.830298461Z caller=handler.go:87 level=warn user=user-1 caller=group.go:551 time=2025-06-06T00:24:21.830288392Z msg="Evaluating rule failed" file=/rules/user-1/test_%2Fencoded_+namespace%2F%3F group="test_encoded_+\"+group_name/?" name=test_rule index=0 rule="record: test_rule\nexpr: up\n" err="empty ring"
2025-06-06T00:26:09.7387959Z 00:24:26 Killing ruler
2025-06-06T00:26:09.7388295Z 00:24:26 Error response from daemon: No such container: e2e-cortex-test-ruler
2025-06-06T00:26:09.7388574Z 
2025-06-06T00:26:09.7388678Z 00:24:26 Killing alertmanager1
2025-06-06T00:26:09.7389051Z 00:24:26 Error response from daemon: No such container: e2e-cortex-test-alertmanager1
2025-06-06T00:26:09.7389478Z 
2025-06-06T00:26:09.7389575Z 00:24:26 Killing minio-9000
2025-06-06T00:26:09.7389936Z 00:24:27 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.7390222Z 
2025-06-06T00:26:09.7390314Z 00:24:27 Killing consul
2025-06-06T00:26:09.7390646Z 00:24:27 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.7390928Z 
2025-06-06T00:26:09.7391164Z --- PASS: TestRulerAlertmanagerTLS (10.91s)
2025-06-06T00:26:09.7391518Z === RUN   TestRulerMetricsForInvalidQueries
2025-06-06T00:26:09.7391806Z 00:24:27 Starting consul
2025-06-06T00:26:09.7392057Z 00:24:27 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.7392335Z 00:24:27 consul: Version: '1.8.4'
2025-06-06T00:26:09.7392647Z 00:24:27 consul: Node ID: '6b62c77e-4223-eb25-5408-b9078659558d'
2025-06-06T00:26:09.7392971Z 00:24:27 consul: Node name: 'consul'
2025-06-06T00:26:09.7393265Z 00:24:27 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.7393592Z 00:24:27 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.7393989Z 00:24:27 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.7394410Z 00:24:27 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.7394920Z 00:24:27 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.7395420Z 00:24:27 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.7395746Z 00:24:27 consul: ==> Consul agent running!
2025-06-06T00:26:09.7396254Z 00:24:27 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32851]
2025-06-06T00:26:09.7396635Z 00:24:27 Starting minio-9000
2025-06-06T00:26:09.7397006Z 00:24:28 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32852]
2025-06-06T00:26:09.7397387Z 00:24:29 Starting distributor
2025-06-06T00:26:09.7397775Z 00:24:30 distributor: 2025/06/06 00:24:30 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7398330Z 00:24:30 Ports for container: e2e-cortex-test-distributor Mapping: map[80:32853 9095:32854]
2025-06-06T00:26:09.7398740Z 00:24:30 Starting ingester
2025-06-06T00:26:09.7399102Z 00:24:30 ingester: 2025/06/06 00:24:30 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7399621Z 00:24:31 Ports for container: e2e-cortex-test-ingester Mapping: map[80:32855 9095:32856]
2025-06-06T00:26:09.7400008Z 00:24:31 Starting ruler
2025-06-06T00:26:09.7400344Z 00:24:31 ruler: 2025/06/06 00:24:31 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7401231Z 00:24:31 ruler: ts=2025-06-06T00:24:31.24737657Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7401982Z 00:24:31 Ports for container: e2e-cortex-test-ruler Mapping: map[80:32857 9095:32858]
2025-06-06T00:26:09.7402437Z === RUN   TestRulerMetricsForInvalidQueries/invalid_group
2025-06-06T00:26:09.7403726Z 00:24:33 ruler: ts=2025-06-06T00:24:33.768694377Z caller=handler.go:87 level=warn user=user caller=group.go:551 time=2025-06-06T00:24:33.768683176Z msg="Evaluating rule failed" file=/rules/user/test group=invalid_group name=rule index=0 rule="record: rule\nexpr: label_replace(metric, \"foo\", \"$1\", \"service\", \"[\")\n" err="invalid regular expression in label_replace(): ["
2025-06-06T00:26:09.7405166Z 00:24:33 ruler: ts=2025-06-06T00:24:33.768870766Z caller=loader.go:113 level=warn msg="unable to read bucket index status" user=user err="context canceled"
2025-06-06T00:26:09.7406086Z 00:24:33 ruler: ts=2025-06-06T00:24:33.768940347Z caller=loader.go:125 level=warn org_id=user msg="received context error when reading bucket index" err="context canceled"
2025-06-06T00:26:09.7407032Z 00:24:35 ruler: ts=2025-06-06T00:24:35.252246077Z caller=notifier.go:64 level=error user=user msg="error starting notifier discovery manager" err="context canceled"
2025-06-06T00:26:09.7407835Z === RUN   TestRulerMetricsForInvalidQueries/too_many_chunks_group
2025-06-06T00:26:09.7408401Z 00:24:39 ruler: ts=2025-06-06T00:24:39.096896522Z caller=loader.go:130 level=warn msg="bucket index not found" user=user
2025-06-06T00:26:09.7409758Z 00:24:39 ruler: ts=2025-06-06T00:24:39.0977016Z caller=handler.go:87 level=warn user=user caller=group.go:551 time=2025-06-06T00:24:39.097691301Z msg="Evaluating rule failed" file=/rules/user/test group=too_many_chunks_group name=rule index=0 rule="record: rule\nexpr: sum(metric)\n" err="the query hit the max number of chunks limit (limit: 5 chunks)"
2025-06-06T00:26:09.7411868Z 00:24:41 ruler: ts=2025-06-06T00:24:41.095942228Z caller=handler.go:87 level=warn user=user caller=group.go:551 time=2025-06-06T00:24:41.095928152Z msg="Evaluating rule failed" file=/rules/user/test group=too_many_chunks_group name=rule index=0 rule="record: rule\nexpr: sum(metric)\n" err="the query hit the max number of chunks limit (limit: 5 chunks)"
2025-06-06T00:26:09.7413311Z 00:24:41 ruler: ts=2025-06-06T00:24:41.251881454Z caller=notifier.go:64 level=error user=user msg="error starting notifier discovery manager" err="context canceled"
2025-06-06T00:26:09.7413950Z === RUN   TestRulerMetricsForInvalidQueries/real_error
2025-06-06T00:26:09.7414268Z 00:24:44 Stopping ingester
2025-06-06T00:26:09.7414627Z 00:24:44 Error response from daemon: No such container: e2e-cortex-test-ingester
2025-06-06T00:26:09.7414912Z 
2025-06-06T00:26:09.7415879Z 00:24:46 ruler: ts=2025-06-06T00:24:46.550991062Z caller=handler.go:87 level=warn user=user caller=group.go:551 time=2025-06-06T00:24:46.550980222Z msg="Evaluating rule failed" file=/rules/user/test group=good_rule name=rule index=0 rule="record: rule\nexpr: sum(metric{foo=~\"1|2\"})\n" err="empty ring"
2025-06-06T00:26:09.7416841Z 00:24:46 Killing ruler
2025-06-06T00:26:09.7417179Z 00:24:47 Error response from daemon: No such container: e2e-cortex-test-ruler
2025-06-06T00:26:09.7417457Z 
2025-06-06T00:26:09.7417552Z 00:24:47 Killing distributor
2025-06-06T00:26:09.7417927Z 00:24:47 Error response from daemon: No such container: e2e-cortex-test-distributor
2025-06-06T00:26:09.7418220Z 
2025-06-06T00:26:09.7418313Z 00:24:47 Killing minio-9000
2025-06-06T00:26:09.7418673Z 00:24:47 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.7418960Z 
2025-06-06T00:26:09.7419049Z 00:24:47 Killing consul
2025-06-06T00:26:09.7419383Z 00:24:47 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.7419662Z 
2025-06-06T00:26:09.7419813Z --- PASS: TestRulerMetricsForInvalidQueries (20.32s)
2025-06-06T00:26:09.7420296Z     --- PASS: TestRulerMetricsForInvalidQueries/invalid_group (4.02s)
2025-06-06T00:26:09.7420843Z     --- PASS: TestRulerMetricsForInvalidQueries/too_many_chunks_group (5.77s)
2025-06-06T00:26:09.7421598Z     --- PASS: TestRulerMetricsForInvalidQueries/real_error (5.55s)
2025-06-06T00:26:09.7421984Z === RUN   TestRulerMetricsWhenIngesterFails
2025-06-06T00:26:09.7422279Z 00:24:47 Starting consul
2025-06-06T00:26:09.7422531Z 00:24:47 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.7422820Z 00:24:47 consul: Version: '1.8.4'
2025-06-06T00:26:09.7423137Z 00:24:47 consul: Node ID: 'a061c142-8a93-1b4f-35a7-8ff61798d69d'
2025-06-06T00:26:09.7423463Z 00:24:47 consul: Node name: 'consul'
2025-06-06T00:26:09.7423761Z 00:24:47 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.7424089Z 00:24:47 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.7424485Z 00:24:47 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.7424921Z 00:24:47 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.7425437Z 00:24:47 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.7425946Z 00:24:47 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.7426279Z 00:24:47 consul: ==> Consul agent running!
2025-06-06T00:26:09.7426805Z 00:24:48 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32859]
2025-06-06T00:26:09.7427175Z 00:24:48 Starting minio-9000
2025-06-06T00:26:09.7427541Z 00:24:48 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32860]
2025-06-06T00:26:09.7427927Z 00:24:50 Starting distributor
2025-06-06T00:26:09.7428314Z 00:24:50 distributor: 2025/06/06 00:24:50 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7428868Z 00:24:50 Ports for container: e2e-cortex-test-distributor Mapping: map[80:32861 9095:32862]
2025-06-06T00:26:09.7429287Z 00:24:50 Starting ingester
2025-06-06T00:26:09.7429648Z 00:24:50 ingester: 2025/06/06 00:24:50 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7430165Z 00:24:51 Ports for container: e2e-cortex-test-ingester Mapping: map[80:32863 9095:32864]
2025-06-06T00:26:09.7430559Z 00:24:51 Starting ruler
2025-06-06T00:26:09.7430895Z 00:24:51 ruler: 2025/06/06 00:24:51 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7431781Z 00:24:51 ruler: ts=2025-06-06T00:24:51.229652013Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7432523Z 00:24:51 Ports for container: e2e-cortex-test-ruler Mapping: map[80:32865 9095:32866]
2025-06-06T00:26:09.7432921Z 00:24:51 Starting store-gateway-1
2025-06-06T00:26:09.7433479Z 00:24:51 store-gateway-1: 2025/06/06 00:24:51 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7434056Z 00:24:52 Ports for container: e2e-cortex-test-store-gateway-1 Mapping: map[80:32867 9095:32868]
2025-06-06T00:26:09.7434671Z === RUN   TestRulerMetricsWhenIngesterFails/real_error
2025-06-06T00:26:09.7435003Z 00:25:01 Stopping ingester
2025-06-06T00:26:09.7435361Z 00:25:02 Error response from daemon: No such container: e2e-cortex-test-ingester
2025-06-06T00:26:09.7435646Z 
2025-06-06T00:26:09.7436428Z 00:25:02 ruler: ts=2025-06-06T00:25:02.550719761Z caller=handler.go:87 level=warn user=user caller=group.go:577 time=2025-06-06T00:25:02.550712498Z msg="Rule sample appending failed" file=/rules/user/test group=good_rule name=rule index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7437330Z 00:25:02 Killing store-gateway-1
2025-06-06T00:26:09.7437726Z 00:25:02 Error response from daemon: No such container: e2e-cortex-test-store-gateway-1
2025-06-06T00:26:09.7438035Z 
2025-06-06T00:26:09.7438120Z 00:25:02 Killing ruler
2025-06-06T00:26:09.7438452Z 00:25:02 Error response from daemon: No such container: e2e-cortex-test-ruler
2025-06-06T00:26:09.7438732Z 
2025-06-06T00:26:09.7438826Z 00:25:02 Killing distributor
2025-06-06T00:26:09.7439201Z 00:25:03 Error response from daemon: No such container: e2e-cortex-test-distributor
2025-06-06T00:26:09.7439491Z 
2025-06-06T00:26:09.7439590Z 00:25:03 Killing minio-9000
2025-06-06T00:26:09.7439949Z 00:25:03 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.7440233Z 
2025-06-06T00:26:09.7440327Z 00:25:03 Killing consul
2025-06-06T00:26:09.7440681Z 00:25:03 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.7440955Z 
2025-06-06T00:26:09.7441240Z --- PASS: TestRulerMetricsWhenIngesterFails (15.93s)
2025-06-06T00:26:09.7441706Z     --- PASS: TestRulerMetricsWhenIngesterFails/real_error (8.33s)
2025-06-06T00:26:09.7442077Z === RUN   TestRulerDisablesRuleGroups
2025-06-06T00:26:09.7442350Z 00:25:03 Starting consul
2025-06-06T00:26:09.7442605Z 00:25:03 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.7442893Z 00:25:03 consul: Version: '1.8.4'
2025-06-06T00:26:09.7443215Z 00:25:03 consul: Node ID: 'ac59aa80-c511-4558-28ab-efd17dff11f1'
2025-06-06T00:26:09.7443544Z 00:25:03 consul: Node name: 'consul'
2025-06-06T00:26:09.7443835Z 00:25:03 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.7444161Z 00:25:03 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.7444557Z 00:25:03 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.7445108Z 00:25:03 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.7445621Z 00:25:03 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.7446124Z 00:25:03 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.7446456Z 00:25:03 consul: ==> Consul agent running!
2025-06-06T00:26:09.7446840Z 00:25:04 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32869]
2025-06-06T00:26:09.7447218Z 00:25:04 Starting minio-9000
2025-06-06T00:26:09.7447585Z 00:25:04 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32870]
2025-06-06T00:26:09.7447972Z 00:25:08 Starting distributor
2025-06-06T00:26:09.7448359Z 00:25:08 distributor: 2025/06/06 00:25:08 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7448910Z 00:25:08 Ports for container: e2e-cortex-test-distributor Mapping: map[80:32871 9095:32872]
2025-06-06T00:26:09.7449322Z 00:25:08 Starting ingester
2025-06-06T00:26:09.7449684Z 00:25:08 ingester: 2025/06/06 00:25:08 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7450196Z 00:25:09 Ports for container: e2e-cortex-test-ingester Mapping: map[80:32873 9095:32874]
2025-06-06T00:26:09.7450586Z 00:25:09 Starting ruler
2025-06-06T00:26:09.7450918Z 00:25:09 ruler: 2025/06/06 00:25:09 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7451779Z 00:25:09 ruler: ts=2025-06-06T00:25:09.305194746Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7452636Z 00:25:09 Ports for container: e2e-cortex-test-ruler Mapping: map[80:32875 9095:32876]
2025-06-06T00:26:09.7453034Z 00:25:09 Starting store-gateway-1
2025-06-06T00:26:09.7453444Z 00:25:09 store-gateway-1: 2025/06/06 00:25:09 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7454013Z 00:25:10 Ports for container: e2e-cortex-test-store-gateway-1 Mapping: map[80:32877 9095:32878]
2025-06-06T00:26:09.7454503Z === RUN   TestRulerDisablesRuleGroups/disable_rule_group
2025-06-06T00:26:09.7454830Z 00:25:13 Killing store-gateway-1
2025-06-06T00:26:09.7455066Z 00:25:13 Error response from daemon: No such container: e2e-cortex-test-store-gateway-1
2025-06-06T00:26:09.7455071Z 
2025-06-06T00:26:09.7455162Z 00:25:13 Killing ruler
2025-06-06T00:26:09.7455368Z 00:25:13 Error response from daemon: No such container: e2e-cortex-test-ruler
2025-06-06T00:26:09.7455373Z 
2025-06-06T00:26:09.7455469Z 00:25:13 Killing ingester
2025-06-06T00:26:09.7455683Z 00:25:13 Error response from daemon: No such container: e2e-cortex-test-ingester
2025-06-06T00:26:09.7455688Z 
2025-06-06T00:26:09.7455787Z 00:25:13 Killing distributor
2025-06-06T00:26:09.7456007Z 00:25:13 Error response from daemon: No such container: e2e-cortex-test-distributor
2025-06-06T00:26:09.7456012Z 
2025-06-06T00:26:09.7456113Z 00:25:13 Killing minio-9000
2025-06-06T00:26:09.7456327Z 00:25:14 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.7456336Z 
2025-06-06T00:26:09.7456429Z 00:25:14 Killing consul
2025-06-06T00:26:09.7456635Z 00:25:14 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.7456645Z 
2025-06-06T00:26:09.7456780Z --- PASS: TestRulerDisablesRuleGroups (10.78s)
2025-06-06T00:26:09.7457025Z     --- PASS: TestRulerDisablesRuleGroups/disable_rule_group (0.86s)
2025-06-06T00:26:09.7457134Z === RUN   TestRulerHAEvaluation
2025-06-06T00:26:09.7457228Z 00:25:14 Starting consul
2025-06-06T00:26:09.7457349Z 00:25:14 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.7457451Z 00:25:14 consul: Version: '1.8.4'
2025-06-06T00:26:09.7457618Z 00:25:14 consul: Node ID: '4583cb8b-982e-3f7f-c88e-f7def78e8504'
2025-06-06T00:26:09.7457716Z 00:25:14 consul: Node name: 'consul'
2025-06-06T00:26:09.7457852Z 00:25:14 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.7457975Z 00:25:14 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.7458302Z 00:25:14 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.7458453Z 00:25:14 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.7458751Z 00:25:14 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.7458900Z 00:25:14 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.7459016Z 00:25:14 consul: ==> Consul agent running!
2025-06-06T00:26:09.7459222Z 00:25:14 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32879]
2025-06-06T00:26:09.7459329Z 00:25:14 Starting minio-9000
2025-06-06T00:26:09.7459544Z 00:25:15 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32880]
2025-06-06T00:26:09.7459641Z 00:25:16 Starting ruler-1
2025-06-06T00:26:09.7459855Z 00:25:17 ruler-1: 2025/06/06 00:25:17 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7460346Z 00:25:17 ruler-1: ts=2025-06-06T00:25:17.080819257Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7460583Z 00:25:17 Ports for container: e2e-cortex-test-ruler-1 Mapping: map[80:32881 9095:32882]
2025-06-06T00:26:09.7460675Z 00:25:17 Starting ruler-2
2025-06-06T00:26:09.7460884Z 00:25:17 ruler-2: 2025/06/06 00:25:17 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7461584Z 00:25:17 ruler-2: ts=2025-06-06T00:25:17.667081635Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7461946Z 00:25:17 Ports for container: e2e-cortex-test-ruler-2 Mapping: map[80:32883 9095:32884]
2025-06-06T00:26:09.7462043Z 00:25:17 Starting ruler-3
2025-06-06T00:26:09.7462255Z 00:25:18 ruler-3: 2025/06/06 00:25:18 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7462727Z 00:25:18 ruler-3: ts=2025-06-06T00:25:18.103130562Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7462960Z 00:25:18 Ports for container: e2e-cortex-test-ruler-3 Mapping: map[80:32885 9095:32886]
2025-06-06T00:26:09.7463750Z 00:25:22 ruler-1: ts=2025-06-06T00:25:22.160729864Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:22.160718472Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_14 name=rule_14 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7464556Z 00:25:22 ruler-1: ts=2025-06-06T00:25:22.194796108Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:22.194787522Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_3 name=rule_3 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7465327Z 00:25:22 ruler-1: ts=2025-06-06T00:25:22.230152666Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:22.230141435Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7466097Z 00:25:22 ruler-1: ts=2025-06-06T00:25:22.399810018Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:22.399802094Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_1 name=rule_1 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7466192Z 00:25:22 Killing consul
2025-06-06T00:26:09.7467022Z 00:25:22 ruler-3: ts=2025-06-06T00:25:22.750524564Z caller=client.go:283 level=error msg="error getting path" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring?index=18&stale=&wait=10000ms\": dial tcp 172.18.0.2:8500: connect: connection refused"
2025-06-06T00:26:09.7467855Z 00:25:22 ruler-3: ts=2025-06-06T00:25:22.750589716Z caller=client.go:283 level=error msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=18&stale=&wait=10000ms\": dial tcp 172.18.0.2:8500: connect: connection refused"
2025-06-06T00:26:09.7468900Z 00:25:22 ruler-3: ts=2025-06-06T00:25:22.750764413Z caller=client.go:283 level=error msg="error getting path" key=collectors/store-gateway err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/store-gateway?index=18&stale=&wait=10000ms\": dial tcp 172.18.0.2:8500: connect: connection refused"
2025-06-06T00:26:09.7469800Z 00:25:22 ruler-2: ts=2025-06-06T00:25:22.751158694Z caller=client.go:283 level=error msg="error getting path" key=collectors/store-gateway err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/store-gateway?index=18&stale=&wait=10000ms\": dial tcp 172.18.0.2:8500: connect: connection refused"
2025-06-06T00:26:09.7470584Z 00:25:22 ruler-2: ts=2025-06-06T00:25:22.7512057Z caller=client.go:283 level=error msg="error getting path" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring?index=18&stale=&wait=10000ms\": dial tcp 172.18.0.2:8500: connect: connection refused"
2025-06-06T00:26:09.7471509Z 00:25:22 ruler-2: ts=2025-06-06T00:25:22.750445678Z caller=client.go:283 level=error msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=18&stale=&wait=10000ms\": dial tcp 172.18.0.2:8500: connect: connection refused"
2025-06-06T00:26:09.7472440Z 00:25:22 ruler-1: ts=2025-06-06T00:25:22.75165184Z caller=client.go:283 level=error msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=18&stale=&wait=10000ms\": dial tcp 172.18.0.2:8500: connect: connection refused"
2025-06-06T00:26:09.7473331Z 00:25:22 ruler-1: ts=2025-06-06T00:25:22.751702273Z caller=client.go:283 level=error msg="error getting path" key=collectors/store-gateway err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/store-gateway?index=18&stale=&wait=10000ms\": dial tcp 172.18.0.2:8500: connect: connection refused"
2025-06-06T00:26:09.7474123Z 00:25:22 ruler-1: ts=2025-06-06T00:25:22.751744944Z caller=client.go:283 level=error msg="error getting path" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring?index=18&stale=&wait=10000ms\": dial tcp 172.18.0.2:8500: connect: connection refused"
2025-06-06T00:26:09.7474341Z 00:25:22 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.7474346Z 
2025-06-06T00:26:09.7474436Z 00:25:22 Killing ruler-2
2025-06-06T00:26:09.7474652Z 00:25:22 Error response from daemon: No such container: e2e-cortex-test-ruler-2
2025-06-06T00:26:09.7474657Z 
2025-06-06T00:26:09.7475439Z 00:25:23 ruler-1: ts=2025-06-06T00:25:23.25782266Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:23.257810016Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_16 name=rule_16 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7476218Z 00:25:23 ruler-3: ts=2025-06-06T00:25:23.584228992Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:23.584218873Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_9 name=rule_9 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7476993Z 00:25:23 ruler-3: ts=2025-06-06T00:25:23.786536839Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:23.786526851Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_13 name=rule_13 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7477971Z 00:25:23 ruler-1: ts=2025-06-06T00:25:23.876855684Z caller=client.go:283 level=error msg="error getting path" key=collectors/store-gateway err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/store-gateway?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7478833Z 00:25:23 ruler-1: ts=2025-06-06T00:25:23.876854636Z caller=client.go:283 level=error msg="error getting path" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7479836Z 00:25:23 ruler-1: ts=2025-06-06T00:25:23.984014389Z caller=client.go:283 level=error msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7480808Z 00:25:24 ruler-3: ts=2025-06-06T00:25:24.117498373Z caller=client.go:283 level=error msg="error getting path" key=collectors/store-gateway err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/store-gateway?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7481719Z 00:25:24 ruler-1: ts=2025-06-06T00:25:24.160799016Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:24.16079081Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_14 name=rule_14 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7482499Z 00:25:24 ruler-1: ts=2025-06-06T00:25:24.194932897Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:24.194926144Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_3 name=rule_3 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7483488Z 00:25:24 ruler-3: ts=2025-06-06T00:25:24.229753445Z caller=client.go:283 level=error msg="error getting path" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7484279Z 00:25:24 ruler-1: ts=2025-06-06T00:25:24.231100497Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:24.231094586Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7485064Z 00:25:24 ruler-1: ts=2025-06-06T00:25:24.400034788Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:24.400027665Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_1 name=rule_1 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7485983Z 00:25:24 ruler-3: ts=2025-06-06T00:25:24.523811536Z caller=client.go:283 level=error msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7486903Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.258102851Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:25.258085158Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_16 name=rule_16 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7487697Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.585182431Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:25.585175688Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_9 name=rule_9 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7488471Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.633052313Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7489242Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.635919576Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7490009Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.638513005Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7490901Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.642214837Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7491883Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.645587919Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7492651Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.649500482Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7493420Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.653181696Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7494300Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.655617366Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7495067Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.658100941Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7495842Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.661448074Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7496341Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.66149913Z caller=basic_lifecycler.go:471 level=warn msg="failed to heartbeat instance in the ring" ring=ruler err="failed to CAS rulers/ring"
2025-06-06T00:26:09.7497136Z 00:25:25 ruler-3: ts=2025-06-06T00:25:25.786821697Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:25.786814423Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_13 name=rule_13 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7497913Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.852035995Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:25.852029483Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_5 name=rule_5 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7498689Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.861702216Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7499459Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.864174902Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7500227Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.866776867Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7500992Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.86929608Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7502123Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.871377289Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7502899Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.873833796Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7503666Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.87640336Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7504430Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.87892673Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7505313Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.881639237Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7506085Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.883886073Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7506584Z 00:25:25 ruler-1: ts=2025-06-06T00:25:25.883950354Z caller=basic_lifecycler.go:471 level=warn msg="failed to heartbeat instance in the ring" ring=ruler err="failed to CAS rulers/ring"
2025-06-06T00:26:09.7507379Z 00:25:26 ruler-1: ts=2025-06-06T00:25:26.161463242Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:26.161457952Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_14 name=rule_14 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7508152Z 00:25:26 ruler-1: ts=2025-06-06T00:25:26.195649914Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:26.195644113Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_3 name=rule_3 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7508929Z 00:25:26 ruler-1: ts=2025-06-06T00:25:26.230888453Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:26.230883453Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7509708Z 00:25:26 ruler-1: ts=2025-06-06T00:25:26.400222082Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:26.40021567Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_1 name=rule_1 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7510475Z 00:25:26 ruler-1: ts=2025-06-06T00:25:26.54661605Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:26.546604308Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_0 name=rule_0 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7511356Z 00:25:26 ruler-1: ts=2025-06-06T00:25:26.65304824Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:26.653041888Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_2 name=rule_2 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7512442Z 00:25:26 ruler-1: ts=2025-06-06T00:25:26.684896881Z caller=client.go:283 level=error msg="error getting path" key=collectors/store-gateway err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/store-gateway?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7513227Z 00:25:26 ruler-3: ts=2025-06-06T00:25:26.813478614Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:26.813464868Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_19 name=rule_19 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7514006Z 00:25:26 ruler-3: ts=2025-06-06T00:25:26.885647118Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:26.885641507Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_7 name=rule_7 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7514776Z 00:25:26 ruler-1: ts=2025-06-06T00:25:26.964548939Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:26.964542277Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_6 name=rule_6 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7515688Z 00:25:27 ruler-3: ts=2025-06-06T00:25:27.113947004Z caller=client.go:283 level=error msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7516572Z 00:25:27 ruler-1: ts=2025-06-06T00:25:27.258247075Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:27.258240162Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_16 name=rule_16 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7517346Z 00:25:27 ruler-3: ts=2025-06-06T00:25:27.401339503Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:27.401333061Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_4 name=rule_4 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7518123Z 00:25:27 ruler-1: ts=2025-06-06T00:25:27.417419461Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:27.417411847Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_8 name=rule_8 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7518987Z 00:25:27 ruler-1: ts=2025-06-06T00:25:27.489463323Z caller=client.go:283 level=error msg="error getting path" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7519763Z 00:25:27 ruler-3: ts=2025-06-06T00:25:27.584671314Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:27.584654563Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_9 name=rule_9 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7520733Z 00:25:27 ruler-3: ts=2025-06-06T00:25:27.651438156Z caller=client.go:283 level=error msg="error getting path" key=collectors/store-gateway err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/store-gateway?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7521637Z 00:25:27 ruler-1: ts=2025-06-06T00:25:27.729940964Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:27.729935113Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_17 name=rule_17 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7522415Z 00:25:27 ruler-3: ts=2025-06-06T00:25:27.786833506Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:27.78682521Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_13 name=rule_13 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7523310Z 00:25:27 ruler-1: ts=2025-06-06T00:25:27.851574374Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:27.851555048Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_5 name=rule_5 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7524213Z 00:25:27 ruler-1: ts=2025-06-06T00:25:27.894522638Z caller=client.go:283 level=error msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7524994Z 00:25:28 ruler-3: ts=2025-06-06T00:25:28.022973481Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:28.022966608Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_12 name=rule_12 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7525853Z 00:25:28 ruler-3: ts=2025-06-06T00:25:28.042971981Z caller=client.go:283 level=error msg="error getting path" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7526636Z 00:25:28 ruler-3: ts=2025-06-06T00:25:28.081222963Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:28.081213456Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_10 name=rule_10 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7527506Z 00:25:28 ruler-1: ts=2025-06-06T00:25:28.16170221Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:28.161694996Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_14 name=rule_14 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7528274Z 00:25:28 ruler-1: ts=2025-06-06T00:25:28.19481663Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:28.19480609Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_3 name=rule_3 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7529058Z 00:25:28 ruler-1: ts=2025-06-06T00:25:28.230256103Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:28.230250192Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7529829Z 00:25:28 ruler-1: ts=2025-06-06T00:25:28.400610651Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:28.400603488Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_1 name=rule_1 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7530597Z 00:25:28 ruler-1: ts=2025-06-06T00:25:28.546880934Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:28.546875273Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_0 name=rule_0 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7531587Z 00:25:28 ruler-1: ts=2025-06-06T00:25:28.653174967Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:28.653168795Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_2 name=rule_2 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7532375Z 00:25:28 ruler-3: ts=2025-06-06T00:25:28.813742632Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:28.813732403Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_19 name=rule_19 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7533146Z 00:25:28 ruler-3: ts=2025-06-06T00:25:28.88588268Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:28.885875557Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_7 name=rule_7 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7534036Z 00:25:28 ruler-1: ts=2025-06-06T00:25:28.964073794Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:28.96406626Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_6 name=rule_6 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7542116Z     ruler_test.go:1538: [13] <nil>
2025-06-06T00:26:09.7542325Z     ruler_test.go:1539: [7] <nil>
2025-06-06T00:26:09.7543174Z 00:25:29 ruler-1: ts=2025-06-06T00:25:29.258594227Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:29.258587365Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_16 name=rule_16 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7543983Z 00:25:29 ruler-3: ts=2025-06-06T00:25:29.401789373Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:29.401780666Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_4 name=rule_4 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7544776Z 00:25:29 ruler-1: ts=2025-06-06T00:25:29.417883452Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:29.417877431Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_8 name=rule_8 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7545737Z 00:25:29 ruler-3: ts=2025-06-06T00:25:29.585098814Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:29.585093204Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_9 name=rule_9 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7546521Z 00:25:29 ruler-1: ts=2025-06-06T00:25:29.730347963Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:29.730340629Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_17 name=rule_17 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7547286Z 00:25:29 ruler-3: ts=2025-06-06T00:25:29.787575276Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:29.787568102Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_13 name=rule_13 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7548047Z 00:25:29 ruler-1: ts=2025-06-06T00:25:29.851613692Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:29.851600387Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_5 name=rule_5 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7548810Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.023097341Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:30.023089366Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_12 name=rule_12 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7549572Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.082037248Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:30.082025085Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_10 name=rule_10 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7550331Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.161706823Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:30.161700812Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_14 name=rule_14 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7551221Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.194917954Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:30.194913185Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_3 name=rule_3 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7551988Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.231168953Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:30.231162211Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_11 name=rule_11 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7552867Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.400525891Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:30.400520982Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_1 name=rule_1 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7553628Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.54709657Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:30.547089055Z msg="Rule sample appending failed" file=/rules/user-1/test1 group=test_0 name=rule_0 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7554405Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.631578372Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7555173Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.634070579Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7555935Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.636553674Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7556786Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.638856651Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7557546Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.641371502Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7558299Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.646209683Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7559060Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.648856434Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7559815Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.651486496Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7560580Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.652239976Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:30.652230688Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_2 name=rule_2 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7561466Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.654358118Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7562222Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.656883848Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7562721Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.656912672Z caller=basic_lifecycler.go:471 level=warn msg="failed to heartbeat instance in the ring" ring=ruler err="failed to CAS rulers/ring"
2025-06-06T00:26:09.7563675Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.81326501Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:30.813258097Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_19 name=rule_19 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7564467Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.859297981Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7565228Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.862066572Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7565982Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.864638581Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7566741Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.867279273Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7567598Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.869611296Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7568350Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.871991265Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7569107Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.874277652Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7569863Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.876807921Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7570616Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.880317833Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7571472Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.882924717Z caller=client.go:196 level=error msg="error getting key" key=rulers/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/rulers/ring\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7571962Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.882948922Z caller=basic_lifecycler.go:471 level=warn msg="failed to heartbeat instance in the ring" ring=ruler err="failed to CAS rulers/ring"
2025-06-06T00:26:09.7572732Z 00:25:30 ruler-3: ts=2025-06-06T00:25:30.88537727Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:30.885371559Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_7 name=rule_7 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7573490Z 00:25:30 ruler-1: ts=2025-06-06T00:25:30.964258701Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:30.964252129Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_6 name=rule_6 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7574380Z 00:25:31 ruler-1: ts=2025-06-06T00:25:31.258213059Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:31.258207459Z msg="Rule sample appending failed" file=/rules/user-1/test2 group=test_16 name=rule_16 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7575176Z 00:25:31 ruler-1: ts=2025-06-06T00:25:31.316600691Z caller=ruler.go:1391 level=error msg="unable to retrieve rules from ruler" addr=172.18.0.5:9095 err="rpc error: code = DeadlineExceeded desc = received context error while waiting for new LB policy update: context deadline exceeded"
2025-06-06T00:26:09.7575276Z 00:25:31 Killing ruler-3
2025-06-06T00:26:09.7576044Z 00:25:31 ruler-1: ts=2025-06-06T00:25:31.417820144Z caller=handler.go:87 level=warn user=user-1 caller=group.go:577 time=2025-06-06T00:25:31.417813632Z msg="Rule sample appending failed" file=/rules/user-1/test5 group=test_8 name=rule_8 index=0 err="DoBatch: InstancesCount <= 0"
2025-06-06T00:26:09.7577017Z 00:25:31 ruler-1: ts=2025-06-06T00:25:31.465261592Z caller=client.go:283 level=error msg="error getting path" key=collectors/store-gateway err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/store-gateway?index=18&stale=&wait=10000ms\": dial tcp: lookup e2e-cortex-test-consul on 127.0.0.11:53: server misbehaving"
2025-06-06T00:26:09.7577239Z 00:25:31 Error response from daemon: No such container: e2e-cortex-test-ruler-3
2025-06-06T00:26:09.7577247Z 
2025-06-06T00:26:09.7577446Z 00:25:31 Killing ruler-1
2025-06-06T00:26:09.7577665Z 00:25:31 Error response from daemon: No such container: e2e-cortex-test-ruler-1
2025-06-06T00:26:09.7577670Z 
2025-06-06T00:26:09.7577765Z 00:25:31 Killing minio-9000
2025-06-06T00:26:09.7577990Z 00:25:31 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.7577995Z 
2025-06-06T00:26:09.7578128Z --- PASS: TestRulerHAEvaluation (17.57s)
2025-06-06T00:26:09.7578232Z === RUN   TestRulerKeepFiring
2025-06-06T00:26:09.7578323Z 00:25:31 Starting consul
2025-06-06T00:26:09.7578445Z 00:25:32 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.7578544Z 00:25:32 consul: Version: '1.8.4'
2025-06-06T00:26:09.7578713Z 00:25:32 consul: Node ID: '2d2a6666-bdf7-f276-2f70-72ef9232082f'
2025-06-06T00:26:09.7578813Z 00:25:32 consul: Node name: 'consul'
2025-06-06T00:26:09.7578955Z 00:25:32 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.7579078Z 00:25:32 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.7579294Z 00:25:32 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.7579452Z 00:25:32 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.7579749Z 00:25:32 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.7579902Z 00:25:32 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.7580023Z 00:25:32 consul: ==> Consul agent running!
2025-06-06T00:26:09.7580232Z 00:25:32 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32887]
2025-06-06T00:26:09.7580333Z 00:25:32 Starting minio-9000
2025-06-06T00:26:09.7580545Z 00:25:33 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32888]
2025-06-06T00:26:09.7580646Z 00:25:34 Starting distributor
2025-06-06T00:26:09.7580881Z 00:25:34 distributor: 2025/06/06 00:25:34 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7581282Z 00:25:34 Ports for container: e2e-cortex-test-distributor Mapping: map[80:32889 9095:32890]
2025-06-06T00:26:09.7581458Z 00:25:34 Starting ingester
2025-06-06T00:26:09.7581689Z 00:25:35 ingester: 2025/06/06 00:25:35 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7581929Z 00:25:35 Ports for container: e2e-cortex-test-ingester Mapping: map[80:32891 9095:32892]
2025-06-06T00:26:09.7582021Z 00:25:35 Starting ruler
2025-06-06T00:26:09.7582367Z 00:25:35 ruler: 2025/06/06 00:25:35 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7582856Z 00:25:35 ruler: ts=2025-06-06T00:25:35.577729796Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7583086Z 00:25:35 Ports for container: e2e-cortex-test-ruler Mapping: map[80:32893 9095:32894]
2025-06-06T00:26:09.7583173Z 00:25:57 Killing ruler
2025-06-06T00:26:09.7583385Z 00:25:57 Error response from daemon: No such container: e2e-cortex-test-ruler
2025-06-06T00:26:09.7583390Z 
2025-06-06T00:26:09.7583487Z 00:25:57 Killing ingester
2025-06-06T00:26:09.7583701Z 00:25:57 Error response from daemon: No such container: e2e-cortex-test-ingester
2025-06-06T00:26:09.7583706Z 
2025-06-06T00:26:09.7583799Z 00:25:57 Killing distributor
2025-06-06T00:26:09.7584022Z 00:25:57 Error response from daemon: No such container: e2e-cortex-test-distributor
2025-06-06T00:26:09.7584032Z 
2025-06-06T00:26:09.7584124Z 00:25:57 Killing minio-9000
2025-06-06T00:26:09.7584343Z 00:25:57 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.7584348Z 
2025-06-06T00:26:09.7584439Z 00:25:57 Killing consul
2025-06-06T00:26:09.7584650Z 00:25:57 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.7584654Z 
2025-06-06T00:26:09.7584769Z --- PASS: TestRulerKeepFiring (26.25s)
2025-06-06T00:26:09.7584891Z === RUN   TestRulerEvalWithQueryFrontend
2025-06-06T00:26:09.7584985Z 00:25:58 Starting consul
2025-06-06T00:26:09.7585213Z 00:25:58 consul: ==> Starting Consul agent...
2025-06-06T00:26:09.7585315Z 00:25:58 consul: Version: '1.8.4'
2025-06-06T00:26:09.7585479Z 00:25:58 consul: Node ID: '7f8255a2-2fab-c22a-7651-4398d5b00e15'
2025-06-06T00:26:09.7585579Z 00:25:58 consul: Node name: 'consul'
2025-06-06T00:26:09.7585717Z 00:25:58 consul: Datacenter: 'dc1' (Segment: '<all>')
2025-06-06T00:26:09.7585843Z 00:25:58 consul: Server: true (Bootstrap: false)
2025-06-06T00:26:09.7586057Z 00:25:58 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2025-06-06T00:26:09.7586206Z 00:25:58 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2025-06-06T00:26:09.7586501Z 00:25:58 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2025-06-06T00:26:09.7586651Z 00:25:58 consul: ==> Log data will now stream in as it occurs:
2025-06-06T00:26:09.7586765Z 00:25:58 consul: ==> Consul agent running!
2025-06-06T00:26:09.7586970Z 00:25:58 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32895]
2025-06-06T00:26:09.7587077Z 00:25:58 Starting minio-9000
2025-06-06T00:26:09.7587297Z 00:25:58 Ports for container: e2e-cortex-test-minio-9000 Mapping: map[9000:32896]
2025-06-06T00:26:09.7587398Z 00:26:00 Starting distributor
2025-06-06T00:26:09.7587632Z 00:26:00 distributor: 2025/06/06 00:26:00 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7587884Z 00:26:00 Ports for container: e2e-cortex-test-distributor Mapping: map[80:32897 9095:32898]
2025-06-06T00:26:09.7587981Z 00:26:00 Starting ingester
2025-06-06T00:26:09.7588200Z 00:26:01 ingester: 2025/06/06 00:26:01 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7588431Z 00:26:01 Ports for container: e2e-cortex-test-ingester Mapping: map[80:32899 9095:32900]
2025-06-06T00:26:09.7588597Z === RUN   TestRulerEvalWithQueryFrontend/format:protobuf
2025-06-06T00:26:09.7588698Z 00:26:01 Starting query-frontend
2025-06-06T00:26:09.7588951Z 00:26:01 query-frontend: 2025/06/06 00:26:01 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7589212Z 00:26:01 Ports for container: e2e-cortex-test-query-frontend Mapping: map[80:32901 9095:32902]
2025-06-06T00:26:09.7589310Z 00:26:01 Starting querier
2025-06-06T00:26:09.7589522Z 00:26:01 querier: 2025/06/06 00:26:01 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7589754Z 00:26:02 Ports for container: e2e-cortex-test-querier Mapping: map[80:32903 9095:32904]
2025-06-06T00:26:09.7589932Z 00:26:02 Starting ruler
2025-06-06T00:26:09.7590142Z 00:26:02 ruler: 2025/06/06 00:26:02 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7590624Z 00:26:02 ruler: ts=2025-06-06T00:26:02.49502241Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7590848Z 00:26:02 Ports for container: e2e-cortex-test-ruler Mapping: map[80:32905 9095:32906]
2025-06-06T00:26:09.7590937Z 00:26:06 Stopping ruler
2025-06-06T00:26:09.7591513Z 00:26:06 ruler: ts=2025-06-06T00:26:06.476510503Z caller=notifier.go:64 level=error user=user msg="error starting notifier discovery manager" err="context canceled"
2025-06-06T00:26:09.7592299Z 00:26:06 ruler: ts=2025-06-06T00:26:06.476904401Z caller=client.go:283 level=error msg="error getting path" key=collectors/store-gateway err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/store-gateway?index=15&stale=&wait=10000ms\": context canceled"
2025-06-06T00:26:09.7593009Z 00:26:06 ruler: ts=2025-06-06T00:26:06.476903491Z caller=client.go:283 level=error msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=15&stale=&wait=10000ms\": context canceled"
2025-06-06T00:26:09.7593222Z 00:26:06 Error response from daemon: No such container: e2e-cortex-test-ruler
2025-06-06T00:26:09.7593228Z 
2025-06-06T00:26:09.7593329Z 00:26:06 Stopping query-frontend
2025-06-06T00:26:09.7593974Z 00:26:06 query-frontend: ts=2025-06-06T00:26:06.635052972Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=4.705233619s err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.7594544Z 00:26:06 querier: ts=2025-06-06T00:26:06.63540251Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=172.18.0.6:9095 err="rpc error: code = Unknown desc = queue is stopped"
2025-06-06T00:26:09.7595405Z 00:26:06 querier: ts=2025-06-06T00:26:06.6878962Z caller=frontend_processor.go:69 level=error msg="error contacting frontend" address=172.18.0.6:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.6:9095: connect: connection refused\""
2025-06-06T00:26:09.7595641Z 00:26:06 Error response from daemon: No such container: e2e-cortex-test-query-frontend
2025-06-06T00:26:09.7595647Z 
2025-06-06T00:26:09.7595743Z 00:26:06 Stopping querier
2025-06-06T00:26:09.7596933Z 00:26:06 querier: ts=2025-06-06T00:26:06.792040822Z caller=frontend_processor.go:57 level=warn msg="failed to notify querier shutdown to query-frontend" address=172.18.0.6:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.6:9095: connect: connection refused\""
2025-06-06T00:26:09.7597759Z 00:26:06 querier: ts=2025-06-06T00:26:06.792093743Z caller=client.go:283 level=error msg="error getting path" key=collectors/store-gateway err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/store-gateway?index=15&stale=&wait=10000ms\": context canceled"
2025-06-06T00:26:09.7598481Z 00:26:06 querier: ts=2025-06-06T00:26:06.792093626Z caller=client.go:283 level=error msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=15&stale=&wait=10000ms\": context canceled"
2025-06-06T00:26:09.7598695Z 00:26:06 Error response from daemon: No such container: e2e-cortex-test-querier
2025-06-06T00:26:09.7598701Z 
2025-06-06T00:26:09.7598851Z === RUN   TestRulerEvalWithQueryFrontend/format:json
2025-06-06T00:26:09.7598959Z 00:26:06 Starting query-frontend
2025-06-06T00:26:09.7599208Z 00:26:07 query-frontend: 2025/06/06 00:26:07 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7599467Z 00:26:07 Ports for container: e2e-cortex-test-query-frontend Mapping: map[80:32907 9095:32908]
2025-06-06T00:26:09.7599563Z 00:26:07 Starting querier
2025-06-06T00:26:09.7599773Z 00:26:07 querier: 2025/06/06 00:26:07 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7600141Z 00:26:07 Ports for container: e2e-cortex-test-querier Mapping: map[80:32909 9095:32910]
2025-06-06T00:26:09.7600231Z 00:26:07 Starting ruler
2025-06-06T00:26:09.7600432Z 00:26:07 ruler: 2025/06/06 00:26:07 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025-06-06T00:26:09.7600907Z 00:26:07 ruler: ts=2025-06-06T00:26:07.977310513Z caller=mapper.go:50 level=error msg="unable to read rules directory" path=/rules err="open /rules: no such file or directory"
2025-06-06T00:26:09.7601273Z 00:26:08 Ports for container: e2e-cortex-test-ruler Mapping: map[80:32911 9095:32912]
2025-06-06T00:26:09.7601367Z 00:26:08 Stopping ruler
2025-06-06T00:26:09.7601812Z 00:26:08 ruler: ts=2025-06-06T00:26:08.527698025Z caller=notifier.go:64 level=error user=user msg="error starting notifier discovery manager" err="context canceled"
2025-06-06T00:26:09.7602016Z 00:26:08 Error response from daemon: No such container: e2e-cortex-test-ruler
2025-06-06T00:26:09.7602028Z 
2025-06-06T00:26:09.7602132Z 00:26:08 Stopping query-frontend
2025-06-06T00:26:09.7602651Z 00:26:08 query-frontend: ts=2025-06-06T00:26:08.684238577Z caller=grpc_logging.go:74 level=warn method=/frontend.Frontend/Process duration=1.161669467s err="queue is stopped" msg=gRPC
2025-06-06T00:26:09.7603220Z 00:26:08 querier: ts=2025-06-06T00:26:08.684461264Z caller=frontend_processor.go:75 level=error msg="error processing requests" address=172.18.0.6:9095 err="rpc error: code = Unknown desc = queue is stopped"
2025-06-06T00:26:09.7604200Z 00:26:08 querier: ts=2025-06-06T00:26:08.771070818Z caller=frontend_processor.go:69 level=error msg="error contacting frontend" address=172.18.0.6:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.6:9095: connect: connection refused\""
2025-06-06T00:26:09.7604564Z 00:26:08 Error response from daemon: No such container: e2e-cortex-test-query-frontend
2025-06-06T00:26:09.7604581Z 
2025-06-06T00:26:09.7604735Z 00:26:08 Stopping querier
2025-06-06T00:26:09.7605704Z 00:26:08 querier: ts=2025-06-06T00:26:08.837890625Z caller=frontend_processor.go:57 level=warn msg="failed to notify querier shutdown to query-frontend" address=172.18.0.6:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.0.6:9095: connect: connection refused\""
2025-06-06T00:26:09.7606517Z 00:26:08 querier: ts=2025-06-06T00:26:08.837975158Z caller=client.go:283 level=error msg="error getting path" key=collectors/store-gateway err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/store-gateway?index=15&stale=&wait=10000ms\": context canceled"
2025-06-06T00:26:09.7607235Z 00:26:08 querier: ts=2025-06-06T00:26:08.837975014Z caller=client.go:283 level=error msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=15&stale=&wait=10000ms\": context canceled"
2025-06-06T00:26:09.7607544Z 00:26:08 Error response from daemon: No such container: e2e-cortex-test-querier
2025-06-06T00:26:09.7607552Z 
2025-06-06T00:26:09.7607705Z 00:26:08 Killing ingester
2025-06-06T00:26:09.7608011Z 00:26:09 Error response from daemon: No such container: e2e-cortex-test-ingester
2025-06-06T00:26:09.7608017Z 
2025-06-06T00:26:09.7608120Z 00:26:09 Killing distributor
2025-06-06T00:26:09.7608344Z 00:26:09 Error response from daemon: No such container: e2e-cortex-test-distributor
2025-06-06T00:26:09.7608349Z 
2025-06-06T00:26:09.7608452Z 00:26:09 Killing minio-9000
2025-06-06T00:26:09.7608677Z 00:26:09 Error response from daemon: No such container: e2e-cortex-test-minio-9000
2025-06-06T00:26:09.7608682Z 
2025-06-06T00:26:09.7608774Z 00:26:09 Killing consul
2025-06-06T00:26:09.7608978Z 00:26:09 Error response from daemon: No such container: e2e-cortex-test-consul
2025-06-06T00:26:09.7608983Z 
2025-06-06T00:26:09.7609134Z --- PASS: TestRulerEvalWithQueryFrontend (11.48s)
2025-06-06T00:26:09.7609389Z     --- PASS: TestRulerEvalWithQueryFrontend/format:protobuf (5.57s)
2025-06-06T00:26:09.7609769Z     --- PASS: TestRulerEvalWithQueryFrontend/format:json (2.04s)
2025-06-06T00:26:09.7609844Z PASS
2025-06-06T00:26:09.7610034Z ok  	github.com/cortexproject/cortex/integration	219.208s
2025-06-06T00:26:09.7610231Z ?   	github.com/cortexproject/cortex/integration/ca	[no test files]
2025-06-06T00:26:09.7610345Z === RUN   TestTimeToMilliseconds
2025-06-06T00:26:09.7610466Z --- PASS: TestTimeToMilliseconds (0.00s)
2025-06-06T00:26:09.7610544Z PASS
2025-06-06T00:26:09.7610709Z ok  	github.com/cortexproject/cortex/integration/e2e	0.018s
2025-06-06T00:26:09.7610934Z ?   	github.com/cortexproject/cortex/integration/e2e/cache	[no test files]
2025-06-06T00:26:09.7611281Z ?   	github.com/cortexproject/cortex/integration/e2e/db	[no test files]
2025-06-06T00:26:09.7611504Z ?   	github.com/cortexproject/cortex/integration/e2e/images	[no test files]
2025-06-06T00:26:09.7611713Z ?   	github.com/cortexproject/cortex/integration/e2ecortex	[no test files]
2025-06-06T00:26:10.0397322Z Post job cleanup.
2025-06-06T00:26:10.1340448Z [command]/usr/bin/git version
2025-06-06T00:26:10.1375868Z git version 2.49.0
2025-06-06T00:26:10.1425746Z Temporarily overriding HOME='/home/runner/work/_temp/8d84bbcc-17fa-45bd-b123-8080aa3c52a9' before making global git config changes
2025-06-06T00:26:10.1426861Z Adding repository directory to the temporary git global config as a safe directory
2025-06-06T00:26:10.1430639Z [command]/usr/bin/git config --global --add safe.directory /home/runner/work/cortex/cortex
2025-06-06T00:26:10.1463827Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-06-06T00:26:10.1495245Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-06-06T00:26:10.1727630Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-06-06T00:26:10.1747767Z http.https://github.com/.extraheader
2025-06-06T00:26:10.1759947Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2025-06-06T00:26:10.1789294Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-06-06T00:26:10.2113383Z Post job cleanup.
2025-06-06T00:26:10.3787233Z [command]/opt/hostedtoolcache/go/1.23.2/x64/bin/go env GOMODCACHE
2025-06-06T00:26:10.3832279Z [command]/opt/hostedtoolcache/go/1.23.2/x64/bin/go env GOCACHE
2025-06-06T00:26:10.3890725Z /home/runner/go/pkg/mod
2025-06-06T00:26:10.3923384Z /home/runner/.cache/go-build
2025-06-06T00:26:10.3928136Z Primary key was not generated. Please check the log messages above for more errors or information
2025-06-06T00:26:10.4020234Z Cleaning up orphan processes