Unable to collect metrics from Vector on 0.4.3

[jrosengren]

I had Parseable working via the helm package on 0.4.1, using the included Vector dependency to collect Kubernetes logs. Since upgrading to 0.4.3, Parseable is unable to receive logs, throwing the following stack trace:

thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', server/src/utils/arrow/merged_reader.rs:78:53
stack backtrace:
   0: rust_begin_unwind
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/core/src/panicking.rs:64:14
   2: core::panicking::panic
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/core/src/panicking.rs:111:5
   3: <F as itertools::kmerge_impl::KMergePredicate<T>>::kmerge_pred
   4: itertools::kmerge_impl::kmerge_by
   5: parseable::storage::staging::convert_disk_files_to_parquet
   6: parseable::storage::object_storage::ObjectStorage::sync::{{closure}}
   7: parseable::object_store_sync::{{closure}}::{{closure}}::{{closure}}::{{closure}}::{{closure}}
   8: <clokwerk::async_scheduler::AsyncSchedulerFuture as core::future::future::Future>::poll
   9: <tokio::task::local::RunUntil<T> as core::future::future::Future>::poll
  10: <core::pin::Pin<P> as core::future::future::Future>::poll
  11: tokio::runtime::scheduler::current_thread::Context::enter
  12: tokio::macros::scoped_tls::ScopedKey<T>::set
  13: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
  14: tokio::runtime::runtime::Runtime::block_on
  15: std::panicking::try
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Since this was working on 0.4.1, I believe I have all of the right environment variables set in the parseable-env-secret:

s3.access.key:  32 bytes
s3.bucket:      8 bytes
s3.secret.key:  64 bytes
s3.url:         65 bytes
fs.dir:         6 bytes
password:       7 bytes
staging.dir:    9 bytes
username:       5 bytes
addr:           12 bytes
s3.region:      9 bytes

The Vector config is exactly as describe in the values.yaml from the helm chart in the parseable repo.

[nitisht]

Thanks for the issue @jrosengren we just noticed this as well. This is a regression. We're working on fixing this.

Meanwhile, it is better to revert to an older version like 0.4.1 that was working for you. Due to the nature of this bug, we'll need to modify Parseable meta files (before you can get up and running again with 0.4.1). So, if you could join us on Slack (https://launchpass.com/parseable) it would be easier to collaborate.

[nitisht]

hey @jrosengren fix for this is available now in the next release parseable/parseable:v0.4.4. It works as expected in our local tests (with vector).

[jrosengren]

Updated to v0.4.4 and everything seems to be working properly as expected. Thanks!

[nitisht]

Thanks for the confirmation.