v0.0.3

.dockerignore

@@ -11,3 +11,6 @@ dist
 
 # Don't put Mac stuff in the docker build env
 cocoa
+
+c\+\+/build
+

.gitignore

@@ -9,3 +9,4 @@ eggs
 build
 dist
 *~
+test_build

ProtobagCocoa.podspec.json

@@ -1,6 +1,6 @@
 {
   "name": "ProtobagCocoa",
-  "version": "0.0.2",
+  "version": "0.0.3",
   "summary": "Protobag: an archive of string-serialized Protobufs",
   "homepage": "https://github.com/StandardCyborg/protobag",
   "license": "Apache 2",
@@ -10,7 +10,7 @@
   "cocoapods_version": ">= 1.0",
   "source": {
     "git": "[email protected]:StandardCyborg/protobag.git",
-    "tag": "v0.0.2"
+    "tag": "v0.0.3"
   },
   "public_header_files": [
     "c++/protobag/**/*.{hpp,h}"

README.md

@@ -1,8 +1,17 @@
-# Protobag: A bag o' String-serialized Protobuf Messages
+# Protobag: A bag o' Serialized Protobuf Messages
 _With built-in support for time-series data_
 
 [![Build Status](https://circleci.com/gh/StandardCyborg/protobag.svg?style=svg&circle-token=ed56e2ec32789fa3e5f664bc8ea73c55e119de4b)](https://app.circleci.com/pipelines/github/StandardCyborg/protobag)
 
+## Quickstart & Demo
+
+See [this python noteboook](examples/notebook-demo/protobag-demo-full.ipynb) 
+for a demo of key features.
+
+Or you can drop into a Protobag development shell using a clone of this repo
+and Docker; FMI see:
+`./pb-dev --help`
+
 ## Summary 
 
 [Protobuf](https://github.com/protocolbuffers/protobuf) is a popular data
@@ -60,31 +69,100 @@ wrappers over `libarchive`.  See
 [ArchiveUtil](c++/protobag/protobag/ArchiveUtil.hpp).
 
 
+## Development
+
+
+## Discussion of Key Features
+
+### Protobag indexes Protobuf message Descriptors
+
+By default, `protobag` not only saves those messages but also 
+**indexes Protobuf message descriptors** so that your `protobag` readers don't
+need your proto schemas to decode your messages.  
+
+#### Wat?
+In order to deserialize a Protobuf message, typically you need
+`protoc`-generated code for that message type (and you need `protoc`-generated
+code for your specific programming language).  This `protoc`-generated code is
+engineered for efficiency and provides a clean API for accessing message
+attributes.  But what if you don't have that `protoc`-generated code?  Or you
+don't even have the `.proto` message definitions to generate such code?
+
+In Protobuf version 3.x, the authors added official support for
+[the self-describing message paradigm](https://developers.google.com/protocol-buffers/docs/techniques).  
+Now a user can serialize not just a message but Protobuf Descriptor data that
+describes the message schema and enables deserialzing the message
+*without protoc-generated code*-- all you need is the `protobuf` library itself.  
+(This is a core feature of other serialization libraries 
+[like Avro](http://avro.apache.org/docs/1.6.1/)).
+
+Note: dynamic message decoding is slower than using `protoc`-generated code.  
+Furthermore, the `protoc`-generated code makes defensive programming a bit 
+easier.  You probably want to use the `protoc`-generated code for your 
+messages if you can.
 
+#### Protobag enables all messages to be self-describing messages
+While Protobuf includes tools for using self-describing messages, the feature 
+isn't simply a toggle in your `.proto` file, and the API is a bit complicated 
+(because Google claims they don't use it much internally).
 
-TODO: quickstart and stuff
+`protobag` automatically indexes the Protobuf Descriptor data for your messages 
+at write time.  (And you can disable this indexing if so desired).  At read 
+time, `protobag` automatically uses this indexed Descriptor data if the user 
+reading your `protobag` file lacks the needed `protoc`-generated code to 
+deserialize a message.
 
-TODO: bag index doc
+What if a message type evolves?  `protobag` indexes each distinct message type 
+for each write session.  If you change your schema for a message type between 
+write sessions, `protobag` will have indexed both schemas and will use the 
+proper one for dynamic deserialization.
 
-TODO: "treat as a map" API
+#### For More Detail
 
+For Python, see:
+ * `protobag.build_fds_for_msg()` -- This method collects the descriptor data
+     needed for any Protobuf Message instance or class.
+ * `protobag.DynamicMessageFactory::dynamic_decode()` -- This method uses 
+     standard Protobuf APIs to deserialize messages given only Protobuf
+     Descriptor data.
 
-coming soon
+For C++, see:
+ * `BagIndexBuilder::DescriptorIndexer::Observe()` -- This method collects the
+     descriptor data needed for any Protobuf Message instance or class.
+ * `DynamicMsgFactory` -- This utility uses uses standard Protobuf APIs to 
+     deserialize messages given only Protobuf Descriptor data.
 
+
+## Cocoa Pods
+
+You can integrate Protobag into an iOS or OSX application using the CocoaPod `ProtobagCocoa.podspec.json`
+podspec included in this repo.  Protobag is explicitly designed to be cross-platform (and utilize only C++
+features friendly to iOS) to facilitate such interoperability.
+
+Note: before pushing, be sure to edit the "version" field of the `ProtobagCocoa.podspec.json` file
+to match the version you're pushing.
 ```
-indocker % cd /opt/protobag/cxx
-indocker % mkdir -p build && cd build
-indocker % cmake -DCMAKE_BUILD_TYPE=DEBUG ..
-indocker % make -j `nproc` && ./protobag_test --gtest_filter=DemoTest*
+ pod repo push  SCCocoaPods ProtobagCocoa.podspec.json  --use-libraries --verbose --allow-warnings
 ```
 
+## C++ Build
+
+Use the existing CMake-based build system.
+
+In c++ subdir:
 ```
- pod repo push  SCCocoaPods ProtobagCocoa.podspec.json  --use-libraries --verbose --allow-warnings
+mkdir build && cd build
+cmake ..
+make -j
+make test
 ```
 
+## Python Build
 
-in python subdir:
+The Python library includes a wheel that leverages the above C++ CMake build system.
+
+In python subdir:
 ```
 python3 setup.py bdist_wheel
 ```
-for both linux and xcode
+

c++/protobag/protobag/BagIndexBuilder.cpp

@@ -5,8 +5,6 @@
 #include <tuple>
 #include <unordered_set>
 
-                                                                  #include <iostream>
-
 #include <google/protobuf/descriptor.h>
 #include <google/protobuf/util/time_util.h>
 
@@ -156,40 +154,36 @@ BagIndex_TopicStats &BagIndexBuilder::GetMutableStats(const std::string &topic)
   return topic_to_stats[topic];
 }
 
-// uint64_t BagIndexBuilder::GetNextFilenum(const std::string &topic) {
-//   const auto &stats = GetMutableStats(topic);
-//   return stats.n_messages() + 1;
-// }
-
-
 void BagIndexBuilder::Observe(
     const Entry &entry, const std::string &final_entryname) {
 
   const std::string entryname = 
     final_entryname.empty() ? entry.entryname : final_entryname;
 
   if (_do_timeseries_indexing) {
-    const auto &maybe_tt = entry.GetTopicTime();
-    if (maybe_tt.has_value()) {
-      TopicTime tt = *maybe_tt;
-      tt.set_entryname(entryname);
-
-      {
-        auto &stats = GetMutableStats(tt.topic());
-        stats.set_n_messages(stats.n_messages() + 1);
-      }
+    if (entry.IsStampedMessage()) {
+      const auto &maybe_tt = entry.GetTopicTime();
+      if (maybe_tt.has_value()) {
+        TopicTime tt = *maybe_tt;
+        tt.set_entryname(entryname);
+
+        {
+          auto &stats = GetMutableStats(tt.topic());
+          stats.set_n_messages(stats.n_messages() + 1);
+        }
 
-      {
-        if (!_tto) {
-          _tto.reset(new TopicTimeOrderer());
+        {
+          if (!_tto) {
+            _tto.reset(new TopicTimeOrderer());
+          }
+          _tto->Observe(tt);
         }
-        _tto->Observe(tt);
-      }
 
-      {
-        const auto &t = tt.timestamp();
-        *_index.mutable_start() = std::min(_index.start(), t);
-        *_index.mutable_end() = std::max(_index.end(), t);
+        {
+          const auto &t = tt.timestamp();
+          *_index.mutable_start() = std::min(_index.start(), t);
+          *_index.mutable_end() = std::max(_index.end(), t);
+        }
       }
     }
   }

c++/protobag/protobag/Entry.cpp

@@ -54,7 +54,18 @@ std::string Entry::ToString() const {
 // }
 
 bool MaybeEntry::IsNotFound() const {
-  return error == archive::Archive::ReadStatus::EntryNotFound().error;
+  static const std::string kIsNotFoundPrefix = 
+    archive::Archive::ReadStatus::EntryNotFound().error + ": ";
+  return error.find(kIsNotFoundPrefix) == 0;
+}
+
+MaybeEntry MaybeEntry::NotFound(const std::string &entryname) {
+  MaybeEntry m;
+  m.error = fmt::format(
+    "{}: {}",
+    archive::Archive::ReadStatus::EntryNotFound().error,
+    entryname);
+  return m;
 }
 
 std::string GetTopicFromEntryname(const std::string &entryname) {

c++/protobag/protobag/Entry.hpp

@@ -127,6 +127,7 @@ struct Entry {
 
   // == Raw Mode ================================
 
+  // Create a raw entry from a Protobuf message instance (force-skips indexing)
   template <typename MT>
   static Result<Entry> CreateRaw(
         const std::string &entryname,
@@ -149,6 +150,7 @@ struct Entry {
     };
   }
 
+  // Create a raw entry from raw bytes
   static Entry CreateRawFromBytes(
         const std::string &entryname,
         std::string &&raw_msg_contents) {
@@ -249,14 +251,15 @@ struct Entry {
     return 
       IsA<StampedMessage>() || (
         // An unpacked StampedDatum is OK too
-        GetTopicTime().has_value());
+        HasTopic());
   }
 
   bool IsRaw() const {
     return msg.type_url().empty();
   }
 
   std::optional<TopicTime> GetTopicTime() const;
+  bool HasTopic() const;
 
   template <typename MT>
   Result<MT> GetAs(bool validate_type_url = true) const {
@@ -306,6 +309,8 @@ struct MaybeEntry : public Result<Entry> {
   // See Archive::ReadStatus for definition; this can be an acceptible error
   bool IsNotFound() const;
 
+  static MaybeEntry NotFound(const std::string &entryname);
+
   static MaybeEntry Err(const std::string &s) {
     MaybeEntry m; m.error = s; return m;
   }
@@ -361,4 +366,9 @@ inline std::optional<TopicTime> Entry::GetTopicTime() const {
   }
 }
 
+inline bool Entry::HasTopic() const {
+  auto maybe_tt = GetTopicTime();
+  return maybe_tt.has_value() && !maybe_tt->topic().empty();
+}
+
 } /* namespace protobag */

c++/protobag/protobag/ReadSession.cpp

@@ -39,7 +39,7 @@ MaybeEntry ReadSession::ReadEntryFrom(
 
   const auto maybe_bytes = archive->ReadAsStr(entryname);
   if (maybe_bytes.IsEntryNotFound()) {
-    return MaybeEntry::Err(maybe_bytes.error);
+    return MaybeEntry::NotFound(entryname);
   } else if (!maybe_bytes.IsOk()) {
     return MaybeEntry::Err(
       fmt::format("Read error for {}: {}", entryname, maybe_bytes.error));
@@ -56,7 +56,7 @@ MaybeEntry ReadSession::ReadEntryFrom(
 
     auto maybe_any = 
       PBFactory::LoadFromContainer<google::protobuf::Any>(*maybe_bytes.value);
-        // do we need to handle text format separately ? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+        // TODO maybe handle text format separately ?
     if (!maybe_any.IsOk()) {
       return MaybeEntry::Err(fmt::format(
         "Could not read protobuf from {}: {}", entryname, maybe_any.error));
@@ -103,7 +103,7 @@ MaybeEntry ReadSession::GetNext() {
     _archive, entryname, _plan.raw_mode, _spec.unpack_stamped_messages);
   if (maybe_entry.IsNotFound()) {
     if (_plan.require_all) {
-      return MaybeEntry::Err(fmt::format("Entry not found: {}", entryname));
+      return MaybeEntry::NotFound(entryname);
     } else {
       return GetNext();
     }
@@ -126,6 +126,21 @@ Result<BagIndex> ReadSession::GetIndex(const std::string &path) {
   return ReadLatestIndex(rp->_archive);
 }
 
+Result<std::vector<std::string>> ReadSession::GetAllTopics(const std::string &path) {
+  auto maybe_index = GetIndex(path);
+  if (!maybe_index.IsOk()) {
+    return {.error = maybe_index.error};
+  }
+
+  const BagIndex &index = *maybe_index.value;
+  std::vector<std::string> topics;
+  topics.reserve(index.topic_to_stats_size());
+  for (const auto &entry : index.topic_to_stats()) {
+    topics.push_back(entry.first);
+  }
+  return {.value = topics};
+}
+
 Result<BagIndex> ReadSession::ReadLatestIndex(archive::Archive::Ptr archive) {
   if (!archive) {
     return {.error = "No archive to read"};
@@ -171,11 +186,12 @@ Result<ReadSession::ReadPlan> ReadSession::GetEntriesToRead(
     return {.error = "No archive to read"};
   }
 
-  auto maybe_index = ReadLatestIndex(archive); // TODO support multiple indices ~~~~~~~~~~~~~~~~
+  auto maybe_index = ReadLatestIndex(archive); // TODO support multiple indices
   if (!maybe_index.IsOk()) {
+    // TODO: support reindexing
     // // Then create one!
     // maybe_index = GetReindexed(archive);
-    return {.error = "Unindexed protobag not supported right now"}; // ~~~~~~~~~~~~~~~~~~~~~~~~~
+    return {.error = "Unindexed protobag not currently supported"};
   }
 
   if (!maybe_index.IsOk()) {
@@ -291,12 +307,13 @@ Result<ReadSession::ReadPlan> ReadSession::GetEntriesToRead(
       if (window.has_end() && (window.end() < tt.timestamp())) {
         continue;
       }
-// std::cout << "entries_to_read: " << tt.entryname() << std::endl;
+
       entries_to_read.push(tt.entryname());
     }
     return {.value = ReadPlan{
       .entries_to_read = entries_to_read,
-      .require_all = false, // TODO should we report if index and archive don't match? ~~~~~~~~~~~~~~
+      .require_all = false, 
+          // TODO should we report if index and archive don't match?
       .raw_mode = false,
     }};