Add tfio.IOTensor.from_prometheus support #438
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yongtang
force-pushed
the
io_tensor_prometheus
branch
from
e2c6abf to
173f5eb
Compare
In the past couple of months I have been trying to come up with a good example of using Prometheus with TensorFlow for infrastructure/compute usage prediction and alerting in case of anomaly. My plan was to use LSTM to combine with prometheus for that. Though there is a PrometheusDataset support, the format of dataset is not very intuitive and hard to do it in a smooth way. The biggest challenge is that, normally for time series data you have a look back window to train, and you normalize the data (with total). Both are not easily available. Further more, to process data in normal TF operations such as tf.roll, it requires reading the whole data into one Tenor. But this is not very straightforward as well with iterable dataset. This PR adds tfio.IOTensor.from_prometheus which allows easily convertion the prometheus observation data into a tuple of Tensors, and do additional feature engineering. Signed-off-by: Yong Tang <[email protected]>
Signed-off-by: Yong Tang <[email protected]>
Signed-off-by: Yong Tang <[email protected]>
yongtang
force-pushed
the
io_tensor_prometheus
branch
from
173f5eb to
9ac5297
Compare
Signed-off-by: Yong Tang <[email protected]>
Signed-off-by: Yong Tang <[email protected]>
Member
Author
|
@jiachengxu This PR adds prometheus support for IOTensor, though the last commit 6844b4f I added |
terrytangyuan
approved these changes
Aug 25, 2019
i-ony
pushed a commit
to i-ony/io
that referenced
this pull request
Feb 8, 2021
* Add tfio.IOTensor.from_prometheus support In the past couple of months I have been trying to come up with a good example of using Prometheus with TensorFlow for infrastructure/compute usage prediction and alerting in case of anomaly. My plan was to use LSTM to combine with prometheus for that. Though there is a PrometheusDataset support, the format of dataset is not very intuitive and hard to do it in a smooth way. The biggest challenge is that, normally for time series data you have a look back window to train, and you normalize the data (with total). Both are not easily available. Further more, to process data in normal TF operations such as tf.roll, it requires reading the whole data into one Tenor. But this is not very straightforward as well with iterable dataset. This PR adds tfio.IOTensor.from_prometheus which allows easily convertion the prometheus observation data into a tuple of Tensors, and do additional feature engineering. Signed-off-by: Yong Tang <[email protected]> * Rename libtensorflow_io_prometheus.so to libtensorflow_io_golang.so Signed-off-by: Yong Tang <[email protected]> * Read single column at a time so that resource could be reused. Signed-off-by: Yong Tang <[email protected]> * Change to shapes, dtypes to match TensorSpec order Signed-off-by: Yong Tang <[email protected]> * Add `records` mode for `from_json` that matches `records` in pandas Signed-off-by: Yong Tang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: This PR depends on PR #437PR #437 has been merged.In the past couple of months I have been trying to come up with a good example of using Prometheus with TensorFlow for infrastructure/compute usage prediction and alerting in case of anomaly. My plan was to use LSTM to combine with prometheus observation data for that.
Though there is a PrometheusDataset support already, the format of dataset is not very intuitive and hard to do it in a smooth way.
The biggest challenge is that, normally for time series data you have to setup a look back window for training, and you normalize the data (with total). Both are not easily available.
Further more, to process data in normal TF operations such as tf.roll, it requires reading the whole data into one Tenor. But this is not very straightforward as well with iterable dataset.
This PR adds tfio.IOTensor.from_prometheus which allows easily conversion the prometheus observation data into a tuple of Tensors, and do additional feature engineering.
Signed-off-by: Yong Tang [email protected]