From 28113e55a32efbb4e6c1f76f3eecc9bed3dc89f7 Mon Sep 17 00:00:00 2001 From: "Mark A. Matney, Jr" Date: Wed, 16 Jun 2021 07:00:00 -0700 Subject: [PATCH 01/10] [SERV-126] Generate binary audiowaveform data for audio resources --- ARCHITECTURE.md | 12 +- README.md | 42 +++- pom.xml | 18 ++ .../library/avpairtree/AvPtConstants.java | 5 + .../ucla/library/avpairtree/AvPtUtils.java | 32 +++ .../edu/ucla/library/avpairtree/Config.java | 5 + .../edu/ucla/library/avpairtree/CsvItem.java | 10 +- .../verticles/ConverterVerticle.java | 21 +- .../avpairtree/verticles/MainVerticle.java | 2 + .../avpairtree/verticles/WatcherVerticle.java | 163 ++++++++----- .../verticles/WaveformVerticle.java | 223 ++++++++++++++++++ src/main/resources/av-pairtree_messages.xml | 8 + src/main/resources/logback.xml | 3 + .../avpairtree/utils/TestConstants.java | 5 + .../verticles/AbstractAvPtTest.java | 6 + .../verticles/WaveformVerticleTest.java | 92 ++++++++ src/test/resources/logback-test.xml | 3 + src/test/resources/soul/audio/uclapasc.dat | Bin 0 -> 5188 bytes src/test/resources/test-config.properties | 3 + 19 files changed, 568 insertions(+), 85 deletions(-) create mode 100644 src/main/java/edu/ucla/library/avpairtree/AvPtUtils.java create mode 100644 src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java create mode 100644 src/test/java/edu/ucla/library/avpairtree/verticles/WaveformVerticleTest.java create mode 100644 src/test/resources/soul/audio/uclapasc.dat diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index a1929ef..d878708 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -6,15 +6,18 @@ This document describes the high level architecture of the av-pairtree program. The av-pairtree program is an application that watches a "drop box" for new or updated CSV files with a certain structure. When a CSV file with A/V files has been put into the drop box, the av-pairtree program picks them up and processes their audio and video files. -The processing of video files (which are MP4s by default) involves putting them into a [Pairtree](https://tools.ietf.org/html/draft-kunze-pairtree-01) structure that's accessible to a media server. The processing of audio files (which are WAVs by default) involves converting the WAV files into MP4 files and, then, putting them into the Pairtree structure. +The processing of video files (which are MP4s by default) involves putting them into a [Pairtree](https://tools.ietf.org/html/draft-kunze-pairtree-01) structure that's accessible to a media server. The processing of audio files (which are WAVs by default) involves two conversions: -After all the A/V files in a CSV file have been processed, the input CSV is updated to include the resources' new access URLs (i.e. the URLs of the media files as served by the media server) and written back out to the file system. + * to MP4 format, the results of which are put into the Pairtree structure; and + * to binary [audiowaveform](https://github.com/bbc/audiowaveform) format, the results of which are deposited to AWS S3. + +After all the A/V files in a CSV file have been processed, the input CSV is updated to include the resources' new access URLs (i.e. the URLs of the media files as served by the media server) and audiowaveform URLs, then written back out to the file system. ![Overview diagram for av-pairtree's components](docs/images/av_pairtree_overview.svg) ## Expected CSV Structure -A CSV file that is going to be processed by av-pairtree should have two required columns: `File Name` and `ItemARK`. The first is used to retrieve the media file to be processed and the second is used to create the Pairtree structure. If the file has been previously processed (or has been processed by the Bucketeer application), it will also have a `IIIF Access URL` column. That's fine. However, older CSV files may have `iiif_access_url` as a column header. Any CSVs with that column header should be manually updated before processing with av-pairtree. +A CSV file that is going to be processed by av-pairtree should have two required columns: `File Name` and `ItemARK`. The first is used to retrieve the media file to be processed and the second is used to create the Pairtree structure. If the file has been previously processed (or has been processed by the Bucketeer application), it will also have a `IIIF Access URL` column; if previously processed, it may also have a `Waveform` column. That's fine. However, older CSV files may have `iiif_access_url` as a column header. Any CSVs with that column header should be manually updated before processing with av-pairtree. ## Code Map @@ -33,7 +36,7 @@ The basic structure of this particular Vert.x program includes: verticles, handl | CsvItem | This is an object which represents a single item (or row) from the CSV file | [src/main/java/edu/ucla/library/avpairtree/CsvItem.java](https://github.com/UCLALibrary/av-pairtree/blob/main/src/main/java/edu/ucla/library/avpairtree/CsvItem.java) | | CsvItemCodec | This codec implements a JSON (de)serialization of CsvItem so that it can be sent over the event bus | [src/main/java/edu/ucla/library/avpairtree/CsvItemCodec.java](https://github.com/UCLALibrary/av-pairtree/blob/main/src/main/java/edu/ucla/library/avpairtree/CsvItemCodec.java) | -Actions (e.g., the parsing of CSV files, conversion of media files, or storage of media files in a Pairtree structure, etc.) are performed by the application's various verticles (e.g., WatcherVerticle, ConverterVerticle, PairtreeVerticle, etc.) Cf. the `verticles` directory for examples. +Actions (e.g., the parsing of CSV files, conversion of media files, or storage of media files in a Pairtree structure, etc.) are performed by the application's various verticles (e.g., WatcherVerticle, ConverterVerticle, PairtreeVerticle, WaveformVerticle, etc.) Cf. the `verticles` directory for examples. ## Sequence Diagram @@ -66,6 +69,7 @@ The properties and default values for an av-pairtree configuration file are as f | audio.channels | The number of channels in the audio stream | 2 | | iiif.access.url | The URL pattern into which to insert the Pairtree path | N/A | | conversion.workers | The number of cores to use for audio file conversion | 2 | +| audiowaveform.s3.bucket | The S3 bucket to use for depositing binary audiowaveform data files | test-audiowaveform-resources | ## Documentation diff --git a/README.md b/README.md index cd4c5e1..37ed9bb 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,36 @@ # A/V Pairtree -This project processes A/V files into a collection of Pairtree structures. +This project processes A/V files into a collection of Pairtree structures and generates waveform data (for visualization) from audio files. ## Pre-requisites -The only hard requirement is that you need a [JDK (>= 11)](https://adoptopenjdk.net/) installed and configured. +In addition to a [JDK (>= 11)](https://adoptopenjdk.net/) installed and configured, you will also need the following in order to generate waveform data for audio files: + + * the BBC's [audiowaveform](https://github.com/bbc/audiowaveform) command line tool, with the executable on your PATH; and + * an AWS S3 bucket for storing the audio waveform data, and write credentials for the bucket (which you'll likely want to be world-readable). There are two sets of build instructions: one for systems with [Maven](https://maven.apache.org/) pre-installed and one for systems without Maven. +## Setting AWS credentials for integration tests + +In order to run the integration tests that use AWS S3, you should have an entry like the following in the `profiles` section of your `/home/.m2/settings.xml` (or another settings file elsewhere): + +```xml + + av-pairtree + + + !skipDefaultProfile + + + + myAwsAccessKey + us-west-2 + myAwsSecretKey + + +``` + ## Building and testing locally without Maven pre-installed To build the project the first time, type: @@ -42,9 +65,20 @@ To process one of the test CSVs, you can copy a CSV file from `src/test/resource ## Running in production -To run av-pairtree from the Jar file, one needs to type the following: +To run av-pairtree from the Jar file, one must set AWS S3 credentials and then run the JAR: + +```bash +#!/bin/bash + +export AWS_ACCESS_KEY_ID=myAwsAccessKey +export AWS_DEFAULT_REGION=us-west-2 +export AWS_SECRET_ACCESS_KEY=myAwsSecretKey - java -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -Dvertx-config-path=config.properties -jar target/av-pairtree-0.0.1-SNAPSHOT.jar run edu.ucla.library.avpairtree.verticles.MainVerticle +java \ + -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory \ + -Dvertx-config-path=config.properties \ + -jar target/av-pairtree-0.0.1-SNAPSHOT.jar run edu.ucla.library.avpairtree.verticles.MainVerticle +``` The application is configured by the value of `vertx-config-path`, which in the example above is a config file residing in the same directory as the Jar file. diff --git a/pom.xml b/pom.xml index 1abe64e..2de969d 100644 --- a/pom.xml +++ b/pom.xml @@ -36,6 +36,7 @@ 1.7.30 0.15.0 2.3.0 + 2.15.15 1.0.23 @@ -67,6 +68,13 @@ pom import + + software.amazon.awssdk + bom + ${awssdk.version} + pom + import + @@ -121,6 +129,10 @@ jave-core ${jave.version} + + software.amazon.awssdk + s3 + @@ -150,6 +162,7 @@ synanon/video/synanon.mp4 soul/audio/uclapasc.wav + soul/audio/uclapasc.dat @@ -249,6 +262,11 @@ io.vertx.core.logging.SLF4JLogDelegateFactory ${project.basedir}/target/test-classes/test-config.properties + + ${avpt.s3.access_key} + ${avpt.s3.region} + ${avpt.s3.secret_key} + 1 ${jacoco.agent.arg} diff --git a/src/main/java/edu/ucla/library/avpairtree/AvPtConstants.java b/src/main/java/edu/ucla/library/avpairtree/AvPtConstants.java index 1e3f31b..322fdcc 100644 --- a/src/main/java/edu/ucla/library/avpairtree/AvPtConstants.java +++ b/src/main/java/edu/ucla/library/avpairtree/AvPtConstants.java @@ -21,6 +21,11 @@ public final class AvPtConstants { */ public static final String SYSTEM_TMP_DIR = System.getProperty("java.io.tmpdir"); + /** + * The template string for AWS S3 resource URLs. The slots are (in order): bucket name, region, and object key. + */ + public static final String AUDIOWAVEFORM_URL_TEMPLATE = "https://{}.s3-{}.amazonaws.com/{}"; + /* * Constant classes have private constructors. */ diff --git a/src/main/java/edu/ucla/library/avpairtree/AvPtUtils.java b/src/main/java/edu/ucla/library/avpairtree/AvPtUtils.java new file mode 100644 index 0000000..2da6f31 --- /dev/null +++ b/src/main/java/edu/ucla/library/avpairtree/AvPtUtils.java @@ -0,0 +1,32 @@ +package edu.ucla.library.avpairtree; + +import java.nio.file.Path; + +import info.freelibrary.util.Constants; + +/** + * A class for utility methods. + */ +public final class AvPtUtils { + + private AvPtUtils() { + } + + /** + * Gets the input file path from available variables. + * + * @param aCsvItem A CSV item sent to us over the wire + * @param aSourceDir A pre-configured source files directory + * @return A file system path for the input file + */ + public static Path getInputFilePath(final CsvItem aCsvItem, final String aSourceDir) { + final String relativeFilePath = aCsvItem.getFilePath(); + + // Use our source folder unless we receive a file path that is absolute + if (!relativeFilePath.startsWith(Constants.SLASH)) { + return Path.of(aSourceDir, relativeFilePath); + } else { + return Path.of(relativeFilePath); + } + } +} diff --git a/src/main/java/edu/ucla/library/avpairtree/Config.java b/src/main/java/edu/ucla/library/avpairtree/Config.java index c77caa6..b23c39e 100644 --- a/src/main/java/edu/ucla/library/avpairtree/Config.java +++ b/src/main/java/edu/ucla/library/avpairtree/Config.java @@ -77,6 +77,11 @@ public final class Config { */ public static final String CONVERSION_WORKERS = "conversion.workers"; + /** + * The configuration property for the S3 bucket for audio waveforms. + */ + public static final String AUDIOWAVEFORM_S3_BUCKET = "audiowaveform.s3.bucket"; + // Constant classes should have private constructors. private Config() { } diff --git a/src/main/java/edu/ucla/library/avpairtree/CsvItem.java b/src/main/java/edu/ucla/library/avpairtree/CsvItem.java index 10cc167..a58d863 100644 --- a/src/main/java/edu/ucla/library/avpairtree/CsvItem.java +++ b/src/main/java/edu/ucla/library/avpairtree/CsvItem.java @@ -22,11 +22,19 @@ public class CsvItem { /** - * The CSV header column for the IIIF access URL. + * The CSV header column for the IIIF access URL. Note that this not used for deserialization; see + * WatcherVerticle.updateCSV for its use in serialization. */ @CsvIgnore public static final String IIIF_ACCESS_URL_HEADER = "IIIF Access URL"; + /** + * The CSV header column for the Waveform. Note that this not used for deserialization; see + * WatcherVerticle.updateCSV for its use in serialization. + */ + @CsvIgnore + public static final String WAVEFORM_HEADER = "Waveform"; + /** * The CSV header column for the item's identifier. */ diff --git a/src/main/java/edu/ucla/library/avpairtree/verticles/ConverterVerticle.java b/src/main/java/edu/ucla/library/avpairtree/verticles/ConverterVerticle.java index e284172..79cdc3e 100644 --- a/src/main/java/edu/ucla/library/avpairtree/verticles/ConverterVerticle.java +++ b/src/main/java/edu/ucla/library/avpairtree/verticles/ConverterVerticle.java @@ -10,6 +10,7 @@ import info.freelibrary.util.Logger; import info.freelibrary.util.LoggerFactory; +import edu.ucla.library.avpairtree.AvPtUtils; import edu.ucla.library.avpairtree.Config; import edu.ucla.library.avpairtree.CsvItem; import edu.ucla.library.avpairtree.MessageCodes; @@ -79,7 +80,7 @@ public void start(final Promise aPromise) { final CsvItem csvItem = message.body(); try { - final Path inputFilePath = getInputFilePath(csvItem, sourceDir); + final Path inputFilePath = AvPtUtils.getInputFilePath(csvItem, sourceDir); final Path outputFilePath = getOutputFilePath(inputFilePath, outputFormat); final EncodingAttributes encoding = new EncodingAttributes(); final AudioAttributes audio = new AudioAttributes(); @@ -138,22 +139,4 @@ private Path getOutputFilePath(final Path aInputFilePath, final String aOutputFo return Path.of(SYSTEM_TMP_DIR, SCRATCH_SPACE, outputFileName); } - - /** - * Gets the input file path from available variables. - * - * @param aCsvItem A CSV item sent to us over the wire - * @param aSourceDir A pre-configured source files directory - * @return A file system path for the input file - */ - private Path getInputFilePath(final CsvItem aCsvItem, final String aSourceDir) { - final String relativeFilePath = aCsvItem.getFilePath(); - - // Use our source folder unless we receive a file path that is absolute - if (!relativeFilePath.startsWith(Constants.SLASH)) { - return Path.of(aSourceDir, relativeFilePath); - } else { - return Path.of(relativeFilePath); - } - } } diff --git a/src/main/java/edu/ucla/library/avpairtree/verticles/MainVerticle.java b/src/main/java/edu/ucla/library/avpairtree/verticles/MainVerticle.java index 148badd..d6848ca 100644 --- a/src/main/java/edu/ucla/library/avpairtree/verticles/MainVerticle.java +++ b/src/main/java/edu/ucla/library/avpairtree/verticles/MainVerticle.java @@ -18,6 +18,7 @@ import edu.ucla.library.avpairtree.handlers.StatusHandler; import io.methvin.watcher.DirectoryWatcher; + import io.vertx.config.ConfigRetriever; import io.vertx.core.AbstractVerticle; import io.vertx.core.CompositeFuture; @@ -131,6 +132,7 @@ private void configureServer(final JsonObject aConfig, final Promise aProm futures.add(deployVerticle(new WatcherVerticle(), aConfig)); futures.add(deployVerticle(new PairtreeVerticle(), aConfig)); futures.add(deployVerticle(new ConverterVerticle(), aConfig.copy().put(WORKER, true))); + futures.add(deployVerticle(new WaveformVerticle(), aConfig)); CompositeFuture.all(futures).onSuccess(result -> { startCsvDirWatcher(aConfig).onComplete(startup -> { diff --git a/src/main/java/edu/ucla/library/avpairtree/verticles/WatcherVerticle.java b/src/main/java/edu/ucla/library/avpairtree/verticles/WatcherVerticle.java index 0ee077d..0b6a513 100644 --- a/src/main/java/edu/ucla/library/avpairtree/verticles/WatcherVerticle.java +++ b/src/main/java/edu/ucla/library/avpairtree/verticles/WatcherVerticle.java @@ -6,11 +6,11 @@ import java.io.StringReader; import java.nio.charset.StandardCharsets; import java.util.ArrayList; -import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.regex.Matcher; import java.util.regex.Pattern; +import java.util.stream.Collectors; import org.csveed.api.CsvClient; import org.csveed.api.CsvClientImpl; @@ -39,6 +39,7 @@ import io.vertx.core.eventbus.DeliveryOptions; import io.vertx.core.eventbus.EventBus; import io.vertx.core.eventbus.Message; +import io.vertx.core.json.JsonObject; /** * A verticle that responds to events from the CSV directory watcher. @@ -69,26 +70,37 @@ public void start(final Promise aPromise) { reader.readBeans().forEach(item -> { item.setPathRoot(item.getFilePath()); - if (item.isAudio()) { // Audio gets converted from wav to a more Web-friendly format - futures.add(eventBus.request(ConverterVerticle.class.getName(), item, options)); + if (item.isAudio()) { + // Audio gets converted from wav to a more Web-friendly format, and a waveform file is generated + futures.add(eventBus.request(ConverterVerticle.class.getName(), item, options)); + futures.add(eventBus.request(WaveformVerticle.class.getName(), item, options)); } else if (item.isVideo()) { // Videos are already in mp4 format so don't need conversion - futures.add(eventBus.request(PairtreeVerticle.class.getName(), item, options)); + futures.add(eventBus.request(PairtreeVerticle.class.getName(), item, options)); } // else, ignore }); CompositeFuture.all(futures).onSuccess(conversions -> { - final Map arkMap = new HashMap<>(); - - conversions.result().list().stream().forEach(object -> { - @SuppressWarnings("unchecked") // Composite futures don't support typing - final CsvItem item = ((Message) object).body(); - final String ark = item.getItemARK(); - - LOGGER.info(MessageCodes.AVPT_009, ark); - arkMap.put(ark, item); - }); - - updateCSV(message.body(), arkMap).onSuccess(csvFilePath -> { + final List> results = conversions.result().list(); + + // Filter the audiowaveform URLs out of the results and combine them all into a single JsonObject, + // which we'll use as a lookup table when updating the CSV with audiowaveform URLs + final JsonObject waveformUriMap = results.stream() + .filter(result -> result.body().getClass().equals(JsonObject.class)) + .map(msg -> (JsonObject) msg.body()) + .reduce(new JsonObject(), (jsonObject1, jsonObject2) -> jsonObject1.mergeIn(jsonObject2)); + + // Map ARKs to their corresponding CsvItem + final Map csvItemMap = results.stream() + .filter(result -> result.body().getClass().equals(CsvItem.class)) + .map(msg -> { + final CsvItem item = (CsvItem) msg.body(); + + LOGGER.info(MessageCodes.AVPT_009, item.getItemARK()); + return item; + }) + .collect(Collectors.toMap(item -> item.getItemARK(), item -> item)); + + updateCSV(message.body(), csvItemMap, waveformUriMap).onSuccess(csvFilePath -> { LOGGER.info(MessageCodes.AVPT_006, csvFilePath); message.reply(Op.SUCCESS); }).onFailure(error -> { @@ -103,13 +115,15 @@ public void start(final Promise aPromise) { } /** - * Update the CSV file with our newly created IIIF access URLs. + * Update the CSV file with our newly created IIIF access URLs and waveform URLs. * * @param aCsvFilePath The path to the existing CSV file - * @param aArkMap A map of ARKs for the items that have been processed + * @param aCsvItemMap A map of ARKs to the items that have been processed + * @param aWaveformMap A map of ARKs to audiowaveform URLs for the items that have been processed * @return The path of the new CSV file */ - private Future updateCSV(final String aCsvFilePath, final Map aArkMap) { + private Future updateCSV(final String aCsvFilePath, final Map aCsvItemMap, + final JsonObject aWaveformMap) { final String newCsvPath = FileUtils.stripExt(aCsvFilePath) + ".out"; // Would be re-watched if ext was .csv final Promise promise = Promise.promise(); @@ -118,59 +132,80 @@ private Future updateCSV(final String aCsvFilePath, final Map reader = new CsvClientImpl<>(csvReader, CsvItem.class); - // Open the CSV file we'll be writing to with the updated information + // Open the CSV file we'll be writing the updated information to try (FileWriter csvWriter = new FileWriter(new File(newCsvPath))) { final CsvClient writer = new CsvClientImpl<>(csvWriter); - final Header header = reader.readHeader(); - final int accessUrlIndex = getUrlAccessIndex(header); + final Header originalHeader = reader.readHeader(); + + final int originalRowSize = originalHeader.size(); + final int originalAccessUrlIndex = getUrlAccessIndex(originalHeader); + final int originalWaveformIndex = getWaveformIndex(originalHeader); + + final int rowSize; + final int accessUrlIndex; + final int waveformIndex; // Override the unusual out of the box defaults for the writer writer.setEscape('"').setQuote('"').setSeparator(','); - if (accessUrlIndex != -1) { - writer.writeHeader(header); + // Deal with the header row first; unless both columns exist already, we have to modify the underlying + // array (it's arguably a limitation of CSVeed that we have to do that) + if (originalWaveformIndex != -1 && originalAccessUrlIndex != -1) { + // Both columns exist already, so just use the header as-is; record some useful metadata first + rowSize = originalRowSize; + accessUrlIndex = originalAccessUrlIndex; + waveformIndex = originalWaveformIndex; + + writer.writeHeader(originalHeader); } else { - final String[] headerRow = new String[header.size() + 1]; + final String[] headerRow; - for (int index = 0; index < headerRow.length; index++) { - headerRow[index] = header.getName(index); + if (originalAccessUrlIndex != -1) { + // IIIF Access URL exists, but not Waveform; expand CSV by one column + rowSize = originalRowSize + 1; + accessUrlIndex = originalAccessUrlIndex; + waveformIndex = rowSize - 1; + } else { + // Neither IIIF Access URL or Waveform exist in the CSV yet (Waveform cannot possibly exist + // without IIIF Access URL); expand CSV by two columns + rowSize = originalRowSize + 2; + accessUrlIndex = rowSize - 2; + waveformIndex = rowSize - 1; + } + headerRow = new String[rowSize]; + + for (int index = 0; index < rowSize; index++) { + // Put the new header fields where they belong + if (accessUrlIndex == index) { + headerRow[index] = CsvItem.IIIF_ACCESS_URL_HEADER; + } else if (waveformIndex == index) { + headerRow[index] = CsvItem.WAVEFORM_HEADER; + } else { + // Copy values from the original header row to the new one + headerRow[index] = originalHeader.getName(index + 1); // Row and Header indices are 1-based + } } - headerRow[header.size()] = CsvItem.IIIF_ACCESS_URL_HEADER; writer.writeHeader(headerRow); } - // Stream through all the rows in our CSV file - reader.readRows().stream().forEach(row -> { - final String ark = row.get(CsvItem.ITEM_ARK_HEADER); - - final int columnCount = row.size(); - final String[] columns; - - // Check to see if this CSV file has an existing "IIIF Access URL" column - if (accessUrlIndex == -1) { - columns = new String[columnCount + 1]; - - for (int index = 0; index < columnCount; index++) { - columns[index] = row.get(index + 1); // Strangely, row is 1-based - } - - if (aArkMap.containsKey(ark)) { - columns[columnCount] = constructAccessURL(aArkMap.get(ark)); - } - } else { - columns = new String[columnCount]; - - for (int index = 0; index < columnCount; index++) { - if (accessUrlIndex != index) { - columns[index] = row.get(index + 1); // Strangely, row is 1-based - } else if (aArkMap.containsKey(ark)) { - columns[index] = constructAccessURL(aArkMap.get(ark)); - } + // Now, stream through all the non-header rows + reader.readRows().stream().forEach(originalRow -> { + final String ark = originalRow.get(CsvItem.ITEM_ARK_HEADER); + final CsvItem csvItem = aCsvItemMap.get(ark); + final String[] row = new String[rowSize]; + + for (int index = 0; index < rowSize; index++) { + if (accessUrlIndex == index) { + row[index] = aCsvItemMap.containsKey(ark) ? constructAccessURL(csvItem) : ""; + } else if (waveformIndex == index) { + row[index] = aWaveformMap.getString(ark, ""); + } else { + row[index] = originalRow.get(index + 1); } } - writer.writeRow(columns); + writer.writeRow(row); }); csvWriter.close(); @@ -191,12 +226,26 @@ private Future updateCSV(final String aCsvFilePath, final Map aPromise) { + final JsonObject config = config(); + final String[] cmd = { "which", AUDIOWAVEFORM }; + final String cmdline = String.join(SPACE, cmd); + final ProcessBuilder pb = new ProcessBuilder(cmd).redirectOutput(ProcessBuilder.Redirect.PIPE); + + final String configErrorMsg; + + // Make sure that audiowaveform is installed on the system + + try { + final Process which = pb.start(); + final int exitValue = which.waitFor(); + final InputStream stdout; + final String cmdResult; + + stdout = which.getInputStream(); + cmdResult = + LOGGER.getMessage(MessageCodes.AVPT_015, cmdline, exitValue, new String(stdout.readAllBytes())); + stdout.close(); + + if (0 == exitValue) { + LOGGER.debug(cmdResult); + } else { + LOGGER.error(cmdResult); + aPromise.fail(cmdResult); + + return; + } + } catch (final IOException | InterruptedException details) { + final String startErrorMsg = LOGGER.getMessage(MessageCodes.AVPT_016, cmdline, details); + + LOGGER.error(startErrorMsg); + aPromise.fail(details); + + return; + } + + mySourceDir = config.getString(Config.SOURCE_DIR); + + // Make sure that configuration and credentials for AWS S3 have been provided + + if (myAwsDefaultRegion == null) { + configErrorMsg = LOGGER.getMessage(MessageCodes.AVPT_018); + + LOGGER.error(configErrorMsg); + aPromise.fail(configErrorMsg); + + return; + } + + if (System.getenv(ProfileProperty.AWS_ACCESS_KEY_ID.toUpperCase()) == null || + System.getenv(ProfileProperty.AWS_SECRET_ACCESS_KEY.toUpperCase()) == null) { + configErrorMsg = LOGGER.getMessage(MessageCodes.AVPT_017); + + LOGGER.error(configErrorMsg); + aPromise.fail(configErrorMsg); + + return; + } + + myS3Bucket = config.getString(Config.AUDIOWAVEFORM_S3_BUCKET); + + if (myS3Bucket == null) { + configErrorMsg = LOGGER.getMessage(MessageCodes.AVPT_020); + + LOGGER.error(configErrorMsg); + aPromise.fail(configErrorMsg); + + return; + } + + myS3Client = S3AsyncClient.builder().region(Region.of(myAwsDefaultRegion)).build(); + + vertx.eventBus().consumer(getClass().getName()).handler(this::handle); + + aPromise.complete(); + } + + /** + * Transforms the source audio file at the given path into audiowaveform data, uploads that data to S3, and replies + * to the message with the URL for the data. If either the transformation or upload fails, sends back error details. + * + * @param aMessage A message with the file path of the audio file to transform + */ + private void handle(final Message aMessage) { + try { + final CsvItem csvItem = aMessage.body(); + final Path audioFilePath = AvPtUtils.getInputFilePath(csvItem, mySourceDir); + + audiowaveform(audioFilePath).onSuccess(s3ObjectData -> { + final String ark = csvItem.getItemARK(); + final String s3ObjectKey = StringUtils.format(S3_OBJECT_KEY_TEMPLATE, ark); + final PutObjectRequest req = PutObjectRequest.builder().bucket(myS3Bucket).key(s3ObjectKey).build(); + final AsyncRequestBody body = AsyncRequestBody.fromByteBuffer(s3ObjectData); + + // Store the audiowaveform data on S3 + myS3Client.putObject(req, body).whenComplete((resp, err) -> { + if (resp != null) { + // Success! + final String audiowaveformURL = StringUtils.format(AvPtConstants.AUDIOWAVEFORM_URL_TEMPLATE, + myS3Bucket, myAwsDefaultRegion, URLEncoder.encode(s3ObjectKey, StandardCharsets.UTF_8)); + + // Reply with a JsonObject associating the item ARK with the URL for the audiowaveform data + aMessage.reply(new JsonObject().put(csvItem.getItemARK(), audiowaveformURL)); + } else { + final String s3ErrorMsg = + LOGGER.getMessage(MessageCodes.AVPT_021, s3ObjectKey, err.getMessage()); + + // Since the sender (WatcherVerticle) just logs all errors, should be okay to use a single + // failureCode for all errors + aMessage.fail(Op.ERROR_CODE, s3ErrorMsg); + } + }); + }).onFailure(details -> { + aMessage.fail(Op.ERROR_CODE, details.getMessage()); + }); + } catch (final IOException details) { + aMessage.fail(Op.ERROR_CODE, details.getMessage()); + } + } + + /** + * Transforms the source audio file at the given path into binary audiowaveform data. + * + * @param anAudioFilePath The path to the audio file to transform + * @return A Future that is completed with a ByteBuffer containing the audiowaveform data + * @throws IOException if an I/O error occurs during the execution of the audiowaveform program + */ + private Future audiowaveform(final Path anAudioFilePath) throws IOException { + final Promise asyncResult = Promise.promise(); + final String[] cmd = { AUDIOWAVEFORM, "--input-filename", anAudioFilePath.toString(), "--output-format", "dat", + "--bits", "8" }; + final String cmdline = String.join(SPACE, cmd); + final ProcessBuilder pb = new ProcessBuilder(cmd).redirectOutput(ProcessBuilder.Redirect.PIPE); + + try { + pb.start().onExit().thenAccept(subprocess -> { + try (InputStream stdout = subprocess.getInputStream(); + InputStream stderr = subprocess.getErrorStream()) { + + final int exitValue = subprocess.exitValue(); + + if (0 == exitValue) { + // Redact the binary audiowaveform data for logging + final String cmdResultMsg = LOGGER.getMessage(MessageCodes.AVPT_015, cmdline, exitValue, + "[binary audiowaveform data]"); + + LOGGER.debug(cmdResultMsg); + + asyncResult.complete(ByteBuffer.wrap(stdout.readAllBytes())); + } else { + final String errorOutput = new String(stderr.readAllBytes()); + + asyncResult.fail(LOGGER.getMessage(MessageCodes.AVPT_015, cmdline, exitValue, errorOutput)); + } + } catch (final IOException details) { + asyncResult.fail(LOGGER.getMessage(MessageCodes.AVPT_016, cmdline, details)); + } + }); + } catch (final IOException details) { + throw new IOException(LOGGER.getMessage(MessageCodes.AVPT_016, cmdline, details)); + } + + return asyncResult.future(); + } +} diff --git a/src/main/resources/av-pairtree_messages.xml b/src/main/resources/av-pairtree_messages.xml index 5421b58..b0827f6 100644 --- a/src/main/resources/av-pairtree_messages.xml +++ b/src/main/resources/av-pairtree_messages.xml @@ -5,6 +5,7 @@ edu.ucla.library.avpairtree.MessageCodes + {}: {} Server started and listening at port {} Verticle undeployed for testing purposes: {} [{}] Running test: {} @@ -19,5 +20,12 @@ {} worker pool size set to '{}' workers Index for IIIF access URL doesn't correspond to number of substitution patterns: {} IIIF access URL pattern doesn't contain a supported number of substitution patterns + Command '{}' exited with code {}. Output: {} + Something went wrong while running command '{}': {} + The environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be set + The environment variable AWS_DEFAULT_REGION must be set + Using S3 bucket '{}' for audiowaveform storage + The system property 'audiowaveform.s3.bucket' must be set + Unable to upload audiowaveform for item '{}' to S3: {} diff --git a/src/main/resources/logback.xml b/src/main/resources/logback.xml index 8d1ccae..2319d87 100644 --- a/src/main/resources/logback.xml +++ b/src/main/resources/logback.xml @@ -32,6 +32,9 @@ + + + diff --git a/src/test/java/edu/ucla/library/avpairtree/utils/TestConstants.java b/src/test/java/edu/ucla/library/avpairtree/utils/TestConstants.java index 8962911..37c21eb 100644 --- a/src/test/java/edu/ucla/library/avpairtree/utils/TestConstants.java +++ b/src/test/java/edu/ucla/library/avpairtree/utils/TestConstants.java @@ -55,6 +55,11 @@ public final class TestConstants { */ public static final String PROCESSED = "Processed"; + /** + * The item's waveform property from the JSON message format. + */ + public static final String WAVEFORM = "Waveform"; + /** * The ARKs from the test fixtures. */ diff --git a/src/test/java/edu/ucla/library/avpairtree/verticles/AbstractAvPtTest.java b/src/test/java/edu/ucla/library/avpairtree/verticles/AbstractAvPtTest.java index 9549f75..b698011 100644 --- a/src/test/java/edu/ucla/library/avpairtree/verticles/AbstractAvPtTest.java +++ b/src/test/java/edu/ucla/library/avpairtree/verticles/AbstractAvPtTest.java @@ -56,6 +56,11 @@ public abstract class AbstractAvPtTest { */ protected String myTestAvServer; + /** + * The name of the S3 bucket we use when testing. + */ + protected String myTestS3Bucket; + /** * Sets up the test. * @@ -69,6 +74,7 @@ public void setUp(final TestContext aContext) { ConfigRetriever.create(vertx).getConfig().onSuccess(config -> { myTestAvServer = config.getString(Config.ACCESS_URL_PATTERN); + myTestS3Bucket = config.getString(Config.AUDIOWAVEFORM_S3_BUCKET); myPtDir = config.getString(Config.OUTPUT_DIR); myPort = PortUtils.getPort(); options.setConfig(config.put(Config.HTTP_PORT, myPort)); diff --git a/src/test/java/edu/ucla/library/avpairtree/verticles/WaveformVerticleTest.java b/src/test/java/edu/ucla/library/avpairtree/verticles/WaveformVerticleTest.java new file mode 100644 index 0000000..d3b5c11 --- /dev/null +++ b/src/test/java/edu/ucla/library/avpairtree/verticles/WaveformVerticleTest.java @@ -0,0 +1,92 @@ +package edu.ucla.library.avpairtree.verticles; + +import static org.junit.Assert.assertEquals; + +import java.net.URI; + +import org.junit.Test; +import org.junit.runner.RunWith; + +import info.freelibrary.util.Logger; +import info.freelibrary.util.LoggerFactory; + +import edu.ucla.library.avpairtree.CsvItem; +import edu.ucla.library.avpairtree.MessageCodes; + +import io.vertx.core.Vertx; +import io.vertx.core.buffer.Buffer; +import io.vertx.core.http.HttpMethod; +import io.vertx.core.json.JsonObject; +import io.vertx.ext.unit.Async; +import io.vertx.ext.unit.TestContext; +import io.vertx.ext.unit.junit.VertxUnitRunner; + +/** + * Tests the waveform verticle. + */ +@RunWith(VertxUnitRunner.class) +public class WaveformVerticleTest extends AbstractAvPtTest { + + private static final Logger LOGGER = LoggerFactory.getLogger(WaveformVerticleTest.class, MessageCodes.BUNDLE); + + /** + * Tests the generation of audiowaveform data and its upload to S3. + * + * @param aContext A test context + */ + @Test + public void testWaveformGenerationAndS3Storage(final TestContext aContext) { + + LOGGER.debug(MessageCodes.AVPT_003, myNames.getMethodName()); + + final Async asyncTask = aContext.async(); + final Vertx vertx = myContext.vertx(); + final CsvItem csvItem = new CsvItem(); + + // Create the CSV item we want the waveform verticle to process + csvItem.setItemARK("ark:/21198/zz002dvxmm"); + csvItem.setFilePath("soul/audio/uclapasc.wav"); + + // TODO: mock the S3 bucket + vertx.eventBus().request(WaveformVerticle.class.getName(), csvItem).onSuccess(transformation -> { + // Compare the data retrieved from S3 with a local test fixture + final URI audiowaveformURL = URI.create(transformation.body().getString(csvItem.getItemARK())); + final String host = audiowaveformURL.getHost(); + final String path = audiowaveformURL.getPath(); + + vertx.createHttpClient().request(HttpMethod.GET, host, path).onSuccess(req -> { + req.send().onSuccess(response -> { + response.body().onSuccess(resp -> { + final Buffer expected = + vertx.fileSystem().readFileBlocking("src/test/resources/soul/audio/uclapasc.dat"); + final Buffer actual = resp; + + try { + assertEquals(expected, actual); + } catch (final AssertionError details) { + LOGGER.error(details, details.getMessage()); + aContext.fail(); + } finally { + // TODO: clean up the S3 bucket + asyncTask.complete(); + } + }); + }).onFailure(error -> { + LOGGER.error(error, error.getMessage()); + aContext.fail(); + }); + }).onFailure(error -> { + LOGGER.error(error, error.getMessage()); + aContext.fail(); + }); + }).onFailure(error -> { + LOGGER.error(error, error.getMessage()); + aContext.fail(); + }); + } + + @Override + protected Logger getLogger() { + return LOGGER; + } +} diff --git a/src/test/resources/logback-test.xml b/src/test/resources/logback-test.xml index 8d1ccae..2319d87 100644 --- a/src/test/resources/logback-test.xml +++ b/src/test/resources/logback-test.xml @@ -32,6 +32,9 @@ + + + diff --git a/src/test/resources/soul/audio/uclapasc.dat b/src/test/resources/soul/audio/uclapasc.dat new file mode 100644 index 0000000000000000000000000000000000000000..8a4ed8bbbc04a4c3ffe72d6c62287fbcb94293a4 GIT binary patch literal 5188 zcmdUz=W^pn5`(JfmWA!zQQ6E;WzJ5H|>*L4j^FiNlt6B5cNLc^6@^pV(V}TbawuFFDJ2t)_MF>sk8S5%$+wqA`+g*Vd@7Re8Ti=ljRwyq;IH zGk?vscXkc?2CMi*_RsIgb7YgX@t3Y>`Q@d>S6m--NIy8LyvZ}t&o`koG@;?m6pS9fzVYY4_+vDMqiY=A`NkUqM*6JCoDplxnzwwNoWYxY z=B$~I7TkGbI3#F=hG^zZ?joV{`e2X*=JU8k}5*0-Xw1Gk)RM`ajL8${4 zgb#ai52`KtZ4NLN%nHXs^RH8*HFqVcBj^Up{vuh@ClEa`Lch%VwUX9=C%6*P$PmFH z!qRxecf@0Z=6I7MF7X$5^LxJy@Of0m+?>s~HzWkUg>i=7BwKvQ6tcb1-5x z@ysQ^;i<{+tk3{9gp-EI2thsZW=smAz!q=(&Knp>o)JYb5P=iM;cu-dWCx07OB|J* zh6oza&v>DR@2xvJ#MbLw11iCSa>W^V5{k)cXC}vHXN2w1AK&36PrgMuqZC<2aL0UQ zrtaYp<~yA*olf!So7gygq6{IvFv8%Xbz~Bd>?j{hj4Y5nq2uTX6BC&sM6_6Dgb;N6 z(u(|+WqiTovKwCqebRbkhW&U_7&OShM}dh9xr;Z%3itpIC?D~hK{|N$41Sb%(8KTW z(4^6@(m*aNPgxC%#S?H1_Sq~K1Bipxvp@ScqI`3)3{MLQ$`|A#4~BplNhp>m9suuJ&_ z|5;X&HTd>tY-M3|85YgSDthy0qF)rA@a!YA_A@^dsB7%& z4ss%_@_SZ>tzttQkeHB5aT6;_bufs*m1Cab&~Lh|Nv_)hsc>~tXIuK5F&k zHF=jdVZ-`$()xB(RKd8G?ei+b-$;l5L9+ah)=8IDn;~g~3YpBz>ZFDxTX4fzbY`@2 zU3CMl0c+?kT~(PTaAnAcJ^=^90@y+>qjYbp7Z#AqWV~|RRxa=NCds)zN)vR!7$iprK*<5a@hzoC=y^|`DQZ(JA^?pRoA6=5z0X$k`uHEC-EOx zq6VgeSQKWW+7Nd!&432j0SChNVtmyJx*Kv_enI1DeHXUXIZ}7T5O`MjB7>>7#GrRz z6R%H*Y1o1?@Rq>s%0)cY2MzcU_PJM&CGAuZ)DOu|l8(3FS?-Af{7anZS7tmcD@Q8m z=n`=vX}ceorN;|kj5UaZ9#YaMhTQ%gBOh<00mXrjyqt zqucv77HF(~4FrX%Cq!=1T9t;Xji0?K#2pT)wahXrO!cKWXc8&7J5>=0Pu!$s~A06Zr&(TfSSlX&LwTLCILKnWXo>Vk%$_ucd zZiGI;`Op)4gqghqxq~%myE5SCCZ4oxE`Gd`Uy}JVi+|px?aOjxyDZO6Uz;CM43;PblMR@|sJa#Ttrgj-< z`fz)+E7*O<=L0cC=Yi^gpMxg2U`=Q>z zVOhDaShq;)yh+s!!G@{jPVSW*=tx&XMP6`5U4{wZCUQfaEOit8S&te3dIoWd*aC#f zugVd$vYoc=FhI5X68X?_mNH#FVQ0i$V3LVdL4|59^`Q&k*hKnW4%hUOU9xifzG-4~ zPC8HgNX@77nS%kX2JaSjw*GePV8lICIw6C-(_MV1j&#kNz^ui9>UhAHa%I^AY3HS$ z)VbTe?)`z7f&g}3%0Jt;e(0C%0X^i=9ely}^o&JXW@UEG>a0lM{e)-yVsS}S;1K*q zrmSzEz_5y04ahZglaq2$zw7C7Vm(muD8`a5HcRkFMO+4INE?;vx`K}C-h}kAR51KR zcOfPMmzoDUzYbwh7O|>WZ`onbevub>8CO|_#_A5LjOV<{iw!k#qx%9&s;rKP4OZHS z*>2{xw%E3NXx{Tq+NhhNuSQQQ+_p@+h+WL)k{9r5k=0?jKjnYzPut>f*}reg!!@s{ zj2GCKIrNTXH0!7fvtCt3VhI}CET{Tq!>0J-yuH4je}DV!^mN^D?yh^#{d|2Yh3!$p1(bvt}Yq2f%m+g6XIh2Rfq1v^Z>(l%0@`8Oo&b#B&@%iodw`Vfq<+Q(U zPtWIA`RAS- z+a7msyE4CI*L{&3pRU_;jz71@r_14Ts1x>UX_Z!q)jGNI=CZ4j;!r+c^S8rwS7)b} zKYu&F9QVi9zn-r9YhLc!?c1i@y**v?>*ly2qxVIAdbvIyUysj4Qbnwl7u)NW6?f;& z8P8ldZAzu}?7kIjz_|45dP66a&|gy7$ZrrshZ_&XY78rCG1&soki&Eb+Bw7g^t|ep zY_IR%)X%FMXV*wwQU6P4t!yUK==6wu-1(E_debATfqivbChIeGwbT{utko58Oe^+R znhj=D7R%|Ggta#8=E5|*{YQ=QRC&hSOdNapIhWnqhW n%gHdZMtq1f7hV=Ob1d9!@AUlr^E)}M-r672F>U-ht>b?KZ@0m< literal 0 HcmV?d00001 diff --git a/src/test/resources/test-config.properties b/src/test/resources/test-config.properties index b8fbd39..249cfd0 100644 --- a/src/test/resources/test-config.properties +++ b/src/test/resources/test-config.properties @@ -28,3 +28,6 @@ iiif.access.url.id.index = 1 # The number of threads working on media file conversions conversion.workers = 2 + +# The S3 bucket to use for depositing binary audiowaveform data files +audiowaveform.s3.bucket = test-audiowaveform-resources From 1c9be9b1a4d9fa869de0a9b4d12542b7195f3490 Mon Sep 17 00:00:00 2001 From: "Mark A. Matney, Jr" Date: Wed, 16 Jun 2021 12:19:02 -0700 Subject: [PATCH 02/10] Install audiowaveform with GitHub Actions --- .github/workflows/build.yml | 2 ++ .github/workflows/release.yml | 2 ++ 2 files changed, 4 insertions(+) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index c33a835..9b36991 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -24,6 +24,8 @@ jobs: with: distribution: 'zulu' java-version: ${{ matrix.java }} + - name: Install Audio Waveform Image Generator + run: sudo add-apt-repository ppa:chris-needham/ppa && sudo apt-get install audiowaveform # If running locally in act, install Maven - name: Set up Maven if needed if: ${{ env.ACT }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 75df7a0..88af000 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -20,6 +20,8 @@ jobs: uses: actions/setup-java@d202f5dbf7256730fb690ec59f6381650114feb2 # v1 with: java-version: 11 + - name: Install Audio Waveform Image Generator + run: sudo add-apt-repository ppa:chris-needham/ppa && sudo apt-get install audiowaveform # If running locally in act, install Maven - name: Set up Maven if needed if: ${{ env.ACT }} From cded4c9c555e90edabc614c99afafacba4ff8fdc Mon Sep 17 00:00:00 2001 From: "Mark A. Matney, Jr" Date: Wed, 16 Jun 2021 13:07:57 -0700 Subject: [PATCH 03/10] Don't overwrite IIIF Access URL for images in the input CSV --- .../ucla/library/avpairtree/verticles/WatcherVerticle.java | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/main/java/edu/ucla/library/avpairtree/verticles/WatcherVerticle.java b/src/main/java/edu/ucla/library/avpairtree/verticles/WatcherVerticle.java index 0b6a513..e26b9fa 100644 --- a/src/main/java/edu/ucla/library/avpairtree/verticles/WatcherVerticle.java +++ b/src/main/java/edu/ucla/library/avpairtree/verticles/WatcherVerticle.java @@ -197,7 +197,12 @@ private Future updateCSV(final String aCsvFilePath, final Map Date: Wed, 16 Jun 2021 13:37:54 -0700 Subject: [PATCH 04/10] Set AWS credentials for GitHub Actions --- .github/workflows/build.yml | 3 +++ .github/workflows/release.yml | 3 +++ 2 files changed, 6 insertions(+) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 9b36991..77ec0f4 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -11,6 +11,9 @@ jobs: name: Maven PR Builder (JDK ${{ matrix.java }}) runs-on: ubuntu-latest env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_DEFAULT_REGION: us-west-2 + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} MAVEN_CACHE_KEY: ${{ secrets.MAVEN_CACHE_KEY }} strategy: matrix: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 88af000..745e04f 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -11,6 +11,9 @@ jobs: runs-on: ubuntu-latest env: AUTORELEASE_ARTIFACT: ${{ secrets.AUTORELEASE_ARTIFACT }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_DEFAULT_REGION: us-west-2 + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} SKIP_JAR_DEPLOYMENT: ${{ secrets.SKIP_JAR_DEPLOYMENT }} MAVEN_CACHE_KEY: ${{ secrets.MAVEN_CACHE_KEY }} steps: From deff0ca396daa8a9d63e31d9cc17cb8bffa72846 Mon Sep 17 00:00:00 2001 From: "Mark A. Matney, Jr" Date: Wed, 16 Jun 2021 13:41:25 -0700 Subject: [PATCH 05/10] Don't set AWS credentials in the POM --- pom.xml | 5 ----- 1 file changed, 5 deletions(-) diff --git a/pom.xml b/pom.xml index 2de969d..701941b 100644 --- a/pom.xml +++ b/pom.xml @@ -262,11 +262,6 @@ io.vertx.core.logging.SLF4JLogDelegateFactory ${project.basedir}/target/test-classes/test-config.properties - - ${avpt.s3.access_key} - ${avpt.s3.region} - ${avpt.s3.secret_key} - 1 ${jacoco.agent.arg} From 05b141eee8941ec063656915e25d58ca8c0988d7 Mon Sep 17 00:00:00 2001 From: "Mark A. Matney, Jr" Date: Wed, 16 Jun 2021 14:45:52 -0700 Subject: [PATCH 06/10] Make the Maven build succeed both locally and on GitHub Actions --- .github/workflows/build.yml | 1 + .github/workflows/release.yml | 1 + pom.xml | 25 ++++++++++++++++++++++++- 3 files changed, 26 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 77ec0f4..edfdcec 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -57,3 +57,4 @@ jobs: maven_args: > -V -ntp -Dorg.slf4j.simpleLogger.log.net.sourceforge.pmd=error -Dsurefire.skipAfterFailureCount=1 -Dfailsafe.skipAfterFailureCount=1 + -DghActions diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 745e04f..b996a3c 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -67,3 +67,4 @@ jobs: -Ddocker.registry.username=${{ secrets.DOCKER_USERNAME }} -Ddocker.registry.account=${{ secrets.DOCKER_REGISTRY_ACCOUNT}} -Ddocker.registry.password=${{ secrets.DOCKER_PASSWORD }} + -DghActions diff --git a/pom.xml b/pom.xml index 701941b..eda2f36 100644 --- a/pom.xml +++ b/pom.xml @@ -391,8 +391,31 @@ - + + + surefire-config-aws-environment-variables + + + !ghActions + + + + + + maven-surefire-plugin + + + ${avpt.s3.access_key} + ${avpt.s3.region} + ${avpt.s3.secret_key} + + + + + + + freelib-parent info.freelibrary From ddff8314cbf1bda7effa4540db03940eb7502f18 Mon Sep 17 00:00:00 2001 From: "Mark A. Matney, Jr" Date: Wed, 16 Jun 2021 15:07:21 -0700 Subject: [PATCH 07/10] Configure audiowaveform S3 bucket with env var --- .github/workflows/build.yml | 1 + .github/workflows/release.yml | 1 + ARCHITECTURE.md | 1 - README.md | 2 ++ pom.xml | 1 + src/main/java/edu/ucla/library/avpairtree/Config.java | 4 ++-- .../ucla/library/avpairtree/verticles/WaveformVerticle.java | 2 +- .../ucla/library/avpairtree/verticles/AbstractAvPtTest.java | 2 +- src/test/resources/test-config.properties | 3 --- 9 files changed, 9 insertions(+), 8 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index edfdcec..110d2a0 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -11,6 +11,7 @@ jobs: name: Maven PR Builder (JDK ${{ matrix.java }}) runs-on: ubuntu-latest env: + AUDIOWAVEFORM_S3_BUCKET: ${{ secrets.AUDIOWAVEFORM_S3_BUCKET }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_DEFAULT_REGION: us-west-2 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b996a3c..b6bc3e2 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -11,6 +11,7 @@ jobs: runs-on: ubuntu-latest env: AUTORELEASE_ARTIFACT: ${{ secrets.AUTORELEASE_ARTIFACT }} + AUDIOWAVEFORM_S3_BUCKET: ${{ secrets.AUDIOWAVEFORM_S3_BUCKET }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_DEFAULT_REGION: us-west-2 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index d878708..71a6871 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -69,7 +69,6 @@ The properties and default values for an av-pairtree configuration file are as f | audio.channels | The number of channels in the audio stream | 2 | | iiif.access.url | The URL pattern into which to insert the Pairtree path | N/A | | conversion.workers | The number of cores to use for audio file conversion | 2 | -| audiowaveform.s3.bucket | The S3 bucket to use for depositing binary audiowaveform data files | test-audiowaveform-resources | ## Documentation diff --git a/README.md b/README.md index 37ed9bb..7349d11 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,7 @@ In order to run the integration tests that use AWS S3, you should have an entry myAwsAccessKey + myAwsS3Bucket us-west-2 myAwsSecretKey @@ -70,6 +71,7 @@ To run av-pairtree from the Jar file, one must set AWS S3 credentials and then r ```bash #!/bin/bash +export AUDIOWAVEFORM_S3_BUCKET=myAwsS3Bucket export AWS_ACCESS_KEY_ID=myAwsAccessKey export AWS_DEFAULT_REGION=us-west-2 export AWS_SECRET_ACCESS_KEY=myAwsSecretKey diff --git a/pom.xml b/pom.xml index eda2f36..6982ba7 100644 --- a/pom.xml +++ b/pom.xml @@ -406,6 +406,7 @@ maven-surefire-plugin + ${avpt.s3.bucket} ${avpt.s3.access_key} ${avpt.s3.region} ${avpt.s3.secret_key} diff --git a/src/main/java/edu/ucla/library/avpairtree/Config.java b/src/main/java/edu/ucla/library/avpairtree/Config.java index b23c39e..273e64d 100644 --- a/src/main/java/edu/ucla/library/avpairtree/Config.java +++ b/src/main/java/edu/ucla/library/avpairtree/Config.java @@ -78,9 +78,9 @@ public final class Config { public static final String CONVERSION_WORKERS = "conversion.workers"; /** - * The configuration property for the S3 bucket for audio waveforms. + * The environment variable for the S3 bucket for audio waveforms. */ - public static final String AUDIOWAVEFORM_S3_BUCKET = "audiowaveform.s3.bucket"; + public static final String AUDIOWAVEFORM_S3_BUCKET = "AUDIOWAVEFORM_S3_BUCKET"; // Constant classes should have private constructors. private Config() { diff --git a/src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java b/src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java index d99b9dd..241470e 100644 --- a/src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java +++ b/src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java @@ -115,7 +115,7 @@ public void start(final Promise aPromise) { return; } - myS3Bucket = config.getString(Config.AUDIOWAVEFORM_S3_BUCKET); + myS3Bucket = System.getenv(Config.AUDIOWAVEFORM_S3_BUCKET); if (myS3Bucket == null) { configErrorMsg = LOGGER.getMessage(MessageCodes.AVPT_020); diff --git a/src/test/java/edu/ucla/library/avpairtree/verticles/AbstractAvPtTest.java b/src/test/java/edu/ucla/library/avpairtree/verticles/AbstractAvPtTest.java index b698011..f067d7d 100644 --- a/src/test/java/edu/ucla/library/avpairtree/verticles/AbstractAvPtTest.java +++ b/src/test/java/edu/ucla/library/avpairtree/verticles/AbstractAvPtTest.java @@ -74,7 +74,7 @@ public void setUp(final TestContext aContext) { ConfigRetriever.create(vertx).getConfig().onSuccess(config -> { myTestAvServer = config.getString(Config.ACCESS_URL_PATTERN); - myTestS3Bucket = config.getString(Config.AUDIOWAVEFORM_S3_BUCKET); + myTestS3Bucket = System.getenv(Config.AUDIOWAVEFORM_S3_BUCKET); myPtDir = config.getString(Config.OUTPUT_DIR); myPort = PortUtils.getPort(); options.setConfig(config.put(Config.HTTP_PORT, myPort)); diff --git a/src/test/resources/test-config.properties b/src/test/resources/test-config.properties index 249cfd0..b8fbd39 100644 --- a/src/test/resources/test-config.properties +++ b/src/test/resources/test-config.properties @@ -28,6 +28,3 @@ iiif.access.url.id.index = 1 # The number of threads working on media file conversions conversion.workers = 2 - -# The S3 bucket to use for depositing binary audiowaveform data files -audiowaveform.s3.bucket = test-audiowaveform-resources From 2b553d8c5a95a74b653c933f213817e8fcc4abfd Mon Sep 17 00:00:00 2001 From: "Mark A. Matney, Jr" Date: Wed, 16 Jun 2021 15:12:56 -0700 Subject: [PATCH 08/10] Use ConfigRetriever to read env vars --- .../library/avpairtree/verticles/WaveformVerticle.java | 10 ++++++---- .../library/avpairtree/verticles/AbstractAvPtTest.java | 2 +- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java b/src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java index 241470e..cbef5e0 100644 --- a/src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java +++ b/src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java @@ -45,7 +45,7 @@ public final class WaveformVerticle extends AbstractVerticle { private static final String SPACE = " "; - private final String myAwsDefaultRegion = System.getenv("AWS_DEFAULT_REGION"); + private String myAwsDefaultRegion; private S3AsyncClient myS3Client; @@ -96,6 +96,8 @@ public void start(final Promise aPromise) { // Make sure that configuration and credentials for AWS S3 have been provided + myAwsDefaultRegion = config.getString("AWS_DEFAULT_REGION"); + if (myAwsDefaultRegion == null) { configErrorMsg = LOGGER.getMessage(MessageCodes.AVPT_018); @@ -105,8 +107,8 @@ public void start(final Promise aPromise) { return; } - if (System.getenv(ProfileProperty.AWS_ACCESS_KEY_ID.toUpperCase()) == null || - System.getenv(ProfileProperty.AWS_SECRET_ACCESS_KEY.toUpperCase()) == null) { + if (config.getString(ProfileProperty.AWS_ACCESS_KEY_ID.toUpperCase()) == null || + config.getString(ProfileProperty.AWS_SECRET_ACCESS_KEY.toUpperCase()) == null) { configErrorMsg = LOGGER.getMessage(MessageCodes.AVPT_017); LOGGER.error(configErrorMsg); @@ -115,7 +117,7 @@ public void start(final Promise aPromise) { return; } - myS3Bucket = System.getenv(Config.AUDIOWAVEFORM_S3_BUCKET); + myS3Bucket = config.getString(Config.AUDIOWAVEFORM_S3_BUCKET); if (myS3Bucket == null) { configErrorMsg = LOGGER.getMessage(MessageCodes.AVPT_020); diff --git a/src/test/java/edu/ucla/library/avpairtree/verticles/AbstractAvPtTest.java b/src/test/java/edu/ucla/library/avpairtree/verticles/AbstractAvPtTest.java index f067d7d..b698011 100644 --- a/src/test/java/edu/ucla/library/avpairtree/verticles/AbstractAvPtTest.java +++ b/src/test/java/edu/ucla/library/avpairtree/verticles/AbstractAvPtTest.java @@ -74,7 +74,7 @@ public void setUp(final TestContext aContext) { ConfigRetriever.create(vertx).getConfig().onSuccess(config -> { myTestAvServer = config.getString(Config.ACCESS_URL_PATTERN); - myTestS3Bucket = System.getenv(Config.AUDIOWAVEFORM_S3_BUCKET); + myTestS3Bucket = config.getString(Config.AUDIOWAVEFORM_S3_BUCKET); myPtDir = config.getString(Config.OUTPUT_DIR); myPort = PortUtils.getPort(); options.setConfig(config.put(Config.HTTP_PORT, myPort)); From 07f89ad84b880d5556078bcb581582e0d7108b8b Mon Sep 17 00:00:00 2001 From: "Mark A. Matney, Jr" Date: Wed, 16 Jun 2021 16:33:49 -0700 Subject: [PATCH 09/10] Use Vert.x WebClient instead of HttpClient --- .../verticles/WaveformVerticleTest.java | 39 +++++++------------ 1 file changed, 14 insertions(+), 25 deletions(-) diff --git a/src/test/java/edu/ucla/library/avpairtree/verticles/WaveformVerticleTest.java b/src/test/java/edu/ucla/library/avpairtree/verticles/WaveformVerticleTest.java index d3b5c11..a7e1b25 100644 --- a/src/test/java/edu/ucla/library/avpairtree/verticles/WaveformVerticleTest.java +++ b/src/test/java/edu/ucla/library/avpairtree/verticles/WaveformVerticleTest.java @@ -2,8 +2,6 @@ import static org.junit.Assert.assertEquals; -import java.net.URI; - import org.junit.Test; import org.junit.runner.RunWith; @@ -15,11 +13,11 @@ import io.vertx.core.Vertx; import io.vertx.core.buffer.Buffer; -import io.vertx.core.http.HttpMethod; import io.vertx.core.json.JsonObject; import io.vertx.ext.unit.Async; import io.vertx.ext.unit.TestContext; import io.vertx.ext.unit.junit.VertxUnitRunner; +import io.vertx.ext.web.client.WebClient; /** * Tests the waveform verticle. @@ -50,31 +48,22 @@ public void testWaveformGenerationAndS3Storage(final TestContext aContext) { // TODO: mock the S3 bucket vertx.eventBus().request(WaveformVerticle.class.getName(), csvItem).onSuccess(transformation -> { // Compare the data retrieved from S3 with a local test fixture - final URI audiowaveformURL = URI.create(transformation.body().getString(csvItem.getItemARK())); - final String host = audiowaveformURL.getHost(); - final String path = audiowaveformURL.getPath(); + final String audiowaveformURL = transformation.body().getString(csvItem.getItemARK()); - vertx.createHttpClient().request(HttpMethod.GET, host, path).onSuccess(req -> { - req.send().onSuccess(response -> { - response.body().onSuccess(resp -> { - final Buffer expected = - vertx.fileSystem().readFileBlocking("src/test/resources/soul/audio/uclapasc.dat"); - final Buffer actual = resp; + WebClient.create(vertx).getAbs(audiowaveformURL).send().onSuccess(resp -> { + final Buffer expected = + vertx.fileSystem().readFileBlocking("src/test/resources/soul/audio/uclapasc.dat"); + final Buffer actual = resp.body(); - try { - assertEquals(expected, actual); - } catch (final AssertionError details) { - LOGGER.error(details, details.getMessage()); - aContext.fail(); - } finally { - // TODO: clean up the S3 bucket - asyncTask.complete(); - } - }); - }).onFailure(error -> { - LOGGER.error(error, error.getMessage()); + try { + assertEquals(expected, actual); + } catch (final AssertionError details) { + LOGGER.error(details, details.getMessage()); aContext.fail(); - }); + } finally { + // TODO: clean up the S3 bucket + asyncTask.complete(); + } }).onFailure(error -> { LOGGER.error(error, error.getMessage()); aContext.fail(); From 794177adc7fb1bd7ac44c41ce070670834270210 Mon Sep 17 00:00:00 2001 From: "Mark A. Matney, Jr" Date: Wed, 16 Jun 2021 17:23:35 -0700 Subject: [PATCH 10/10] Configure audiowaveform S3 object URL template with env var --- .github/workflows/build.yml | 1 + .github/workflows/release.yml | 3 ++- README.md | 2 ++ pom.xml | 1 + .../library/avpairtree/AvPtConstants.java | 5 ----- .../edu/ucla/library/avpairtree/Config.java | 5 +++++ .../verticles/WaveformVerticle.java | 20 +++++++++++++++---- src/main/resources/av-pairtree_messages.xml | 5 +++-- 8 files changed, 30 insertions(+), 12 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 110d2a0..482925a 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -12,6 +12,7 @@ jobs: runs-on: ubuntu-latest env: AUDIOWAVEFORM_S3_BUCKET: ${{ secrets.AUDIOWAVEFORM_S3_BUCKET }} + AUDIOWAVEFORM_S3_OBJECT_URL_TEMPLATE: ${{ secrets.AUDIOWAVEFORM_S3_OBJECT_URL_TEMPLATE }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_DEFAULT_REGION: us-west-2 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b6bc3e2..fe43289 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -10,8 +10,9 @@ jobs: name: Maven Artifact Publisher (JDK 11) runs-on: ubuntu-latest env: - AUTORELEASE_ARTIFACT: ${{ secrets.AUTORELEASE_ARTIFACT }} AUDIOWAVEFORM_S3_BUCKET: ${{ secrets.AUDIOWAVEFORM_S3_BUCKET }} + AUDIOWAVEFORM_S3_OBJECT_URL_TEMPLATE: ${{ secrets.AUDIOWAVEFORM_S3_OBJECT_URL_TEMPLATE }} + AUTORELEASE_ARTIFACT: ${{ secrets.AUTORELEASE_ARTIFACT }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_DEFAULT_REGION: us-west-2 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} diff --git a/README.md b/README.md index 7349d11..416caee 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,7 @@ In order to run the integration tests that use AWS S3, you should have an entry myAwsAccessKey myAwsS3Bucket + https://${avpt.s3.bucket}.s3-${avpt.s3.region}.amazonaws.com/{} us-west-2 myAwsSecretKey @@ -72,6 +73,7 @@ To run av-pairtree from the Jar file, one must set AWS S3 credentials and then r #!/bin/bash export AUDIOWAVEFORM_S3_BUCKET=myAwsS3Bucket +export AUDIOWAVEFORM_S3_OBJECT_URL_TEMPLATE=http://example.com/{} export AWS_ACCESS_KEY_ID=myAwsAccessKey export AWS_DEFAULT_REGION=us-west-2 export AWS_SECRET_ACCESS_KEY=myAwsSecretKey diff --git a/pom.xml b/pom.xml index 6982ba7..ea7135f 100644 --- a/pom.xml +++ b/pom.xml @@ -407,6 +407,7 @@ ${avpt.s3.bucket} + ${avpt.s3.object.url.template} ${avpt.s3.access_key} ${avpt.s3.region} ${avpt.s3.secret_key} diff --git a/src/main/java/edu/ucla/library/avpairtree/AvPtConstants.java b/src/main/java/edu/ucla/library/avpairtree/AvPtConstants.java index 322fdcc..1e3f31b 100644 --- a/src/main/java/edu/ucla/library/avpairtree/AvPtConstants.java +++ b/src/main/java/edu/ucla/library/avpairtree/AvPtConstants.java @@ -21,11 +21,6 @@ public final class AvPtConstants { */ public static final String SYSTEM_TMP_DIR = System.getProperty("java.io.tmpdir"); - /** - * The template string for AWS S3 resource URLs. The slots are (in order): bucket name, region, and object key. - */ - public static final String AUDIOWAVEFORM_URL_TEMPLATE = "https://{}.s3-{}.amazonaws.com/{}"; - /* * Constant classes have private constructors. */ diff --git a/src/main/java/edu/ucla/library/avpairtree/Config.java b/src/main/java/edu/ucla/library/avpairtree/Config.java index 273e64d..17a721a 100644 --- a/src/main/java/edu/ucla/library/avpairtree/Config.java +++ b/src/main/java/edu/ucla/library/avpairtree/Config.java @@ -82,6 +82,11 @@ public final class Config { */ public static final String AUDIOWAVEFORM_S3_BUCKET = "AUDIOWAVEFORM_S3_BUCKET"; + /** + * The environment variable for the S3 object URL template for audio waveforms. + */ + public static final String AUDIOWAVEFORM_S3_OBJECT_URL_TEMPLATE = "AUDIOWAVEFORM_S3_OBJECT_URL_TEMPLATE"; + // Constant classes should have private constructors. private Config() { } diff --git a/src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java b/src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java index cbef5e0..fb8e38a 100644 --- a/src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java +++ b/src/main/java/edu/ucla/library/avpairtree/verticles/WaveformVerticle.java @@ -11,7 +11,6 @@ import info.freelibrary.util.LoggerFactory; import info.freelibrary.util.StringUtils; -import edu.ucla.library.avpairtree.AvPtConstants; import edu.ucla.library.avpairtree.AvPtUtils; import edu.ucla.library.avpairtree.Config; import edu.ucla.library.avpairtree.CsvItem; @@ -51,6 +50,8 @@ public final class WaveformVerticle extends AbstractVerticle { private String myS3Bucket; + private String myS3ObjectUrlTemplate; + private String mySourceDir; @Override @@ -128,6 +129,17 @@ public void start(final Promise aPromise) { return; } + myS3ObjectUrlTemplate = config.getString(Config.AUDIOWAVEFORM_S3_OBJECT_URL_TEMPLATE); + + if (myS3ObjectUrlTemplate == null) { + configErrorMsg = LOGGER.getMessage(MessageCodes.AVPT_021); + + LOGGER.error(configErrorMsg); + aPromise.fail(configErrorMsg); + + return; + } + myS3Client = S3AsyncClient.builder().region(Region.of(myAwsDefaultRegion)).build(); vertx.eventBus().consumer(getClass().getName()).handler(this::handle); @@ -156,14 +168,14 @@ private void handle(final Message aMessage) { myS3Client.putObject(req, body).whenComplete((resp, err) -> { if (resp != null) { // Success! - final String audiowaveformURL = StringUtils.format(AvPtConstants.AUDIOWAVEFORM_URL_TEMPLATE, - myS3Bucket, myAwsDefaultRegion, URLEncoder.encode(s3ObjectKey, StandardCharsets.UTF_8)); + final String audiowaveformURL = StringUtils.format(myS3ObjectUrlTemplate, + URLEncoder.encode(s3ObjectKey, StandardCharsets.UTF_8)); // Reply with a JsonObject associating the item ARK with the URL for the audiowaveform data aMessage.reply(new JsonObject().put(csvItem.getItemARK(), audiowaveformURL)); } else { final String s3ErrorMsg = - LOGGER.getMessage(MessageCodes.AVPT_021, s3ObjectKey, err.getMessage()); + LOGGER.getMessage(MessageCodes.AVPT_022, s3ObjectKey, err.getMessage()); // Since the sender (WatcherVerticle) just logs all errors, should be okay to use a single // failureCode for all errors diff --git a/src/main/resources/av-pairtree_messages.xml b/src/main/resources/av-pairtree_messages.xml index b0827f6..5792725 100644 --- a/src/main/resources/av-pairtree_messages.xml +++ b/src/main/resources/av-pairtree_messages.xml @@ -25,7 +25,8 @@ The environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be set The environment variable AWS_DEFAULT_REGION must be set Using S3 bucket '{}' for audiowaveform storage - The system property 'audiowaveform.s3.bucket' must be set - Unable to upload audiowaveform for item '{}' to S3: {} + The environment variable AUDIOWAVEFORM_S3_BUCKET must be set + The environment variable AUDIOWAVEFORM_S3_OBJECT_URL_TEMPLATE must be set + Unable to upload audiowaveform for item '{}' to S3: {}