[Spike]Testing Bibliographic metadata from catalogue through Zoom/Z39.50

[weiweishi]

Descriptive summary

We'd like to test the bibliographic record import again in the UAT/Test environment. Some related past issues are #13 #82

[seanluyk]

@weiweishi considering this as a spike, with the caveat that we're interested in using this for new items/submission, not editing

[jefferya]

From meeting with Mariana:

Question for the team:

• What should the z39.50 source be? Should ERA A+V use UAL's Catalog / Unicorn server? Or LOC? Or another?

Testing process (assume UAL server for testing) :

• via batch ingest
  • create batch manifest file
    • fill in only the following fields: bibliographic ID and the masterfile path
      • bibliographic ID: from the UAL catalog lookup item, use the unicorn id - from the url
        • For example: https://search.library.ualberta.ca/catalog/1760346 ==> unicorn id: 1760346
  • after batch ingest, compare metadata
    • pay particular attention to the terms of use / license (see MODS as it offers distinction)
• via UI
  • removed UI component in Avalon 6: Remove bibliographic import #373
    • should this decision be reversed?

Current ERA A+V config:

SETTINGS__BIB_RETRIEVER__PROTOCOL="z39.50"
SETTINGS__BIB_RETRIEVER__HOST=${UAL_z39_50_UNICORN_SERVER}
SETTINGS__BIB_RETRIEVER__PORT="2200"
SETTINGS__BIB_RETRIEVER__DATABASE="Unicorn"
SETTINGS__BIB_RETRIEVER__ATTRIBUTE="12"

Other tools to help with z39.50 debugging

• Yaz
• MarcEdit

[jefferya]

I've gone through the upstream Avalon code/documentation along with the local UAL added code related to z39.50. I was able setup on my desktop and the following are my initial findings and associated questions on the bibliographic metadata retriever z39.50 functionality @seanluyk .

Method:

• Setup experiment in a local Docker environment; the following are the results from the experiment.

Questions:

• What should the z39.50 source be? Should the source be UAL's Catalog / Unicorn server? Or LOC? The below experiment assumed UAL's Catalog.
• Will the z39.50 functionality be active for batch ingest only? Or should it be available also through the resource description metadata edit UI? The below experiment assumes the former, batch ingest only.
• The Genre term acquired via z39.50 from the UAL Catalog is sometimes not present in ERA A+V's PBCore genre authority and thus the ingest fails due to a UAL added genre validation step. UAL code offers the ability to override the z39.50 acquired genre term by adding a genre term to the batch manifest. Checking, is this process ok?
• The Terms of Use property is not being acquired via z39.50 in my test and the ingest fails due to a UAL added terms of use validation step. Should the terms of use property allow the batch manifest to override the z39.50 value like the genre term?

Depending on the answers to the above questions, setting up the UAT server for further testing should be fairly trivial.

Changes required to the existing set to run the tests (details below):

• Config: add environment variables
• Tool install: install yaz with libraries
• Avalon install: add optional Gem to the `bundle install
• Code changes: depends on the answers to the above questions and testing results; changes appear minimal

More details on the experiment -- setup batch manifest with bibliographic id, genre, terms of use, and masterfile, made code change below to allow term of use override. The result:

Initial findings for the config/setup:

Some or all of the following env vars required. Example: docker-compose.yml

      - SETTINGS__BIB_RETRIEVER__PROTOCOL=z39.50
      - SETTINGS__BIB_RETRIEVER__HOST=${ual_unicorn_z39.50}
      - SETTINGS__BIB_RETRIEVER__PORT=2200
      - SETTINGS__BIB_RETRIEVER__DATABASE=Unicorn
      - SETTINGS__BIB_RETRIEVER__ATTRIBUTE=12
      - SETTINGS__BIB_RETRIEVER__RETRIEVER_CLASS=Avalon::BibRetriever::Zoom
      - SETTINGS__BIB_RETRIEVER__RETRIEVER_CLASS_REQUIRE=avalon/bib_retriever/zoom

Install yaz z39.50 library. In the Docker context, modify: bin/docker-start:

pt-get clean && rm -rf /var/lib/apt/lists/; 
apt-get update
# Note: need libyaz4-dev otherwise yaz-config not found: apt-cache search yaz;
apt-get install -y yaz libyaz4 libyaz4-dev;
#yaz-config --help
# cat /usr/local/bundle/extensions/x86_64-linux/2.5.0/zoom-0.5.0/gem_make.out

The above two steps reduces the errors with testing with a batch manifest containing only the bibliographic id and master file path: genre field and Terms of Use field required. Related PRs #302 and #341 (but may need some additional code with respect to z39.50 bib retrieval, using #302 as a template)

• The following may offer a thread, override z39.50 lookup with these batch manifest included parameters (ToDo: terms_of_use may need to be added):

  avalon/lib/avalon/batch/entry.rb

  Lines 48 to 50 in 27f54df

  	# Sometimes we want to override the bib import data for some of the
  	# required fields (particularly if import data doesn't pass validations)
  	mo.assign_attributes(fields.slice(:language, :topical_subject, :genre))

• Also, the genre term in z39.50 server response needs to a PCore term otherwise UAL added genre validation fails (for id: 8127126, validation fails):

  avalon/app/models/mods_document.rb

  Lines 238 to 243 in 27f54df

  	# Want genre from our controlled vocabulary -- ignore it if it isn't
  	genre_terms = self.genre.collect(&:strip).uniq
  	self.genre = nil
  	genre_terms.each do |genre|
  	self.add_genre(genre) if GenreTerm.has_term?(genre)
  	end

  
  Details:

ActiveJob] [IngestBatchEntryJob] [79d847d5-3271-49f9-9179-731b01097e11]   BatchEntries Update (0.3ms)  UPDATE `batch_entries` SET `error` = TRUE, `current_status` = 'Invalid Entry', `error_message` = 'Genre field is required. and Terms of use field is required.', `updated_at` = '2020-08-21 15:34:32' WHERE `batch_entries`.`id` = 6

An example to test if z39.50 connecting using https://search.library.ualberta.ca/catalog/8127126

require 'avalon/bib_retriever/zoom'
bib_id=8127126
Avalon::BibRetriever.instance.get_record(bib_id)

[jefferya]

documentation reference:
https://wiki.dlib.indiana.edu/display/VarVideo/Bibliographic+Import+Configuration

[seanluyk]

@jefferya @anayram re: Z39.50 source. I think the UofA catalog?

[jefferya]

Process described above. Will reopen if needed.