arxiv.canonical package

Core abstractions and methods for the arXiv canonical record.

Goals

In priority order:

1. Record article source and metadata

2. Record announcement record (what articles were announced on what day)

3. Long term durability: low risk of corruption, easy to secure, easy to verify, easy to repair

4. Changes to articles are only though the publication system and are recorded in the publication records. Write access to the canonical record will be limited to the announcement process.

5. Works with with Legacy and seamless transition from Legacy to NG

6. Testable

7. Easy to backup, replicate, mirror, share

8. Minimal dependencies

9. Conceptually as simple as possible

10. Complete public readability of data

Overview

The canonical record can be stored on any system that supports a key-binary data structure, such as a filesystem or an object store. The two core data structures in the record are:

• E-prints, comprised of metadata(domain.EPrint), submitted content, and the first rendering of the PDF.

• Announcement records, representing a single announcement-related event, such as a new version, a withdrawal, or a cross-list; these records are:

  • organized into daily announcement listing files and

  • emitted via a notification broker in real time, to trigger updates to downstream services and data stores.

E-prints

An e-print is comprised of (1) a metadata record, (2) a source package, containing the original content provided by the submitter, and (3) a canonical rendering of the e-print in PDF format. A manifest is also stored for each e-print, containing the keys for the resources above and a base-64 encoded MD5 hash of their binary content. The key prefix structure for an e-print record is:

e-prints/<YYYY>/<MM>/<arXiv ID>/v<version>/

Where YYYY is the year and MM the month during which the first version of the e-print was announced. Sub-keys are:

• Metadata record: <arXiv ID>v<version>.json

• Source package: <arXiv ID>v<version>.tar

• PDF: <arXiv ID>v<version>.pdf

• Manifest: <arXiv ID>v<version>.manifest.json

The purpose of the e-print record is to provide the ultimate source of truth regarding a particular e-print, and facilitate ease of access to all of the versions of an e-print.

Announcement listings

The announcement listings commemorate the announcement-related events that occur on a given day. This includes new e-prints/versions, withdrawals, cross-lists, etc.

The key prefix structure for an announcement listing file is:

announcement/<YYYY>/<MM>/<DD>/

Each daily key prefix may contain one or more sub-keys. Each sub-key ending in .json is treated as a listing file. This allows for the possibility of sharded/multi-threaded announcement processes that write separate listing files, e.g. for specific classification domains.

YYYY is the year, MM the month, and DD the day on which the announcement events encoded therein occurred and on which the subordinate listing files were generated.

Preservation record

The preservation record is a daily digest containing e-print content, announcement listings, and any suppress or remove directives (with corresponding tombstones).

announcement/<listing>.json
e-prints/<arXiv ID>v<version>/
    <arXiv ID>v<version>.json      # Metadata record
    <arXiv ID>v<version>.tar       # Source package
    <arXiv ID>v<version>.pdf       # PDF
Manifest: <arXiv ID>v<version>.manifest.json
suppress/<arXiv ID>v<version>/tombstone
preservation.manifest.json

The preservation.manifest.json record is similar to the e-print manifest record; it contains all of the keys and corresponding checksums for the items in the preservation record.

Differences from legacy arXiv

Metadata

• Legacy metadata records use an arcane, non-standard serialization format. The NG metadata record is serialized as JSON and stored as UTF-8 encoded text files.

• Legacy metadata records contain information not suitable for uncontrolled public distribution (such as submitter e-mail addresses), which limits replication options and increases complexity. NG metadata records contain only metadata suitable for public consumption, and can be distributed without further processing.

• In the legacy system, if the whole record of changes to an article is needed, many listings files have to be visited. The NG metadata record includes history/changes element to the metadata record, the elements of which are announcement records, i.e. updates to that particular version (timestamp + description).

Announcement listings

• Legacy arXiv uses monthly listing records, whereas NG listing records are created on a daily basis. This cuts down on the number of secondary indices/files that must be consistent to provide daily and weekly listings on the public site.

• The legacy system splits listings out by classification archive/category. This bakes the arXiv classification taxonomy into the structure of the data. The NG listing record decouples classification concerns from data structure; no assumptions are made about the classification of the e-prints described in an announcement listing record.

Subpackages

• arxiv.canonical.classic package
  • Subpackages
    • arxiv.canonical.classic.tests package
      • Submodules
        • arxiv.canonical.classic.tests.test_abs module
        • arxiv.canonical.classic.tests.test_backfill module
        • arxiv.canonical.classic.tests.test_content module
        • arxiv.canonical.classic.tests.test_daily module
        • arxiv.canonical.classic.tests.test_serialize_classic module
  • Submodules
    • arxiv.canonical.classic.abs module
    • arxiv.canonical.classic.backfill module
      • What version is this?
    • arxiv.canonical.classic.cli module
      • Next step: propagating events during backfill
    • arxiv.canonical.classic.content module
    • arxiv.canonical.classic.daily module
    • arxiv.canonical.classic.util module
• arxiv.canonical.domain package
  • Subpackages
    • arxiv.canonical.domain.tests package
      • Submodules
        • arxiv.canonical.domain.tests.test_content module
        • arxiv.canonical.domain.tests.test_file module
        • arxiv.canonical.domain.tests.test_identifier module
  • Submodules
    • arxiv.canonical.domain.base module
    • arxiv.canonical.domain.block module
    • arxiv.canonical.domain.content module
    • arxiv.canonical.domain.eprint module
    • arxiv.canonical.domain.file module
    • arxiv.canonical.domain.identifier module
    • arxiv.canonical.domain.license module
    • arxiv.canonical.domain.listing module
    • arxiv.canonical.domain.person module
    • arxiv.canonical.domain.preservation module
    • arxiv.canonical.domain.util module
    • arxiv.canonical.domain.version module
• arxiv.canonical.integrity package
  • Subpackages
    • arxiv.canonical.integrity.tests package
      • Submodules
        • arxiv.canonical.integrity.tests.test_version module
  • Submodules
    • arxiv.canonical.integrity.checksum module
    • arxiv.canonical.integrity.core module
    • arxiv.canonical.integrity.exceptions module
    • arxiv.canonical.integrity.listing module
    • arxiv.canonical.integrity.metadata module
    • arxiv.canonical.integrity.preservation module
    • arxiv.canonical.integrity.version module
• arxiv.canonical.log package
  • Submodules
    • arxiv.canonical.log.log module
    • arxiv.canonical.log.tests module
• arxiv.canonical.record package
  • Subpackages
    • arxiv.canonical.record.tests package
      • Submodules
        • arxiv.canonical.record.tests.test_listing module
        • arxiv.canonical.record.tests.test_version module
  • Submodules
    • arxiv.canonical.record.core module
    • arxiv.canonical.record.file module
    • arxiv.canonical.record.listing module
    • arxiv.canonical.record.metadata module
    • arxiv.canonical.record.preservation module
    • arxiv.canonical.record.version module
• arxiv.canonical.register package
  • Subpackages
    • arxiv.canonical.register.tests package
      • Submodules
        • arxiv.canonical.register.tests.test_api module
  • Submodules
    • arxiv.canonical.register.api module
    • arxiv.canonical.register.core module
    • arxiv.canonical.register.eprint module
    • arxiv.canonical.register.exceptions module
    • arxiv.canonical.register.file module
    • arxiv.canonical.register.listing module
    • arxiv.canonical.register.metadata module
    • arxiv.canonical.register.methods module
    • arxiv.canonical.register.util module
    • arxiv.canonical.register.version module
• arxiv.canonical.role package
  • Subpackages
    • arxiv.canonical.role.tests package
      • Submodules
        • arxiv.canonical.role.tests.test_register module
  • Submodules
    • arxiv.canonical.role.proxy module
    • arxiv.canonical.role.register module
    • arxiv.canonical.role.role module
    • arxiv.canonical.role.stream module
• arxiv.canonical.serialize package
  • Subpackages
    • arxiv.canonical.serialize.tests package
      • Submodules
        • arxiv.canonical.serialize.tests.test_serialize module
  • Submodules
    • arxiv.canonical.serialize.decoder module
    • arxiv.canonical.serialize.encoder module
    • arxiv.canonical.serialize.record module
• arxiv.canonical.services package
  • Subpackages
    • arxiv.canonical.services.tests package
      • Submodules
        • arxiv.canonical.services.tests.test_filesystem module
        • arxiv.canonical.services.tests.test_readable module
        • arxiv.canonical.services.tests.test_remote module
        • arxiv.canonical.services.tests.test_repository module
        • arxiv.canonical.services.tests.test_store module
  • Submodules
    • arxiv.canonical.services.cache module
    • arxiv.canonical.services.filesystem module
    • arxiv.canonical.services.readable module
    • arxiv.canonical.services.remote module
    • arxiv.canonical.services.repository module
    • arxiv.canonical.services.store module
    • arxiv.canonical.services.stream module
• arxiv.canonical.tests package

Submodules

• arxiv.canonical.core module
• arxiv.canonical.manifest module
• arxiv.canonical.preservation module
• arxiv.canonical.util module