API

A julia serialized object (JLSO) file format for storing checkpoint data.

Structure

The .jlso files are BSON files containing the dictionaries with a specific schema. NOTE: The raw dictionary should be loadable by any BSON library even if serialized objects themselves aren't reconstructable.

Example)

Dict(
    "metadata" => Dict(
        "version" => v"2.0",
        "julia" => v"1.0.4",
        "format" => :bson,  # Could also be :julia_serialize
        "compression" => :gzip_fastest, # could also be: :none, :gzip_smallest, or :gzip
        "image" => "xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/myrepository:latest"
        "project" => Dict{String, Any}(...),
        "manifest" => Dict{String, Any}(...),
    ),
    "objects" => Dict(
        "var1" => [0x35, 0x10, 0x01, 0x04, 0x44],
        "var2" => [...],
    ),
)

WARNING: Regardless of serialization format, the serialized objects can not be deserialized into structures with different fields, or if the types have been renamed or removed from the packages. Further, the :julia_serialize format is not intended for long term storage and is not portable across julia versions. As a result, we're storing the serialized object data in a json file which should also be able to load the docker image and versioninfo to allow reconstruction.

JLSOFile(data; format=:julia_serialize, compression=:gzip, kwargs...)

Stores the information needed to write a .jlso file.

Arguments

• data - The objects to be stored in the file.

Keywords

• image="" - The docker image URI that was used to generate the file
• julia=1.5.4 - The julia version used to write the file
• version=v"4" - The file schema version
• format=:julia_serialize - The format to use for serializing individual objects. While :bson is recommended for longer term object storage, :julia_serialize tends to be the faster choice for adhoc serialization.
• compression=:gzip, what form of compression to apply to the objects. Use :none, to not compress. :gzip_fastest for the fastest gzip compression, :gzip_smallest for the most compact (but slowest), or :gzip for a generally good compromise. Due to the time taken for disk IO, :none is not normally as fast as using some compression.

load(io, objects...) -> Dict{Symbol, Any}
load(path, objects...) -> Dict{Symbol, Any}

Load the JLSOFile from the io and deserialize the specified objects. If no object names are specified then all objects in the file are returned.

save(io, data)
save(path, data)

Creates a JLSOFile with the specified data and kwargs and writes it back to the io.

getindex(jlso, name)

Returns the deserialized object with the specified name.

setindex!(jlso, value, name)

Adds the object to the file and serializes it.

complete_compression(compressing_buffer)

Writes any end of compression sequence to the compressing buffer; but does not close the underlying stream. The compressing_buffer itself should not be used after this operation