API

JLSO.JLSO — Module.

A julia serialized object (JLSO) file format for storing checkpoint data.

Structure

The .jlso files are BSON files containing the dictionaries with a specific schema. NOTE: The raw dictionary should be loadable by any BSON library even if serialized objects themselves aren't reconstructable.

Example)

Dict(
    "metadata" => Dict(
        "version" => v"2.0",
        "julia" => v"1.0.4",
        "format" => :bson,  # Could also be :julia_serialize
        "compression" => :gzip_fastest, # could also be: :none, :gzip_smallest, or :gzip
        "image" => "xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/myrepository:latest"
        "pkgs" => Dict(
            "AxisArrays" => v"0.2.1",
            ...
        )
    ),
    "objects" => Dict(
        "var1" => [0x35, 0x10, 0x01, 0x04, 0x44],
        "var2" => [...],
    ),
)

WARNING: Regardless of serialization format, the serialized objects can not be deserialized into structures with different fields, or if the types have been renamed or removed from the packages. Further, the :julia_serialize format is not intended for long term storage and is not portable across julia versions. As a result, we're storing the serialized object data in a json file which should also be able to load the docker image and versioninfo to allow reconstruction.

source

JLSO.JLSOFile — Method.

JLSOFile(data; format=:julia_serialize, compression=:gzip, kwargs...)

Stores the information needed to write a .jlso file.

Arguments

data - The objects to be stored in the file.

Keywords

image="" - The docker image URI that was used to generate the file
julia=1.0.5 - The julia version used to write the file
version=v"2.0" - The file schema version
format=:julia_serialize - The format to use for serializing individual objects. While :bson is recommended for longer term object storage, :julia_serialize tends to be the faster choice for adhoc serialization.
compression=:gzip, what form of compression to apply to the objects. Use :none, to not compress. :gzipfastest for the fastest gzip compression, :gzipsmallest for the most compact (but slowest), or :gzip for a generally good compromize. Due to the time taken for disk IO, :none is not normally as fast as using some compression.

source

JLSO.load — Method.

load(io, objects...) -> Dict{String, Any}
load(path, objects...) -> Dict{String, Any}

Load the JLSOFile from the io and deserialize the specified objects. If no object names are specified then all objects in the file are returned.

source

JLSO.save — Method.

save(io, data)
save(path, data)

Creates a JLSOFile with the specified data and kwargs and writes it back to the io.

source

Base.getindex — Method.

getindex(jlso, name)

Returns the deserialized object with the specified name.

source

Base.setindex! — Method.

setindex!(jlso, value, name)

Adds the object to the file and serializes it.

source

JLSO.complete_compression — Method.

complete_compression(compressing_buffer)

Writes any end of compression sequence to the compressing buffer; but does not close the underlying stream. The compressing_buffer itself should not be used after this operation

source