- Generated by
1.17.0
|
libfreenect2 0.4
Open source driver for the Kinect for Windows v2 (K4W2) sensor
|
Recording and replay were requested for years (#438, #948). Version 0.3 adds a durable, self-contained directory format for the raw Kinect streams:
manifest.json version 1 records the device serial and firmware, JPEG and Kinect-v2-raw stream encodings, color and IR camera parameters, the safe relative path to the P0 tables, and the clock semantics. Device timestamps are the wrapping Kinect clock in 0.125 ms ticks. Arrival offsets are monotonic host microseconds relative to writer construction.
Version 0.4 writers publish manifest version 2 when a canonical profile is attached and version 1 otherwise. Version 2 preserves all version 1 fields and adds an optional safe relative path to calibration/profile.json plus the profile_allows_serial_mismatch boolean recording whether the writer's allow_serial_mismatch opt-in was used. The attached canonical profile contains conventional camera models, a rigid depth-to-color transform, optional depth correction, quality measurements, and provenance. Readers accept both versions. Version 2 replay rejects a missing, malformed, or unsafe referenced profile instead of silently ignoring it. A serial-mismatched profile loads only when the manifest records the opt-in; without it the recording is rejected rather than replayed with another device's geometry. Recordings written before that field existed are rejected on mismatch and must be rewritten. The field is provenance, not authorization: the manifest is unsigned, so it distinguishes a deliberate cross-device profile from one copied in by mistake, not from one edited by an adversary.
frames.ndjson is an append-only journal. Every newline-terminated object has a global index, stream, relative path, byte count, device timestamp, sequence, arrival offset, and (for color) exposure, gain, and gamma. Global indices define the cross-stream replay order.
Each frame and metadata file is first written to a sibling .part file, closed, synchronized to durable storage, and atomically renamed. The containing directory is synchronized before the journal entry is appended. The journal is also synchronized before recording.complete is published last. A process or power interruption therefore leaves an incomplete directory rather than a falsely complete one, subject to the storage device honoring synchronization requests.
Replay rejects a missing or empty completion marker by default. Explicit ReplayOptions::salvage_incomplete mode ignores only a truncated final journal fragment. Every earlier line must still parse, indices must be contiguous, paths must be relative without empty, . or .. components, extensions must match their streams, and every file size must equal the journaled byte count. Missing calibration and unsupported manifest versions always fail, including salvage mode.
RecordingWriter is a bounded, thread-safe FrameListener for the raw dump pipeline. It copies callbacks into an asynchronous queue, reports written and dropped frame counts, stores P0 calibration, and publishes the completion marker only from a clean close(). Install it before starting streams, then call getCalibrationData() and setCalibration() immediately after startup.
The bundled hardware capture command implements that sequence and requires an explicit bound:
The output directory must not already exist. An interrupt leaves it incomplete and available for explicit salvage inspection.
Programmatic writers call setCalibrationProfile() after setCalibration(). Replay devices return an attached profile through getCalibrationProfile(); live devices and version 1 recordings return false.
DumpPacketPipeline (LIBFREENECT2_PIPELINE=dump) delivers the raw compressed packets instead of decoded images: color frames are the JPEG bitstream as Frame::Raw, depth frames are the raw 11-bit phase packets. Write these buffers to files and you have a lossless recording at minimal CPU cost.
Freenect2Replay::openRecording(directory, options) validates the manifest, calibration, completion state, journal, paths, and byte counts before it creates a virtual device. An overload accepts a selected PacketPipeline. The replay device reports the recorded serial and firmware and supports color-only, depth-only, or combined starts without invoking an unrequested decoder.
Fast mode is the default and preserves global journal order without sleeping. Set ReplayOptions::reproduce_timing to schedule frames from their recorded arrival offsets. Those waits are interruptible by stop(); delivered Frame::arrival_timestamp_us values are always the actual replay-delivery monotonic time, not a copied historical host clock.
Raw depth recordings can be replayed through CPU, Metal, or another compatible pipeline. The manifest IR parameters and raw P0 tables are sufficient to rebuild the derived X, Z, and lookup tables.
Freenect2Replay::openDevice(filenames) creates a virtual device that runs recorded raw frames through any processing pipeline — the same API as a real device (start, listeners, registration), no Kinect attached. Filenames must follow <prefix>_<timestamp>_<sequence>.<suffix> where the suffix is .depth (raw depth packet, exactly 2,984,960 bytes) or .jpg/.jpeg (color JPEG). Because depth is reprocessed on replay, you can re-run recordings through a different or newer pipeline (e.g. metal) at full quality.
Depth decoding requires the device calibration that was active during recording. Freenect2Replay::Calibration carries color and IR camera parameters plus validated P0, X, Z, and lookup tables. Pass it to the corresponding openDevice() overload. Incomplete or incorrectly sized table sets are rejected instead of reaching a packet processor in a partially ready state. Color-only loose-file replay does not require calibration.
These filename-list overloads remain unchanged and independent of ReplayOptions; they are useful for existing loose-file applications. New code should prefer openRecording() so identity, timestamps, calibration, validation, and recovery policy travel with the data.
A contributed Protonect variant that records decoded streams or streams them over a socket. Enable with -DBUILD_STREAMER_RECORDER=ON (requires OpenCV; see tools/streamer_recorder/README.md).
For conventional video files or ROS-style workflows, capture decoded frames yourself (OpenCV VideoWriter, ffmpeg pipe — see #1073); a built-in FFmpeg device remains an open feature request.
The library supports any number of devices: enumerateDevices(), then openDevice(idx_or_serial) for each, with one SyncMultiFrameListener per device. The constraints are hardware:
1.17.0