- Generated by
1.17.0
|
libfreenect2 0.4
Open source driver for the Kinect for Windows v2 (K4W2) sensor
|
Answers to recurring API questions from the upstream tracker.
All streams (color, IR, depth) are horizontally mirrored relative to the Microsoft SDK's output — you see the scene as if looking into a mirror. This is consistent across streams, so registration and getPointXYZ() remain self-consistent; there is no API option to change it. If you need SDK-style orientation, flip the final images yourself, e.g. cv::flip(img, img, 1), or negate X when exporting point clouds. Flip at the end of your pipeline: flipping before registration would break the mapping.
No. Unlike the Kinect v1, no protocol command is known that disables the Kinect v2's illuminator while streaming; the emitter is controlled by the firmware together with the depth stream. Physically covering the emitters kills depth measurements (the IR image remains usable with ambient IR). If you only need the color stream, note that startStreams(rgb=true, depth=false) still powers the sensor bar as the firmware dictates.
No. libfreenect2 returns color, infrared, and depth frames; it does not implement the Kinect SDK's body tracker or its 25-joint skeleton frame.
The supported pose workflow for this fork is to capture aligned frames through pylibfreenect3, run MediaPipe on the color image, and lift landmarks with valid registered depth into metric XYZ coordinates. See Using libfreenect2 from Python. Those joints are estimates from a machine-learning model combined with measured depth. They are not sensor-provided ground truth, and they do not carry the Kinect SDK's tracking states or body identities.
Two ways:
In code, install your own logger or a quieter console logger before creating Freenect2:
Frame::timestamp is the device's clock, in ticks of 0.125 ms (so it advances by ~266 per frame at 30 Hz, ~533 at 15 Hz in low light). It is not wall-clock time, wraps as a 32-bit value, and resets when the device restarts. Frame::arrival_timestamp_us records the monotonic host time of the first contributing USB transfer. It has no wall-clock epoch; sample your wall clock separately if you need UTC correlation.
Multiply by 0.125f to get milliseconds: double ms = frame->timestamp * 0.125;
Color and depth frames carry timestamps from the same device clock, but the two cameras expose independently. SyncMultiFrameListener groups requested frame types without enforcing a timestamp delta; it does not provide hardware synchronization. For motion-sensitive work, use TimestampAlignedFrameListener with an explicit threshold and inspect its drop/delta statistics. See the Frame timing and software pairing guide for clock semantics, wraparound, listener behavior, and capture diagnostics.
Yes. Color arrives on a bulk endpoint and IR/depth on an isochronous endpoint, and the two run concurrently — request Frame::Color | Frame::Ir | Frame::Depth from a single listener.
This is worth stating because the answer for the Kinect v1 was no: there, color and IR were two modes of one stream and you could have only one at a time. If you found advice to that effect, it does not apply here. See Kinect v1 versus Kinect v2.
No. This is a sensor driver: it delivers color, IR, and depth frames, plus registration between them. There is no skeleton, joint, or body-index API, and adding one is out of scope.
For pose estimation, the maintained route is MediaPipe on the color image with landmarks lifted into metric coordinates through the registration maps — see Using libfreenect2 from Python, which walks through the whole workflow. The OpenNI2 driver (BUILD_OPENNI2_DRIVER) is another option if you need to feed an existing OpenNI/NiTE-based middleware stack.
No, because the hardware has neither. The Kinect v2's base is fixed; the tilt motor and accelerometer belong to the Kinect v1. Aim it by moving the sensor.
The v2 does have two controllable status LEDs — Freenect2Device::setLedStatus() takes an intensity from 0 to 1000 and a constant or blink mode. See Kinect v2 USB protocol.
Not through libfreenect2. The sensor exposes a four-microphone array on a separate USB interface association, and this library does not implement it. Calibrated directional audio is one of the features this fork explicitly does not provide.
On Linux the kernel handles it for you: recent kernels bind snd-usb-audio to those interfaces, so the array shows up as a normal ALSA capture device without any help from this library. Upstream reports it enumerating as 16 kHz, 4 channels, S32_LE:
This project does not test the audio path, and the channel layout is not documented anywhere authoritative — treat the parameters above as a starting point rather than a specification. There is no equivalent on macOS or Windows.
Several distinct causes, which is why the holes do not all look alike:
Invalid pixels read as 0, not 2047; the 2047 sentinel is a Kinect v1 convention.
They can, and the mechanism is different from the Kinect v1's. The v1 projected a fixed speckle pattern, so two units overlaid two patterns and confused each other's matching. The v2 measures the phase of amplitude-modulated IR, so a second unit's illumination adds unmodelled light to the first unit's measurement, which shows up as noise and outliers in the overlap region rather than as an outright failure.
In practice, overlapping views degrade gracefully rather than break, and how much depends on geometry and overlap. This project does not characterize it. Do not carry over v1-era advice about polarizing filters — it addressed a mechanism the v2 does not have.
Note that the far more common multi-sensor problem is USB bandwidth, not optical interference: each sensor needs its own host controller. See USB bandwidth and transfer tuning and Recording, replay, and multiple Kinects.
1.17.0