In 1885, Wilson Bentley captured the first photograph of a snowflake. More than a century later, filming one as it melts is still challenging. The crystal spans millimeters, its branches carry sub-micron detail, and the whole structure changes from second to second. A lens sharp enough to resolve the finest branches sees only a small patch of the crystal, and a lens wide enough to see the whole crystal blurs the branches away.
In our recent Nature Communications paper, we show that a single image sensor, with no lens at all, can film such scenes as gigapixel video.1 The device resolves 308-nm linewidths across centimeter-scale fields at 30 frames per second, a throughput of 20.8 billion pixels per second (see Fig. 1). The key is to let the correlations in a moving scene supply the information that a single frame can’t.
Two ceilings on pixels
The space-bandwidth product (SBP) counts the resolvable pixels an imaging system delivers,2 and two ceilings cap it. One is the lens, whose aberrations grow with aperture and field angle so resolution and field of view trade against each other. The other is the detector, which cannot report more resolvable spots than its total pixel count.
Existing approaches work around these ceilings rather than lifting them. Whole-slide scanners and Fourier ptychography build SBP from sequential measurements, so a gigapixel dataset takes minutes.3 Camera arrays capture gigapixel snapshots in parallel by adding lenses and sensors.4 Either way, twice the SBP means twice the measurements or twice the sensors.
Coding the light at the sensor
Our device lifts both ceilings at once. A laser diode illuminates the sample by way of a fold mirror, with no lens in the light path. Beneath the sample is a conventional image sensor. On its cover glass, facing the pixels, we fabricate a disorder-engineered coded surface, a random layer of sub-micron scatterers that modulate both the intensity and the phase of the light. The sample sits 0.2 to 2 mm above this surface and never touches it, so the code tolerates contamination and extended operation, including inside a cell incubator. It is calibrated in advance with a static blood smear sample.
During acquisition, voice-coil actuators translate the coded sensor beneath the sample, and a static fiducial marker tracks its position with sub-pixel accuracy. Each frame sees the sample through a different patch of the code. Because the code scrambles fine spatial frequencies into diffraction patterns the pixels can record, every frame carries information about detail finer than the pixel pitch, which ptychographic phase retrieval can then recover. And with no lens, resolution and field of view decouple so the field of view is simply the active area of the sensor.
Let the scene do the work
Moving the sensor solves only half of the problem. A melting crystal or a living cell also changes between frames, which breaks classic ptychographic algorithms built for static objects. Our key observation is that natural dynamic scenes are highly correlated: Structures persist across frames and their changes follow smooth paths in time.
We describe the entire video with a single compact model, a.k.a. a neural field. Instead of storing each frame separately, the model keeps one shared map of spatial detail plus about 10 temporal “keyframes” that capture how the scene changes. A small neural network combines the two to predict the light field, including its phase, at any point and moment. To fit the model, we simulate how the predicted light passes through the coded surface to the sensor, and adjust the network until the simulation matches every recorded frame. No training data are needed. The model learns directly from each experiment’s measurements.
Because all frames are fit jointly, every frame does two jobs—pushing spatial resolution toward gigapixel scales and sampling the dynamic object at video rate. A 50-second recording at 30 frames per second, about 1,500 raw frames, compresses from gigabytes of data into a megabyte-scale model. Over the full sensor, each reconstructed frame holds 30,480 × 22,800 pixels. At 30 frames per second, this is 20.8 billion pixels per second from a single sensor. In this new imaging scheme, SBP no longer grows with the number of measurements or sensors. Instead, it grows with how much correlation the reconstruction can extract from the sample dynamics.
From snowflakes to stem cells
In Figure 1, the recovered phase tracks the melting front of a snowflake and the thickness of its meltwater film while the finest dendrites stay resolved. We also filmed crystals growing from solution, from the first dendrites to a polycrystalline film spanning the whole sensor.
For biology, label-free quantitative phase imaging avoids the phototoxicity of fluorescence. We placed the device in a standard incubator and imaged bone marrow mesenchymal stem cells during a wound-healing assay (see Fig. 2). The height maps resolve tunneling nanotubes, thin membrane bridges that cells use to exchange cargo and that usually require fluorescent labels to see. At camera frame rate, we followed cell division, cargo moving along nanotubes, and necrosis after hypotonic shock. Because the reconstruction recovers the complex field leaving the sample, cells that drift out of focus can be refocused digitally after acquisition.