Video to Rules

from visual ingredients to spatial constraints

Abstract

Video to Rules (VTR) started from a feeling: the uncanny valley. We know at once when something is not of our kin, even though we have no words for what gives it away. The capability already manifests itself; the question of how was yet to be excavated. I set out to excavate it: find the rules, write them down as mathematical relationships, and use them to read images and eventually video. I built video-inspection tools, an image collection and profiling pipeline, a visual vocabulary extracted from a pretrained vision model, experimental person detectors, and synthetic body-part libraries. The work moved from counting visual ingredients to measuring how they fit together, then from predicting appearances to eliminating incompatible configurations. Some internal tests were encouraging. Others exposed background contamination, framing bias, an extraction-model mismatch, and the gap between recognizing ingredients and locating an object. The road eventually led to a separate experiment using raw body geometry rather than image features. VTR did not become a complete video-understanding system. This is an account of what I built, what changed my mind, and what the records actually establish.

Why

VTR started from a feeling: the uncanny valley.

In nature, the thing that is trying to be you is usually the most dangerous thing. You can see it as a reflex in cats. Present a cat with a version of yourself wearing the mask of a cat or a dog, and the very first response is hostility.

Why is that? How do we immediately know when something is not of our kin, even though we lack the words to describe it? We do know what constitutes us, our kin; we have just never managed to put it into words. Our body and mind remember. They know, because whatever deviates from your kin is usually the thing that is pretending, and the thing that would harm you.

Then it struck me that we have an innate capacity to detect deviations from a pattern. Not just us: other living beings have it too, in some capacity we have yet to name.

From that observation I started connecting dots. I can tell that my friend is coming toward me without seeing his face, solely from the way he walks, even from behind. So the information that tells you it is your friend is there, and you are capable of extracting it.

So I went to the bedrock: the human body. This is where things become clear. Anatomy obeys its own rules, no matter what. The body has a limited set of postures it can take, given variables we already know. When I see a picture of someone with an arm behind a curtain while the rest of the body is visible, I infer the missing piece with high confidence, because something tells me what must be there. What I do in my mind is simulate the missing pieces from the given ones.

It is more than knowing. If the possibility already manifests itself, then how? The answer was already given; the question of how was yet to be excavated. VTR was that excavation: find the rules, write them down as mathematical relationships, and use them to read images, and eventually video, where they would matter most.

The way in was not a separate opaque classifier for everything I wanted to see. A person contains parts. Those parts contain recurring visual ingredients. The ingredients overlap with other things, but their relationships might let me distinguish what I was looking at. Could I extract those relationships, inspect them, and build a library that could be queried?

The practical ambition was to make image and long-video information available to language systems without repeatedly asking a large model to interpret everything. A language model could translate a question into a lookup; the visual engine would do the measurement. That was an intended use, not a finished service or a demonstrated cost advantage.

I also did not claim to know what every feature value meant. My working question was whether the relationships between values were stable and useful. Giving a cluster a convincing name would not establish that.

Approach

Extract ingredients, then measure their relationships

I used DINOv2 as the visual feature extractor. Instead of taking one summary vector for an entire image, I worked with patch-level features. I clustered those features into a shared vocabulary and assigned image patches to their nearest cluster. I called the clusters rules. Here, a rule means an empirical visual-feature cluster, not a discovered law of nature.

The later vocabulary had 80 centroids, each with 384 coordinates. A concept's spectrum was the normalized distribution of assignments across those centroids. It described which ingredients appeared and in what proportions. It did not preserve their arrangement.

For the first detector, I used non-negative least squares: approximate an image's spectrum with a non-negative mixture of body-part and background spectra. That made the decomposition inspectable. But a mixture could resemble a body without being a person.

The next representation kept the patch grid and counted neighboring rule pairs. I compared those neighborhoods against person and not-person reference matrices. Later constraint indices recorded rule positions, spread, coverage, and relationships across rendered views. The intended operation became: given what is visible, discard configurations that cannot fit it. Viewpoint and distance should come out of that process, rather than having to be known beforehand.

There is an important boundary in the method. The matching and constraint calculations were explicit arithmetic, but extracting features from a new image still used a pretrained neural network. Some project notes describe inference as having no neural network. That is too broad for the complete image-to-result pipeline I actually built.

The road

March 1–2: stop collapsing the thing I wanted to understand

The precursor was a movie-processing experiment. I aggregated frames into thermal-style projections, hoping that persistent structure would survive. The projections washed out motion. Collapsing time into one image removed the temporal structure I needed.

That led to a different question: what could I learn from a video before decoding all its pixels? I built tiered inspection: container information, packet and keyframe information, motion vectors where the decoder exposed them, then targeted low-resolution decoding. In the tested workflow, motion-vector extraction worked for H.264 but not for HEVC. That was a limitation of the extraction route I used, not evidence that HEVC contains no motion information.

This also changed the role I imagined for language. Rather than hand a language model an enormous spatial grid, I explored giving it ordered descriptions and letting it act as a librarian over extracted information. That remained an interface idea, not an evaluation of language-model comprehension.

The rule-library idea emerged alongside this work. Looking at whole-image summaries initially made the feature distributions look uninformative. Patch-level features exposed structure that the summary hid. I started comparing body-part distributions and looking for what persisted across examples.

The first vocabulary came from person images. That made it a circular foundation for measuring person-ness. I moved to a vocabulary built across concepts and explored different cluster counts. The records describe anatomical groupings persisting across the tested resolutions. That was a reason to continue, not proof that the clusters were universal physical primitives.

An early interpretation also had to be walked back. Two clusters appeared stronger in whole-person images than isolated parts, and were described as encoding composition. The notes explicitly mark that interpretation unverified: different zoom and framing could explain it. The observation survived; the claimed meaning did not become an established result.

March 6–7: the missing signal was partly the background

The expanded vocabulary build needed recovery after a crash, then profiling and persistence. More importantly, reconstructing the person spectrum from body-part spectra left a gap. Visual inspection put much of it on sky, ground, walls, and other surroundings rather than on the body.

The comparison had mixed two kinds of pictures: people in scenes and isolated close-ups of parts. I added background concepts and used them in the decomposition. Reconstruction improved, but reconstruction similarity was not detection accuracy.

I built a single-image detector around that decomposition. It could separate body-like ingredients from background better than raw similarity to a person spectrum. Then a squirrel received a high body score. Cars also shared ingredients with body parts, including reflective surfaces and illumination patterns.

That was not an argument for finding a magic person-only ingredient. It showed why the arrangement mattered. Shared ingredients were useful data; treating them as identity was the mistake.

I also expanded the collection pipeline to cover people in varied activities and visual conditions rather than mainly clean portraits. The pipeline handled acquisition, duplicate removal, profiling, and testing. An Android inspection prototype and interactive visualizations were built along the way. They were ways to inspect the work, not evidence of a deployed recognition product.

March 9: two failed volumes, three failed quality scores

The first spatial representation stacked person rule grids into a probability volume. It produced a visible person silhouette, but only because the selected pictures tended to contain centered, full-frame people. It was a position-locked template.

The second version included the broader person collection. That admitted enough background that not-person images scored higher. Calibration assigned the volume no useful weight. Making the collection larger had not fixed the representation; it had exposed its contamination.

I switched to local co-occurrence: which rules appear next to which other rules? This removed the requirement that a person occupy the same image coordinates. Combined with the spectrum decomposition, it produced the strongest recorded internal person-detection result, detailed below.

I then tried adding a quality-consistency channel. The idea was that genuine parts of one object should degrade coherently under blur or low quality. Three implementations failed. Raw confidence smoothness also appeared in non-person objects. A thresholded version inserted a hand-chosen cutoff instead of deriving it from data. Removing the cutoff did not make the score discriminate; calibration again assigned it no useful weight.

Some summaries still call this a three-channel detector. The recorded successful combination is the spectrum and co-occurrence pair. The memory record is clearer than the unfinished changelog entry about the quality channel's failures.

The architecture then shifted toward a constraint library assembled from parts, with complete-person images intended as a later composition test. That proposed separation must not be confused with the earlier detector: person images had already been used to construct its reference statistics.

March 10–11: use controlled renders as an experiment

I mapped body-part relationships across person images and built several visualizations. An apparent shared location for unrelated parts turned out to be an image-framing artifact. The visualizations also repeatedly confused feature identity with spatial coordinates. Changing the display did not, by itself, recover depth.

The next hypothesis was to compare observed appearance against controlled configurations. If a segment was sufficiently rigid, I could vary viewpoint, distance, pose, lens, and lighting separately and measure what changed.

I used a rigged human model in Blender. An initial forearm sweep gave encouraging agreement with real-image spatial patterns for selected rules. But rules present across the forearm views also occurred in other concepts. Presence alone still did not identify a forearm.

On March 11 I investigated lens effects. The first setup mixed changes in camera variables; I corrected it to keep distance fixed while changing focal length. The point of rendering was not to make more pictures indiscriminately. It was to use controlled changes as probes into the representation.

The notes describe inverse-focal-length fits and agreement between tested body segments. They also use stronger language about decoding optics than those experiments warrant. I had measured responses under particular rendered conditions, not completely decoded the model or established camera-independent recognition.

March 12–13: stop solving the inverse problem backward

I tried predicting rule grids from viewpoint parameters. Parallel searches explored interpolation, nearest-neighbor methods, and a neural decoder. The decoder was set aside because it did not meet the goal of an inspectable mathematical representation. The non-neural searches plateaued within that task.

There were simpler mistakes too. A full-grid comparison matched floor tiles rather than the forearm. I introduced foreground-focused scoring. More elaborate equations would not have repaired a score aimed at the wrong content.

The decisive turn was direction. In a photograph I do not already know angle, distance, and lens. I observe appearances. The useful operation is not “given the camera, generate rules,” but “given these rules and relationships, which configurations remain compatible?”

The notes call this five wrong prediction machines, but their detailed inventory includes proposals interrupted before implementation. What was actually tried included per-patch binary fits and continuous coverage fits. Additional harmonic and decomposition approaches were considered. I replaced the prediction direction with constraint indexing rather than treating every proposal as a completed experiment.

The indices stored observations from controlled views. Early indices covered forearm, shin, thigh, and upper arm. I expanded rendering to include twist where the setup supported it. Hands, feet, and neck also exposed the limits of treating every named segment as one simple rigid rotation.

Scaling the renders brought an engineering detour. Switching rendering backends and lowering samples did not solve the main overhead; too many simultaneous instances also exhausted GPU memory. Keyframing the sweep and rendering it as an animation avoided repeated per-frame setup. The pipeline could then proceed without treating a larger GPU as the answer to every bottleneck.

March 15: a silent mismatch changed what I could trust

The vocabulary had been built with DINOv2's non-register small model. Some extraction used the register variant. Those were not interchangeable feature spaces. The log records a collapse from 26 active rules to three while outputs still looked plausible.

I corrected the model choice, and the record says the rendered segments were re-extracted at the standard grid resolution with the matching model. This does not automatically revalidate every earlier experiment. I do not have a result-by-result audit showing which outputs were affected or a documented rerun of the earlier detector score after the correction.

That day also produced resolution comparisons, cross-model comparisons, and a synthetic shin constraint test. They were useful diagnostics, but the summaries sometimes called them universal or proven in a broader sense than their scope supports.

March 16–17: more data did not finish object detection

The later memory record reports higher-resolution extraction complete across the rendered segments. That advances the older status pages, which still say extraction was running.

It also records five more detection attempts that failed to produce the intended coherent object: rule-frequency matching lit up skin generally; co-occurrence and transition scoring remained statistical matches; sliding templates had scale problems; pair-and-offset lookup missed object-level coherence; boundary scoring was diffuse. These were later localization attempts, not a documented numerical retraction of the earlier binary detector test. They show why that test could not stand in for the complete system.

I explored continuous transformations across adjacent rendered views, then inspected internal activations under controlled perturbations. Some apparently interesting activity mainly tracked body position. Distance changed the representation substantially, complicating the idea of a simple scale-normalized identity. This investigation did not produce a finished replacement detector.

The road then left VTR's feature foundation. I dropped DINOv2 for a separate spatial-understanding experiment using raw body geometry. That is a related project, not a completed layer of VTR.

Results, with their scope

↑ back to contents

All figures here are historical project records, not new reruns or live deployment measurements.

The data work had its own concrete outputs. The inventory lists 133,270 original images across 35 concepts. A three-day expansion completed by March 9 reports 947,216 files, 190 GB, 81 concept directories, and 103 computed spectra. These are inventory figures, not independently recounted unique-image totals. I do not add the original and expanded figures because their overlap is unspecified.

The March 13 rendering target was 217,080 renders. The March 15 completed-and-downloaded record is 199,596 renders across 11 segments. I cannot explain the full difference from the inspected records; the target is not the delivered count. The March 16–17 memory record reports 93,049 higher-resolution grids across those 11 segments, not one verified grid for every downloaded render.

Other numbers do not support headline performance claims. My undated note says earlier reliability fell from 95% to 85% when conditions broadened, but gives no metric, sample size, split, or evaluation date. The March 12 search summary reports 50-plus experiments and a 64.5% plateau on rule-grid prediction, but the evaluation denominator and split are not established. Neither is overall VTR accuracy, and an observed search plateau is not a mathematical ceiling.

Limitations and open questions

↑ back to contents

The strongest unresolved issue is generalization. Search-derived concept collections include framing and background biases. Exact-file deduplication does not establish an independent test set or remove all near-duplicates. Synthetic sweeps provide controlled variables but do not establish performance across real people, clothing, cameras, occlusions, and motion.

The model mismatch makes provenance another first-class problem. A plausible array is not enough: it needs the matching extractor, vocabulary, and evaluation history. The corrected extraction is a real correction, not permission to carry every old number forward unchanged.

I also had not solved the transition from ingredient compatibility to coherent object localization. The later failures are evidence of that gap. Full scene reading, occlusion reasoning, cross-segment joints, and temporal persistence remained unfinished. In particular, the early video-inspection tools and later image-rule experiments were not demonstrated as one complete general video pipeline.

There is no recorded end-to-end comparison establishing a commercial cost advantage, general video accuracy, or a shipped service. The research produced prototypes and measurements. It did not establish all the claims the original ambition would require.

Where the road led, and where it stands

The separate spatial project changed both input and task: raw bone positions, with a missing frame to predict from surrounding frames. It used 303 parameters (three weights for each of 101 bones) to adjust interpolation using neighboring velocity and curvature. Evolutionary search retained successful directions and avoided previously tried configurations.

Its March 17 record reports 98.92% improvement over the linear-interpolation baseline, with score 0.00000700, on the single-bone, single-missing-frame curriculum stage. The dataset inventory lists 16 single-bone movement sequences; the evaluated gap count and independent holdout are not specified. This is a task-relative optimization result, not 98.92% visual understanding. Multi-frame gaps, compound movement evaluation, and cross-subject validation remained unchecked. Generated compound sequences were preparation, not completed validation.

Trying to build a library of rules is where the project grew. Once I was writing rules for what a body can and cannot do, and for what fills in what you cannot see, the scope skyrocketed. It expanded instantly, and that is what led me to shelve it. Funding was the other half: “Research necessitates funds, and funds don't come on their own.” I am one person, and I chose to put the work that supports the company first.

Shelving it does not mean there is no value to be extracted.

If I return to VTR, the next step I would take is narrower than another broad feature search: freeze the matching extractor and vocabulary, audit affected artifacts, and build an independent real-image evaluation. Then test whether the constraint method can locate and distinguish a small set of body segments outside its construction data. Only after that would I join segments and test persistence across video frames. That is a proposed restart plan, not work already completed.

The question remains worth asking. The useful thing I have now is not just the question. It is a record of representations that failed, measurements worth reproducing, and a clearer distinction between finding visual ingredients and understanding what they make.

Suggested citation: Yuval Halfon. Video to Rules: from visual ingredients to spatial constraints. Account prepared November 8, 2026. Published November 11, 2026. https://yuvalhalfon.com/projects/video-to-rules.html.

Questions or corrections: contact@yuvalhalfon.com.