Research methods
Feature Extraction & Multimodal Content Understanding
A proposed framework for keeping text, image and video evidence traceable: preserve source locations, separate observations from interpretations and test what each extraction method misses.
By Terry Chen · Updated
Originally published
A video can contain spoken claims, visible interface behavior and audience reactions that support different conclusions. Combining them into one fluent summary can make those differences hard to inspect. This design proposal keeps the source location and interpretation of each observation attached to it. The example below is illustrative, and the evaluation is a method to run rather than a reported benchmark.
Preserve the source and the part you inspected
Record a source identifier, access context, observation date and the relevant location: a passage, frame, time range or image region. Keep a transcript separate from the original audio and an extracted label separate from the image. Those derived records can be useful without becoming interchangeable with their sources.
The W3C provenance overview provides a reference model for describing the entities, activities and people involved in producing data. Here, that suggests retaining which transformation produced an observation and who reviewed it. The record below is a practical proposal, not a PROV implementation.
For media locations, W3C Media Fragments defines temporal and spatial addressing. A time-range fragment such as #t=12,16 identifies a portion of a media resource. Record the human-readable range too, and check that the actual source player can open the intended segment before treating the link as a usable citation.
Choose an extraction method against a defined task
Compare candidate approaches using the same permitted inputs and review criteria. Do not assume one approach has better coverage, lower cost or greater reliability without observing the task you need it to perform.
- Transcript or extracted text: useful to inspect spoken or written language. Check what the text leaves out, including visible state, layout and timing.
- Text with selected frames: lets a reviewer compare a claim with a chosen visual context. Record how the frames were selected so uninspected intervals remain visible.
- Direct multimodal interpretation: can be evaluated on the original combination of inputs. Require source locations for important findings and test omissions, mistaken relationships and unsupported conclusions.
Keep the creator’s content and audience reactions as different source types. A reply may describe a separate experience or express an opinion; it does not independently verify what the video demonstrates.
An illustrative disagreement across modalities
Illustrative example — no real recording or customer result
Spoken claim: the narrator says a report can be exported. Visible observation: a frame shows an Export button, but the recording ends before a file is opened. Audience reaction: a comment asks whether the export preserves the report’s source references.
Supported finding: the demo presents an export action. Unresolved question: whether the resulting file is usable and retains the required references.
Next test: inspect a permitted output file and follow its references. Keep that test result separate from the demo and the comment.
A combined summary that says “users can export a source-backed report” would skip the missing result. Retaining the separate observations makes the unresolved requirement visible.
Keep a record a reviewer can follow
Source reference, type and permitted use: Observed date, source version and location: Original passage / frame / time range: Transformation or extraction method and version: Observation supported by that source: Interpretation and competing explanations: Missing context, contradictions and uncertainty: Reviewer, verification result and next test:
When sources disagree, preserve both references and describe the disagreement. Use an uncertainty label with a reason rather than an unexplained confidence percentage. Keep private media and identifying material in the authorized environment.
Evaluate errors and the effort of checking them
- Define the findings that matter. Select representative, permitted examples and agree what a reviewer should be able to establish from them.
- Test source traceability. Can another person open the referenced passage or segment and verify the observation?
- Record distinct error types. Keep omissions, unsupported additions, incorrect source locations and mistaken relationships separate.
- Measure the whole review. Include preparation, extraction, waiting, checking and correction effort, along with actual cost.
- State the accepted scope. Record which inputs and tasks the result supports, remaining failures and what would require a new evaluation.
This evaluation can inform a pipeline choice. It does not establish a general model ranking, economic trend or guaranteed improvement in downstream decisions.
Hand off a bounded finding
Pass the supported observation, its source and the unresolved question into the agent specification brief. Keep the owner’s decision and acceptance criteria explicit. A traceable observation is useful input; choosing and implementing a product change still requires a decision.
This article describes a proposed representation and review method. For current CrowdListen services and their actual source/output boundaries, inspect the expert catalog. Agree scope and estimate for a supported task before collection.
