Amazon Nova Multimodal Embeddings: Technical Deep Dive

Title: Multimodal Embeddings at Scale: A Technical Deep Dive into Amazon’s AI Data Lake for Media and Entertainment

Executive Summary

• Amazon demonstrates a production-grade multimodal video search system that processes 792,270 videos (8,480 hours / 30.5 million seconds) in 41 hours using Amazon Nova Multimodal Embeddings and Nova Pro, at a first-year cost of $23,632–$27,328.
• The architecture leverages Nova’s AUDIO_VIDEO_COMBINED mode to produce 1024-dimensional embeddings (chosen over 3072-dim for 3× storage savings with minimal accuracy loss) stored in an Amazon OpenSearch Service k-NN index, supporting text-to-video, video-to-video, and hybrid (70% vector + 30% BM25) search.
• Ingestion scales to 19,400 videos/hour using 4× c7i.48xlarge Spot instances (600 parallel workers) while respecting Bedrock’s 30 concurrent async jobs per account through a job queue + polling pattern.
• This solution represents Amazon’s blueprint for turning traditional media data lakes into semantic AI data lakes, replacing manual tagging with rich multimodal embeddings and hybrid retrieval.

Technical Architecture

The system is built around two primary workflows: ingestion and search, orchestrated on top of Amazon S3, Amazon Bedrock, and Amazon OpenSearch Service.

Ingestion Pipeline

Videos are first uploaded to S3. The pipeline uses four c7i.48xlarge EC2 instances running 600 worker threads. Because the Amazon Bedrock asynchronous inference API for Nova Multimodal Embeddings has a hard quota of 30 concurrent jobs per account, the orchestrator implements a job queue with continuous polling.

Each worker:

1. Submits an async embedding job (up to the concurrency limit).
2. Polls for completion via GetAsyncInvoke.
3. On completion, downloads the embedding output from the designated S3 location.

Nova Multimodal Embeddings (amazon.nova-2-multimodal-embeddings-v1:0) operates in AUDIO_VIDEO_COMBINED mode. The service automatically segments videos into 15-second chunks — a deliberate trade-off that balances scene change capture with manageable embedding cardinality. Each chunk produces a 1024-dimensional embedding vector. The choice of 1024-dim over the full 3072-dim representation yields approximately 3× storage savings in OpenSearch with “minimal accuracy impact,” according to the authors.

Concurrently, Nova Pro (us.amazon.nova-pro-v1:0) or the newer Nova Lite (us.amazon.nova-2-lite-v1:0) is used to generate 10–15 descriptive tags per video from a predefined taxonomy. These tags are stored in a separate BM25-optimized text index for hybrid retrieval. The post notes that Nova 2 Lite offers improved accuracy at lower cost for tagging and recommends it for new deployments.

Search Architecture

OpenSearch Service 2.11+ with the k-NN plugin hosts two indexes:

• A vector index using the HNSW (Hierarchical Navigable Small World) algorithm for approximate nearest neighbor search on the 1024-dim embeddings.
• A text index containing Nova-generated tags and any other metadata.

Three query modes are supported:

• Text-to-video: User prompt is converted to an embedding via the same Nova model (text modality) and used for k-NN search.
• Video-to-video: A query video is chunked and embedded; the system can perform either per-chunk similarity or aggregated video-level similarity.
• Hybrid search: Combines vector similarity (70% weight) with BM25 keyword matching on tags (30% weight). This is implemented using OpenSearch’s hybrid query DSL with knn and match clauses.

The architecture diagram (not reproduced here) shows a clean separation between the asynchronous ingestion path and the low-latency search path, with S3 acting as both the video lake and the embedding result staging area.

Performance Analysis

Scale Achieved

• Total videos: 792,270 (787,479 from Multimedia Commons + 4,791 from MEVA)
• Total duration: 8,480 hours (30.5 million seconds)
• Average video length: ~37 seconds (Multimedia Commons), ~5 minutes (MEVA)
• Processing time: 41 hours
• Throughput: 19,400 videos per hour

Cost Breakdown (First Year)

• One-time ingestion: $18,088
  • EC2 compute (4× c7i.48xlarge Spot @ $2.57/hr × 41 hrs): $421
  • Nova Multimodal Embeddings (30.5M seconds × $0.00056/sec batch pricing): $17,096
  • Nova Pro tagging (792K videos × ~600 tokens avg.): $571
• Annual OpenSearch Service:
  • On-demand: $9,240
  • Reserved Instances: $5,544

Total first-year cost: $27,328 (on-demand) or $23,632 (with Reserved Instances).

Embedding Dimension Trade-off The decision to use 1024-dimensional embeddings instead of 3072 is significant. Storage cost in OpenSearch scales linearly with vector dimension. A 3× reduction in vector size provides substantial savings in both memory and disk, especially at this scale (tens of millions of 15-second chunks). The authors report only minimal accuracy degradation, suggesting that for retrieval-focused workloads the extra dimensions yield diminishing returns.

Comparison with Alternative Approaches

While the post does not provide direct benchmark tables against competitors, we can contextualize the results:

The primary differentiator is Amazon’s tight integration between Bedrock’s native multimodal embedding model and OpenSearch’s k-NN capabilities, combined with explicit quota-aware async orchestration.

Technical Implications

This work signals a broader industry shift from multimodal data lakes to multimodal AI data lakes (or lakehouses). Traditional media archives relied on manual metadata or limited keyword tagging. By generating dense, semantically rich embeddings at ingestion time, every 15-second chunk becomes searchable via natural language.

For the media and entertainment industry, this has profound implications:

• Content discovery moves from rigid taxonomies to semantic understanding (e.g., “find scenes with a tense conversation in a rainy night city street”).
• Video-to-video search enables duplicate detection, content reuse, and recommendation at scale.
• The hybrid approach mitigates the well-known weaknesses of pure vector search (hallucinated similarity, lack of precise keyword control).

From an infrastructure perspective, the solution demonstrates that Bedrock’s async multimodal API, despite its 30-job concurrency limit, can be effectively scaled to petabyte-scale media workloads using relatively modest compute (four large EC2 instances).

Limitations and Trade-offs

Several limitations are worth noting:

• Concurrency quota: The 30 concurrent async jobs per account is a significant bottleneck. The solution works around it with polling and queuing, but this adds latency and operational complexity. Enterprises will need to request quota increases for larger deployments.
• 15-second chunking: While pragmatic, fixed chunking can split semantically coherent scenes. More advanced scene-boundary detection could improve quality but would increase complexity and cost.
• Embedding freshness: Once ingested, embeddings are static. If the underlying Nova model is updated, re-embedding the entire corpus becomes expensive ($17k+ for this dataset).
• Tag taxonomy rigidity: Reliance on a predefined taxonomy for Nova Pro tagging limits open-vocabulary description. The post acknowledges Nova Lite may be superior here.
• Cost predictability: While batch pricing is used, embedding cost is charged per second of video. Long videos or high reprocessing rates can become expensive quickly.
• Dimensionality: Choosing 1024-dim is a conscious trade-off. Some sophisticated downstream tasks (fine-grained classification, clustering) may benefit from the full 3072-dim representation.

Expert Perspective

This is one of the most concrete, numbers-driven demonstrations of multimodal embeddings at scale published by a hyperscaler to date. The transparency around cost ($0.00056 per second), throughput (19.4k videos/hour), exact instance types, and deliberate architectural choices (1024 vs 3072, 15s chunks, 70/30 hybrid weighting) makes this post unusually actionable for ML engineers.

The most significant contribution is the proof that a unified embedding + hybrid search architecture can replace years of accumulated manual tagging labor in media archives. By treating video as a first-class citizen in the embedding space rather than an afterthought, Amazon is positioning OpenSearch as a serious contender in the emerging multimodal retrieval market.

The recommendation to use Nova 2 Lite for tagging instead of Nova Pro is also telling — it shows internal iteration and a willingness to course-correct in public documentation. This level of pragmatism is refreshing.

For senior ML engineers building similar systems, the key takeaway is the importance of quota-aware orchestration and cost-vs-accuracy embedding dimension tuning. These two factors often dominate total cost of ownership more than raw model quality.

Technical FAQ

How does the 1024-dimensional embedding compare to the full 3072-dimensional output on retrieval quality?

The post states that 1024-dim was chosen over 3072-dim for 3× storage savings “with minimal accuracy impact.” No exact recall@K or nDCG numbers are provided, but the authors consider the trade-off acceptable for semantic video search. In practice, many retrieval systems find that beyond ~768–1536 dimensions, gains are marginal for coarse retrieval tasks.

Is the solution backwards-compatible with existing OpenSearch k-NN indexes?

Yes. The architecture uses standard OpenSearch 2.11+ k-NN indices with the HNSW engine. Any application already using OpenSearch vector search can adopt the Nova embeddings by simply ingesting the new 1024-dim vectors. The hybrid search component uses standard knn + match query combination.

How does the async Bedrock quota affect production deployments?

The 30 concurrent async jobs per account is the primary scaling constraint. The demonstrated solution uses a polling orchestrator across 600 workers to saturate the quota continuously. For larger scale, customers must either shard across multiple AWS accounts or request a quota increase from AWS. This is currently the biggest operational limitation.

Can this architecture be extended to other modalities or newer Nova models?

Yes. The ingestion pipeline is model-agnostic as long as the embedding API returns compatible vectors. The post already suggests using Nova 2 Lite for tagging. Future Nova multimodal models that support longer contexts, higher resolution, or additional modalities (e.g., depth, text overlays) can be swapped in with relatively minor changes to the worker code.

References

• Amazon Nova Multimodal Embeddings API schema and documentation
• Amazon Bedrock asynchronous inference quotas
• OpenSearch k-NN plugin and hybrid search documentation

Sources

• Multimodal embeddings at scale: AI data lake for media and entertainment workloads | AWS Machine Learning Blog
• Additional context from AWS first-party announcement verification

All technical specifications, pricing, and benchmark data in this article are sourced directly from official announcements. Competitor comparisons use publicly available data at time of publication. We update our coverage as new information becomes available.