Optimizing Shared AI Model Caching in Kubernetes with Dragonfly and vCluster

This analysis showcases how Dragonfly and vCluster can efficiently share AI model caches across multiple tenants in Kubernetes, minimizing duplicated resource use.

In a multi-tenant Kubernetes environment, internal teams typically operate within isolated API boundaries, allowing for independent scalability. However, this often results in redundant downloads of sizable AI model files when multiple tenants concurrently access the same data from a central model repository, leading to extensive network traffic and storage wastage. The challenge is to maintain operational independence while also maximizing efficiency in data handling.

This report examines a laboratory setup designed to explore whether a shared peer-to-peer (P2P) cache can effectively facilitate the reuse of public AI model payloads across distinct tenants without compromising their independence or requiring elevated privileges for their workloads.

The Platform-Centric Data Flow

Utilizing Dragonfly, the system was configured to operate solely on the host cluster without a management layer. This involved scheduling tasks and managing peer connections while running a client DaemonSet on each node. Each client instance established an HTTP proxy on port 4001 that handled connections for the caching service. The infrastructure components and cache management remained under the platform team's control.

In this experiment, Tenant A and Tenant B were provisioned through their independent open-source vCluster API servers. Both entities launched model-download jobs, which were designed as non-root Pods. Notably, the deployment manifest avoided using privileged settings such as hostPath, hostNetwork, or additional Linux capabilities, which helped in maintaining workload isolation while allowing effective model downloading via Dragonfly. Each download process communicated with the node's local HTTP proxy, which managed the transfer of model data securely and efficiently.

Initially, the model-download job would initiate a HEAD request to fetch a signed CDN URL from Hugging Face, which then allowed the model payload to be downloaded through Dragonfly. Importantly, the setup was designed to normalize the cache identity by stripping unnecessary signed query parameters from the cache mapping, ensuring that both tenants could reference and receive data from the same cache task correctly, despite originating from different nodes.

Performance Observations

During the test, Tenant A executed its task from the control-plane node, resulting in the download of a file size approximately 267,954,768 bytes, with an expected SHA-256 checksum validation. The operation completed within an approximate time frame of 11 seconds.

Next, for Tenant B positioned on a different node (dragonfly-host-worker2), it was observed that the same file was accessed even though it couldn't leverage the local cache of the first tenant. Dragonfly, however, reused the task ID initially allocated to Tenant A, requiring the scheduler to fetch and duplicate the process, but effectively managing the request via existing data pieces already cached. Notably, Tenant B received the same file and its retrieval time was significantly reduced to around three seconds. This efficiency offered tangible proof of inter-tenant resource sharing with critical logs confirming that the transfer was executed via remote peer communications.

Insights for Platform Engineering

This experiment yielded valuable insights for platform engineering. First, treating the cache as a service representative at the platform level allows for centralized control while ensuring tenants engage with a slimmed-down interface for their data needs. Secondly, managing cache identity verification is essential; incorrectly handled URLs can hinder data accessibility and reuse across tenants.

Another lesson is the importance of transfer-source verification over simply measuring the time taken for downloads. While a faster turnaround for subsequent requests may hint at efficient caching, it does not confirm the precise source—local, remote, or unit-source. This underscores the need for ongoing monitoring of data integrity and accessibility within multi-tenant scenarios.

Evaluating Alternatives for Production

Beyond Dragonfly, platform engineers have various alternatives at their disposal such as utilizing a per-node registry mirror, implementing a pull-through cache, or establishing a shared persistent-volume cache. Each method carries unique implications in terms of data formats, trust configurations, and the operational management models required, therefore the context dictates the best choice.

The principle that emerged from these tests is to centralize the management of frequently accessed artifacts as a shared platform capability while presenting the narrowest interface necessary for tenant interactions. Begin with public artifacts and prescribed cache keys, and continually evaluate separate operational considerations around private-model access, caching policies, error recovery, and resource capacity as distinct and fundamental operational necessities.

Conclusion

Overall, this laboratory exploration provides insights into optimizing resource use within multi-tenant platforms. Implementing a shared caching system enhances efficiency, reduces unnecessary network loads, and establishes a model for artificial intelligence workflows that can adapt as the demands of both technology and services evolve.

Author Bio: Pavan Madduri, a Senior Cloud Platform Engineer at W.W. Grainger, also contributes extensively to Dragonfly and serves in leadership roles within the CNCF's community.