S3-compatible storage client for Psyche AI Virtual Assistant.
Overview#
psyche-storage provides a unified storage abstraction aligned with the
@oshun/storage TypeScript patterns for cross-language compatibility. It
includes specialized asset managers for avatar models, voice recordings, and
knowledge documents.
Features#
- S3/MinIO Support: Compatible with AWS S3 and MinIO backends
- Async Operations: All operations are async/await compatible
- Multipart Uploads: Automatic chunking for large files with progress tracking
- Signed URLs: Generate pre-signed URLs for direct upload/download
- Asset Managers: Specialized managers for Psyche-specific assets
- Metadata Tracking: Automatic metadata for asset categorization and tracking
Installation#
bash
# Basic installation
poetry add psyche-storage
# For development
poetry add psyche-storage[dev]
Quick Start#
Creating a Storage Client#
python
from psyche_storage import S3Config, StorageProvider, create_s3_client, create_minio_client
# AWS S3
config = S3Config(
provider=StorageProvider.S3,
region="us-east-1",
access_key_id="your-access-key",
secret_access_key="your-secret-key",
bucket="psyche-assets",
)
client = create_s3_client(config)
# MinIO (local development)
client = create_minio_client(
endpoint="localhost:9000",
access_key_id="minioadmin",
secret_access_key="minioadmin",
bucket="psyche-assets",
use_ssl=False,
)
Basic Operations#
python
# Upload a file
result = await client.upload(
key="path/to/file.txt",
body=b"file content",
options=UploadOptions(
content_type="text/plain",
metadata={"custom-key": "custom-value"},
),
)
# Download a file
result = await client.download("path/to/file.txt")
print(result.body)
# Check if file exists
exists = await client.exists("path/to/file.txt")
# Get file metadata
metadata = await client.get_metadata("path/to/file.txt")
# List files
result = await client.list_objects(
ListObjectsOptions(prefix="path/to/", max_keys=100)
)
for obj in result.objects:
print(f"{obj.key}: {obj.size} bytes")
# Delete a file
await client.delete("path/to/file.txt")
Multipart Upload for Large Files#
python
from psyche_storage import MultipartUploadConfig, MultipartUploadProgress
def progress_callback(progress: MultipartUploadProgress):
print(f"Upload progress: {progress.percentage}%")
result = await client.upload_multipart(
key="large-file.zip",
body=large_data,
options=UploadOptions(content_type="application/zip"),
multipart_config=MultipartUploadConfig(
part_size=10 * 1024 * 1024, # 10 MB parts
concurrency=4,
on_progress=progress_callback,
),
)
Signed URLs#
python
from psyche_storage import SignedUrlOptions
# Generate download URL (expires in 1 hour)
result = await client.get_signed_download_url(
"path/to/file.pdf",
SignedUrlOptions(
expires_in=3600,
response_content_disposition='attachment; filename="document.pdf"',
),
)
print(f"Download URL: {result.url}")
print(f"Expires at: {result.expires_at}")
# Generate upload URL
result = await client.get_signed_upload_url(
"uploads/new-file.jpg",
SignedUrlOptions(
expires_in=3600,
content_type="image/jpeg",
),
)
Asset Managers#
Avatar Storage#
python
from psyche_storage import AvatarStorageManager
avatar_manager = AvatarStorageManager(client)
# Upload avatar model
result = await avatar_manager.upload_model(
avatar_id="avatar-123",
data=model_data,
filename="character.glb",
version="v2",
persona_id="persona-456",
)
# Upload texture
result = await avatar_manager.upload_texture(
avatar_id="avatar-123",
data=texture_data,
filename="diffuse.png",
texture_type="diffuse",
)
# Upload animation
result = await avatar_manager.upload_animation(
avatar_id="avatar-123",
data=animation_data,
filename="wave.glb",
animation_name="wave_greeting",
)
# Upload thumbnail
result = await avatar_manager.upload_thumbnail(
avatar_id="avatar-123",
data=thumbnail_data,
)
# List avatar assets
assets = await avatar_manager.list_avatar_assets(
avatar_id="avatar-123",
category="models",
)
# Delete all avatar assets
await avatar_manager.delete_avatar("avatar-123")
Voice Storage#
python
from psyche_storage import VoiceStorageManager
voice_manager = VoiceStorageManager(client)
# Upload voice recording
result = await voice_manager.upload_recording(
persona_id="persona-123",
data=audio_data,
filename="session_001.mp3",
session_id="session-456",
duration_ms=30000,
transcript="Hello, how can I help you today?",
)
# Upload voice sample for cloning
result = await voice_manager.upload_sample(
persona_id="persona-123",
data=sample_data,
filename="voice_sample.wav",
sample_type="voice_clone",
)
# Upload voice model
result = await voice_manager.upload_voice_model(
persona_id="persona-123",
data=model_data,
filename="tts_model.pt",
model_type="tts",
version="v1",
)
# List recordings
recordings = await voice_manager.list_recordings(
persona_id="persona-123",
session_id="session-456",
)
Knowledge Document Storage#
python
from psyche_storage import KnowledgeStorageManager
knowledge_manager = KnowledgeStorageManager(client)
# Upload document
result = await knowledge_manager.upload_document(
persona_id="persona-123",
data=document_data,
filename="company_info.pdf",
title="Company Information",
document_type="reference",
tags=["company", "faq"],
)
# Upload pre-computed embeddings
result = await knowledge_manager.upload_embeddings(
persona_id="persona-123",
data=embeddings_data,
document_key="knowledge/documents/persona-123/2024/01/01/doc.pdf",
embedding_model="openai/text-embedding-3-small",
)
# Download document
doc = await knowledge_manager.download_document(
"knowledge/documents/persona-123/2024/01/01/uuid-doc.pdf"
)
# List documents
docs = await knowledge_manager.list_documents(persona_id="persona-123")
Session Storage#
python
from psyche_storage import SessionStorageManager
session_manager = SessionStorageManager(client)
# Upload session recording
result = await session_manager.upload_recording(
session_id="session-123",
data=recording_data,
filename="full_session.webm",
recording_type="full",
)
# Upload transcript
result = await session_manager.upload_transcript(
session_id="session-123",
transcript='{"turns": [...]}',
format="json",
)
Training Data Storage#
python
from psyche_storage import TrainingStorageManager
training_manager = TrainingStorageManager(client)
# Upload training data
result = await training_manager.upload_training_data(
model_id="model-123",
data=dataset_data,
filename="training_data.jsonl",
data_type="dataset",
)
# Upload checkpoint
result = await training_manager.upload_checkpoint(
model_id="model-123",
data=checkpoint_data,
filename="checkpoint_epoch_10.pt",
epoch=10,
step=50000,
metrics={"loss": 0.05, "accuracy": 0.95},
)
# List checkpoints
checkpoints = await training_manager.list_checkpoints(model_id="model-123")
Utility Functions#
python
from psyche_storage import (
format_bytes,
parse_bytes,
get_mime_type,
sanitize_filename,
compute_sha256,
generate_file_key,
)
# Format bytes
print(format_bytes(1536)) # "1.50 KB"
# Parse size string
size = parse_bytes("1.5 MB") # 1572864
# Get MIME type
mime = get_mime_type("image.jpg") # "image/jpeg"
# Sanitize filename
safe = sanitize_filename("unsafe/name<>.txt") # "unsafe_name.txt"
# Compute checksum
hash = compute_sha256(data)
# Generate unique key
key = generate_file_key(prefix="uploads", filename="image.jpg")
Asset Categories#
The library defines the following asset categories for organized storage:
| Category | Path Prefix |
|---|---|
| Avatar Models | avatars/models/ |
| Avatar Textures | avatars/textures/ |
| Avatar Animations | avatars/animations/ |
| Avatar Thumbnails | avatars/thumbnails/ |
| Voice Recordings | voice/recordings/ |
| Voice Samples | voice/samples/ |
| Voice Models | voice/models/ |
| Knowledge Documents | knowledge/documents/ |
| Knowledge Embeddings | knowledge/embeddings/ |
| Training Data | training/data/ |
| Training Checkpoints | training/checkpoints/ |
| Session Recordings | sessions/recordings/ |
| Session Transcripts | sessions/transcripts/ |
Environment Variables#
bash
# AWS S3
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_REGION=us-east-1
S3_BUCKET=psyche-assets
# MinIO (local)
MINIO_ENDPOINT=localhost:9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin
MINIO_BUCKET=psyche-assets
Integration with @oshun/storage#
This library is designed to be compatible with the TypeScript @oshun/storage:
- Same storage class options
- Same signed URL patterns
- Compatible metadata structure
- Interoperable asset organization
Assets stored from Python can be accessed by TypeScript services and vice versa.
Dependencies#
pydantic>= 2.5.0 - Type validationboto3>= 1.34.0 - AWS S3 SDK
License#
Proprietary - Oshun Platform