Eve library

Psyche Storage

Other

Revision 1

On this page

S3-compatible storage client for Psyche AI Virtual Assistant.

Overview#

psyche-storage provides a unified storage abstraction aligned with the @oshun/storage TypeScript patterns for cross-language compatibility. It includes specialized asset managers for avatar models, voice recordings, and knowledge documents.

Features#

  • S3/MinIO Support: Compatible with AWS S3 and MinIO backends
  • Async Operations: All operations are async/await compatible
  • Multipart Uploads: Automatic chunking for large files with progress tracking
  • Signed URLs: Generate pre-signed URLs for direct upload/download
  • Asset Managers: Specialized managers for Psyche-specific assets
  • Metadata Tracking: Automatic metadata for asset categorization and tracking

Installation#

bash
# Basic installation
poetry add psyche-storage

# For development
poetry add psyche-storage[dev]

Quick Start#

Creating a Storage Client#

python
from psyche_storage import S3Config, StorageProvider, create_s3_client, create_minio_client

# AWS S3
config = S3Config(
    provider=StorageProvider.S3,
    region="us-east-1",
    access_key_id="your-access-key",
    secret_access_key="your-secret-key",
    bucket="psyche-assets",
)
client = create_s3_client(config)

# MinIO (local development)
client = create_minio_client(
    endpoint="localhost:9000",
    access_key_id="minioadmin",
    secret_access_key="minioadmin",
    bucket="psyche-assets",
    use_ssl=False,
)

Basic Operations#

python
# Upload a file
result = await client.upload(
    key="path/to/file.txt",
    body=b"file content",
    options=UploadOptions(
        content_type="text/plain",
        metadata={"custom-key": "custom-value"},
    ),
)

# Download a file
result = await client.download("path/to/file.txt")
print(result.body)

# Check if file exists
exists = await client.exists("path/to/file.txt")

# Get file metadata
metadata = await client.get_metadata("path/to/file.txt")

# List files
result = await client.list_objects(
    ListObjectsOptions(prefix="path/to/", max_keys=100)
)
for obj in result.objects:
    print(f"{obj.key}: {obj.size} bytes")

# Delete a file
await client.delete("path/to/file.txt")

Multipart Upload for Large Files#

python
from psyche_storage import MultipartUploadConfig, MultipartUploadProgress

def progress_callback(progress: MultipartUploadProgress):
    print(f"Upload progress: {progress.percentage}%")

result = await client.upload_multipart(
    key="large-file.zip",
    body=large_data,
    options=UploadOptions(content_type="application/zip"),
    multipart_config=MultipartUploadConfig(
        part_size=10 * 1024 * 1024,  # 10 MB parts
        concurrency=4,
        on_progress=progress_callback,
    ),
)

Signed URLs#

python
from psyche_storage import SignedUrlOptions

# Generate download URL (expires in 1 hour)
result = await client.get_signed_download_url(
    "path/to/file.pdf",
    SignedUrlOptions(
        expires_in=3600,
        response_content_disposition='attachment; filename="document.pdf"',
    ),
)
print(f"Download URL: {result.url}")
print(f"Expires at: {result.expires_at}")

# Generate upload URL
result = await client.get_signed_upload_url(
    "uploads/new-file.jpg",
    SignedUrlOptions(
        expires_in=3600,
        content_type="image/jpeg",
    ),
)

Asset Managers#

Avatar Storage#

python
from psyche_storage import AvatarStorageManager

avatar_manager = AvatarStorageManager(client)

# Upload avatar model
result = await avatar_manager.upload_model(
    avatar_id="avatar-123",
    data=model_data,
    filename="character.glb",
    version="v2",
    persona_id="persona-456",
)

# Upload texture
result = await avatar_manager.upload_texture(
    avatar_id="avatar-123",
    data=texture_data,
    filename="diffuse.png",
    texture_type="diffuse",
)

# Upload animation
result = await avatar_manager.upload_animation(
    avatar_id="avatar-123",
    data=animation_data,
    filename="wave.glb",
    animation_name="wave_greeting",
)

# Upload thumbnail
result = await avatar_manager.upload_thumbnail(
    avatar_id="avatar-123",
    data=thumbnail_data,
)

# List avatar assets
assets = await avatar_manager.list_avatar_assets(
    avatar_id="avatar-123",
    category="models",
)

# Delete all avatar assets
await avatar_manager.delete_avatar("avatar-123")

Voice Storage#

python
from psyche_storage import VoiceStorageManager

voice_manager = VoiceStorageManager(client)

# Upload voice recording
result = await voice_manager.upload_recording(
    persona_id="persona-123",
    data=audio_data,
    filename="session_001.mp3",
    session_id="session-456",
    duration_ms=30000,
    transcript="Hello, how can I help you today?",
)

# Upload voice sample for cloning
result = await voice_manager.upload_sample(
    persona_id="persona-123",
    data=sample_data,
    filename="voice_sample.wav",
    sample_type="voice_clone",
)

# Upload voice model
result = await voice_manager.upload_voice_model(
    persona_id="persona-123",
    data=model_data,
    filename="tts_model.pt",
    model_type="tts",
    version="v1",
)

# List recordings
recordings = await voice_manager.list_recordings(
    persona_id="persona-123",
    session_id="session-456",
)

Knowledge Document Storage#

python
from psyche_storage import KnowledgeStorageManager

knowledge_manager = KnowledgeStorageManager(client)

# Upload document
result = await knowledge_manager.upload_document(
    persona_id="persona-123",
    data=document_data,
    filename="company_info.pdf",
    title="Company Information",
    document_type="reference",
    tags=["company", "faq"],
)

# Upload pre-computed embeddings
result = await knowledge_manager.upload_embeddings(
    persona_id="persona-123",
    data=embeddings_data,
    document_key="knowledge/documents/persona-123/2024/01/01/doc.pdf",
    embedding_model="openai/text-embedding-3-small",
)

# Download document
doc = await knowledge_manager.download_document(
    "knowledge/documents/persona-123/2024/01/01/uuid-doc.pdf"
)

# List documents
docs = await knowledge_manager.list_documents(persona_id="persona-123")

Session Storage#

python
from psyche_storage import SessionStorageManager

session_manager = SessionStorageManager(client)

# Upload session recording
result = await session_manager.upload_recording(
    session_id="session-123",
    data=recording_data,
    filename="full_session.webm",
    recording_type="full",
)

# Upload transcript
result = await session_manager.upload_transcript(
    session_id="session-123",
    transcript='{"turns": [...]}',
    format="json",
)

Training Data Storage#

python
from psyche_storage import TrainingStorageManager

training_manager = TrainingStorageManager(client)

# Upload training data
result = await training_manager.upload_training_data(
    model_id="model-123",
    data=dataset_data,
    filename="training_data.jsonl",
    data_type="dataset",
)

# Upload checkpoint
result = await training_manager.upload_checkpoint(
    model_id="model-123",
    data=checkpoint_data,
    filename="checkpoint_epoch_10.pt",
    epoch=10,
    step=50000,
    metrics={"loss": 0.05, "accuracy": 0.95},
)

# List checkpoints
checkpoints = await training_manager.list_checkpoints(model_id="model-123")

Utility Functions#

python
from psyche_storage import (
    format_bytes,
    parse_bytes,
    get_mime_type,
    sanitize_filename,
    compute_sha256,
    generate_file_key,
)

# Format bytes
print(format_bytes(1536))  # "1.50 KB"

# Parse size string
size = parse_bytes("1.5 MB")  # 1572864

# Get MIME type
mime = get_mime_type("image.jpg")  # "image/jpeg"

# Sanitize filename
safe = sanitize_filename("unsafe/name<>.txt")  # "unsafe_name.txt"

# Compute checksum
hash = compute_sha256(data)

# Generate unique key
key = generate_file_key(prefix="uploads", filename="image.jpg")

Asset Categories#

The library defines the following asset categories for organized storage:

Category Path Prefix
Avatar Models avatars/models/
Avatar Textures avatars/textures/
Avatar Animations avatars/animations/
Avatar Thumbnails avatars/thumbnails/
Voice Recordings voice/recordings/
Voice Samples voice/samples/
Voice Models voice/models/
Knowledge Documents knowledge/documents/
Knowledge Embeddings knowledge/embeddings/
Training Data training/data/
Training Checkpoints training/checkpoints/
Session Recordings sessions/recordings/
Session Transcripts sessions/transcripts/

Environment Variables#

bash
# AWS S3
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_REGION=us-east-1
S3_BUCKET=psyche-assets

# MinIO (local)
MINIO_ENDPOINT=localhost:9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin
MINIO_BUCKET=psyche-assets

Integration with @oshun/storage#

This library is designed to be compatible with the TypeScript @oshun/storage:

  • Same storage class options
  • Same signed URL patterns
  • Compatible metadata structure
  • Interoperable asset organization

Assets stored from Python can be accessed by TypeScript services and vice versa.

Dependencies#

  • pydantic >= 2.5.0 - Type validation
  • boto3 >= 1.34.0 - AWS S3 SDK

License#

Proprietary - Oshun Platform