Goodreads Vision
Vision
Uvilo’s Self-Help Resources system needs a comprehensive, high-quality book database to power personalized reading recommendations within the Uvilo Method. Goodreads listopia lists provide community-curated, ranked book recommendations that serve as a valuable seed source for this database.
This project builds a pipeline to extract book data from Goodreads, transform it into Uvilo’s Resource schema, enrich it with supplementary metadata, and load it into the Uvilo resource database — making books available for taxonomy linking, AI-powered recommendations, and coaching conversations.
Background
The Uvilo Method (Section 9: Self-Help Resources) defines a multi-step ETL pipeline for indexing books with Google Books and Goodreads as designated data sources. A browser-based HTML-to-CSV converter tool (Goodreads_List_to_CSV.html) already exists for offline extraction of Goodreads listopia data. Five CSV files containing approximately 495 books have been exported from a self-help book listopia list, each with title, authors, ratings, cover image URLs, and Goodreads book IDs.
Research Instructions
Research should investigate:
- The existing Uvilo resource indexing pipeline (Section 9.5 of The Uvilo Method) and how Goodreads data feeds into it
- The production database schema for Resource Item, Book, Author, Resource Link, and Taxonomy Relation models
- The Google Books API as a supplementary metadata source (ISBN, publisher, pages, description)
- The existing CSV data format and the
goo_ID prefix convention - Whether the existing HTML-to-CSV tool needs enhancement or whether a different extraction approach is needed
- How Cloudinary image upload works in the existing indexing pipeline
Requirements
R1: Goodreads Data Extraction
The project must provide a reliable method for extracting book data from Goodreads listopia lists into a structured format. The existing HTML-to-CSV converter and exported CSV data serve as the starting point.
R2: Schema Transformation
The project must transform extracted Goodreads data into Uvilo’s canonical Book and Resource Item schema, mapping available fields (title, subtitle, authors, ratings, image URLs, book URLs) and generating required derived fields (slugs, IDs).
R3: Metadata Enrichment
The project must enrich book records with metadata not available from Goodreads list exports — including ISBN, publisher, page count, and full description — using supplementary sources such as the Google Books API.
R4: Deduplication
The project must identify and handle duplicate books appearing across multiple Goodreads lists, ensuring each book is represented once in the Uvilo resource database.
R5: Database Loading
The project must load transformed and enriched book records into the Uvilo production database, creating Resource Item and Book records with Resource Links to Goodreads. This project modifies production infrastructure and must verify data persistence and service correctness.
R6: Image Handling
The project must handle book cover images from Goodreads, consistent with the Uvilo resource indexing pipeline (Cloudinary upload or URL reference).
R7: Production Safety & Security
The project must safeguard production infrastructure and data throughout the loading pipeline. Secrets and credentials must not be committed to source control; database credentials must be provided via approved environment-based configuration with least-privilege or controlled access. Writes to the production database must be preceded by idempotent, dry-run, or verification safeguards to prevent unintended data corruption or duplication.
Success Criteria
| # | Criterion | Measured by |
|---|---|---|
| V1 | Book data is extractable from Goodreads listopia lists | Structured output contains all available fields for each book |
| V2 | Goodreads data maps to Uvilo Book/Resource Item schema | All required fields populated for each record |
| V3 | Book records are enriched with ISBN, publisher, pages, and description | Enrichment succeeds for ≥90% of books with valid Goodreads IDs |
| V4 | Duplicate books across lists are identified and merged | No duplicate books (by Goodreads book_id or ISBN) in the final database |
| V5 | Book records are loaded into the production database | Records are queryable via the resources API |
| V6 | Book cover images are accessible in the resource viewer | Images load correctly when viewing a book resource |
| V7 | Infrastructure changes are verified for data persistence and service correctness | Database writes persist across service restarts; Railway service version matches expected |
| V8 | Production safety and security safeguards are in place | No secrets committed to source control; credentials sourced from approved environment configuration with least-privilege access; production writes preceded by dry-run or verification step |
Out Of Scope
- AI-generated summaries, tags, and embeddings (handled by the existing resource indexing pipeline)
- Taxonomy linking (handled by the existing resource indexing pipeline)
- Publishing workflow and human approval (handled by the existing resource status system)
- Podcast, video, and other non-book resource types
- User-facing book recommendation UI (handled by the Uvilo coaching platform)
Charter Considerations
- Stewardship — The book database directly powers reading recommendations in the Uvilo Method. Data quality impacts user trust and recommendation accuracy.
- Performance tracking — Enrichment and loading steps involve external API calls and database operations. Cost telemetry should be tracked for any AI-assisted steps.
- Security — The pipeline writes to the production database and uses privileged tooling. Secrets must never be committed; credentials must use approved environment-based access; production writes must include dry-run or verification safeguards before execution.