From a4724841f7dda5f4ad8196dd054db145f6fd11db Mon Sep 17 00:00:00 2001 From: Pauljd Date: Sun, 1 Nov 2026 18:53:32 +0000 Subject: [PATCH] Add full library import and request missing metadata (0.19.0) The box test showed year, rating, date and original title empty because requests never asked for them. Asks for them now, restarts the test import so no rows lack them, and adds an import-everything option paced for a library of about 132,000 titles. Fits the coverage table on screen. Co-Authored-By: Claude Opus 5.5 (1M context) --- VERIFICATION-0.19.0.md | 13 +++++++++++++ android/app/build.gradle.kts | 4 ++-- .../java/tv/orbit/classic/CatalogueImport.kt | 14 +++++++++----- .../java/tv/orbit/classic/CatalogueScreen.kt | 17 +++++++++++++---- .../java/tv/orbit/classic/CatalogueStore.kt | 16 +++++++++++----- .../main/java/tv/orbit/classic/EmbyClient.kt | 2 +- .../tv/orbit/classic/EmbyConnectionScreen.kt | 9 ++++++--- .../tv/orbit/classic/EmbyPlaybackProtocol.kt | 4 ++-- .../main/java/tv/orbit/classic/MetadataAudit.kt | 2 +- .../tv/orbit/classic/MetadataCoverageScreen.kt | 13 ++++++++++--- .../tv/orbit/classic/SecureMediaDataSource.kt | 2 +- .../tv/orbit/classic/CatalogueImportTest.kt | 12 ++++++++++-- .../java/tv/orbit/classic/MetadataAuditTest.kt | 2 +- docs/metadata-and-matching.md | 8 ++++++-- 14 files changed, 86 insertions(+), 32 deletions(-) create mode 100644 VERIFICATION-0.19.0.md diff --git a/VERIFICATION-0.19.0.md b/VERIFICATION-0.19.0.md new file mode 100644 index 0000000..0ca5baf --- /dev/null +++ b/VERIFICATION-0.19.0.md @@ -0,0 +1,13 @@ +# Orbit TV 0.19.0 — full library import + +Follows the results from 0.17 and 0.18 on the box. Includes everything since 0.16. + +Evidence from Paul's account, 1 November 2026: 98,441 films, 34,103 series, 1,053,303 episodes. Metadata sample of 20 of each: genres 20/20 on films and series, external IDs 20/20 on films and series and 18/20 on episodes, studios 16/20 and 19/20, cast and crew 20/20 and 16/20, overview 20/20. Year, rating, date and original title were 0/20 on all types because those requests didn't ask for them; the tags, runtime and episode rows were off the screen. Import test: 1,000 titles in 11 seconds with no provider errors over 4,000 titles, about 266 bytes per title, and genres on 3,998 and external IDs on 3,997 of the 4,000. Year and rating were 0 there too. The box has 21 GB free; the full import should take about 35 MB. + +Changes: both queries now request ProductionYear, OfficialRating, PremiereDate, OriginalTitle and Tags. The catalogue stores original title and tags too. Its database version rises to 2, which discards the earlier test import, because those rows have no year or rating; the import restarts from the beginning. Import everything runs all pages (about 663) with the same 1.5-second pause, about half an hour, keeping the screen on while it runs. It stops on a rate-limit, login or permission reply, or on leaving the screen, and carries on from the last whole page. Import next 1,000 stays. The summary refreshes every five pages, as it counts the whole table. The coverage table is two side-by-side tables, so all fields fit on screen. + +Limits: Fields lists aren't all in Emby's published options, so whether the provider honours each new name is only proved by the next run. An unrecognised name may return nothing. Pause-to-pause pacing hasn't been tried over a whole library, and neither the box nor the provider has done this much in one go. After Dark and series searches still don't request year and rating; change them once this run confirms the names. + +Validation: offline assembleDebug, testDebugUnitTest and lintDebug passed; 108 unit tests (107 existing, 1 new), zero failures or errors, no lint errors. A new fixture test imports a 132,544-title library to the end: 663 pages, 662 pauses, no repeated page. SQLite code and real provider behaviour need the Xiaomi test. No private account, catalogue, media or device data was read by tools, beyond the counts Paul sent and free storage on the box. + +Rollback: reinstall outputs/revision-0.18.0 if needed. Its database version is older, so clear Orbit TV's app data first and sign in again, which also removes channels and saved setup. Otherwise revert this commit. diff --git a/android/app/build.gradle.kts b/android/app/build.gradle.kts index 83484e4..79e6267 100644 --- a/android/app/build.gradle.kts +++ b/android/app/build.gradle.kts @@ -9,8 +9,8 @@ android { applicationId = "tv.orbit.classic" minSdk = 26 targetSdk = 36 - versionCode = 18 - versionName = "0.18.0-catalogue-import" + versionCode = 19 + versionName = "0.19.0-full-import" } buildFeatures { compose = true } compileOptions { sourceCompatibility = JavaVersion.VERSION_17; targetCompatibility = JavaVersion.VERSION_17 } diff --git a/android/app/src/main/java/tv/orbit/classic/CatalogueImport.kt b/android/app/src/main/java/tv/orbit/classic/CatalogueImport.kt index af627dd..1d7bfec 100644 --- a/android/app/src/main/java/tv/orbit/classic/CatalogueImport.kt +++ b/android/app/src/main/java/tv/orbit/classic/CatalogueImport.kt @@ -4,7 +4,8 @@ import org.json.JSONArray import org.json.JSONObject internal data class CatalogueRecord(val id: String, val type: String, val title: String, val year: Int?, val rating: String, - val genres: List, val studios: List, val providerIds: Map, val premiere: String, val runtimeMs: Long?) + val genres: List, val studios: List, val providerIds: Map, val premiere: String, val runtimeMs: Long?, + val originalTitle: String = "", val tags: List = emptyList()) internal class CataloguePage(val records: List, val rawCount: Int, val total: Long?) internal data class CatalogueRun(val pages: Int, val records: Int, val finished: Boolean) @@ -16,6 +17,7 @@ internal interface CatalogueSink { internal const val CATALOGUE_PAGE_SIZE = 200 internal const val CATALOGUE_PAGES_PER_RUN = 5 +internal const val CATALOGUE_PAGES_FULL_RUN = 5_000 internal const val CATALOGUE_PAUSE_MS = 1_500L private fun names(array: JSONArray?, limit: Int = 8): List = (0 until minOf(array?.length() ?: 0, 32)).mapNotNull { index -> @@ -39,13 +41,15 @@ private fun catalogueRecord(row: JSONObject): CatalogueRecord? { val runtime = (row.opt("RunTimeTicks") as? Number)?.toLong()?.div(10_000)?.takeIf { it in 1_000..SampleChannel.MAX_DURATION_MS } return CatalogueRecord(id, type, title, (row.opt("ProductionYear") as? Number)?.toInt()?.takeIf { it in 1800..2200 }, EmbyClient.clean((row.opt("OfficialRating") as? String).orEmpty(), 40), names(row.optJSONArray("Genres")), names(row.optJSONArray("Studios")), - external, EmbyClient.clean((row.opt("PremiereDate") as? String).orEmpty(), 40), runtime) + external, EmbyClient.clean((row.opt("PremiereDate") as? String).orEmpty(), 40), runtime, + EmbyClient.clean((row.opt("OriginalTitle") as? String).orEmpty(), 160), + names(row.optJSONArray("TagItems") ?: row.optJSONArray("Tags"))) } /** Oldest first, so titles added during an import land after the current position. */ internal fun fetchCataloguePage(transport: EmbyTransport, userId: String, token: () -> String, start: Long): CataloguePage { require(EmbyClient.validId(userId) && start >= 0) - val reply = transport.execute(EmbyRequest("GET", "/Items?UserId=$userId&Recursive=true&IncludeItemTypes=Movie,Series&StartIndex=$start&Limit=$CATALOGUE_PAGE_SIZE&SortBy=DateCreated&SortOrder=Ascending&Fields=Genres,ProviderIds,Studios&EnableImages=false&EnableUserData=false", token = token())) + val reply = transport.execute(EmbyRequest("GET", "/Items?UserId=$userId&Recursive=true&IncludeItemTypes=Movie,Series&StartIndex=$start&Limit=$CATALOGUE_PAGE_SIZE&SortBy=DateCreated&SortOrder=Ascending&Fields=Genres,OfficialRating,OriginalTitle,PremiereDate,ProductionYear,ProviderIds,Studios,Tags&EnableImages=false&EnableUserData=false", token = token())) EmbyClient.expectSuccess(reply.status, false) val page = EmbyClient.parse(reply.body) val rows = page.optJSONArray("Items") ?: throw EmbyCheckException("The provider did not return a catalogue page.") @@ -55,10 +59,10 @@ internal fun fetchCataloguePage(transport: EmbyTransport, userId: String, token: return CataloguePage(records, count, total) } -/** One explicit run of at most five pages. A rate-limit or login failure stops it; finished pages stay saved. */ +/** One explicit run, either five pages or the whole library. A rate-limit or login failure stops it; finished pages stay saved. */ internal fun importCatalogue(transport: EmbyTransport, userId: String, token: () -> String, sink: CatalogueSink, maxPages: Int = CATALOGUE_PAGES_PER_RUN, pause: () -> Unit = {}, checkActive: () -> Unit = {}, onPage: () -> Unit = {}): CatalogueRun { - require(maxPages in 1..CATALOGUE_PAGES_PER_RUN) + require(maxPages in 1..CATALOGUE_PAGES_FULL_RUN) var start = sink.position(userId) var pages = 0 var written = 0 diff --git a/android/app/src/main/java/tv/orbit/classic/CatalogueScreen.kt b/android/app/src/main/java/tv/orbit/classic/CatalogueScreen.kt index 88ec563..7b7647c 100644 --- a/android/app/src/main/java/tv/orbit/classic/CatalogueScreen.kt +++ b/android/app/src/main/java/tv/orbit/classic/CatalogueScreen.kt @@ -4,12 +4,14 @@ import androidx.activity.compose.BackHandler import androidx.compose.foundation.background import androidx.compose.foundation.layout.* import androidx.compose.runtime.Composable +import androidx.compose.runtime.DisposableEffect import androidx.compose.runtime.LaunchedEffect import androidx.compose.runtime.remember import androidx.compose.ui.Modifier import androidx.compose.ui.focus.FocusRequester import androidx.compose.ui.focus.focusRequester import androidx.compose.ui.graphics.Color +import androidx.compose.ui.platform.LocalView import androidx.compose.ui.unit.dp import androidx.compose.ui.unit.sp import androidx.tv.material3.Text @@ -19,15 +21,22 @@ private fun whole(value: Long) = String.format(Locale.UK, "%,d", value) @Composable internal fun CatalogueScreen(summary: CatalogueSummary?, busy: Boolean, canRun: Boolean, message: String?, lastRun: String?, - modifier: Modifier, onRun: () -> Unit, onBack: () -> Unit) { + modifier: Modifier, onRun: (Boolean) -> Unit, onBack: () -> Unit) { BackHandler(onBack = onBack) val backFocus = remember { FocusRequester() } LaunchedEffect(Unit) { backFocus.requestFocus() } + // A screensaver stops the app, and so the import, so the screen stays on only while it runs. + val view = LocalView.current + DisposableEffect(view, busy) { + view.keepScreenOn = busy + onDispose { view.keepScreenOn = false } + } Column(modifier.fillMaxWidth().background(Navy).padding(18.dp), verticalArrangement = Arrangement.spacedBy(8.dp)) { Text("Library import test", color = Yellow, fontSize = 26.sp) - Text("Saves film and series details on this box, 1,000 titles per press. Channels and schedules are not changed, and no episodes are read.", color = Color.White, fontSize = 15.sp) + Text("Saves film and series details on this box. Channels and schedules are not changed, and no episodes are read. Importing everything takes about half an hour.", color = Color.White, fontSize = 15.sp) Row(horizontalArrangement = Arrangement.spacedBy(8.dp)) { - OrbitButton(if (busy) "Importing…" else if (summary?.finished == true) "Import finished" else "Import next 1,000 titles", enabled = canRun && summary?.finished != true, onClick = onRun) + OrbitButton(if (busy) "Importing…" else if (summary?.finished == true) "Import finished" else "Import everything", enabled = canRun && summary?.finished != true, onClick = { onRun(true) }) + OrbitButton("Import next 1,000", enabled = canRun && summary?.finished != true, onClick = { onRun(false) }) OrbitButton("Back", Modifier.focusRequester(backFocus), onClick = onBack) } message?.let { Text(it, color = Yellow, fontSize = 15.sp, maxLines = 3) } @@ -36,7 +45,7 @@ internal fun CatalogueScreen(summary: CatalogueSummary?, busy: Boolean, canRun: Column(Modifier.fillMaxWidth().background(Panel).padding(12.dp), verticalArrangement = Arrangement.spacedBy(4.dp)) { Text("Position ${whole(s.next)}${s.total?.let { " of ${whole(it)}" }.orEmpty()}", color = Color.White, fontSize = 19.sp) Text("Saved: ${whole(s.films)} films, ${whole(s.series)} series", color = Color.White, fontSize = 17.sp) - Text("With genres ${whole(s.withGenres)} • external IDs ${whole(s.withIds)} • year ${whole(s.withYear)} • rating ${whole(s.withRating)}", color = Muted, fontSize = 15.sp) + Text("With genres ${whole(s.withGenres)} • external IDs ${whole(s.withIds)} • year ${whole(s.withYear)} • rating ${whole(s.withRating)} • date ${whole(s.withDate)} • tags ${whole(s.withTags)}", color = Muted, fontSize = 15.sp) val perRecord = if (s.records > 0) " • about ${whole(s.bytes / s.records)} bytes per title" else "" Text("Database ${whole(s.bytes / 1024)} KB$perRecord", color = Muted, fontSize = 15.sp) } diff --git a/android/app/src/main/java/tv/orbit/classic/CatalogueStore.kt b/android/app/src/main/java/tv/orbit/classic/CatalogueStore.kt index c6be9ac..60a59a9 100644 --- a/android/app/src/main/java/tv/orbit/classic/CatalogueStore.kt +++ b/android/app/src/main/java/tv/orbit/classic/CatalogueStore.kt @@ -8,20 +8,25 @@ import org.json.JSONObject import java.io.File internal data class CatalogueSummary(val films: Long, val series: Long, val next: Long, val total: Long?, val finished: Boolean, - val withGenres: Long, val withIds: Long, val withYear: Long, val withRating: Long, val bytes: Long) { + val withGenres: Long, val withIds: Long, val withYear: Long, val withRating: Long, val withDate: Long, val withTags: Long, val bytes: Long) { val records get() = films + series } /** Library metadata for one signed-in account, in app-private storage excluded from backup. Holds no login. */ internal class CatalogueStore private constructor(context: Context) : CatalogueSink, - SQLiteOpenHelper(context, File(context.noBackupFilesDir, "catalogue.db").path, null, 1) { + SQLiteOpenHelper(context, File(context.noBackupFilesDir, "catalogue.db").path, null, 2) { private val file = File(context.noBackupFilesDir, "catalogue.db") override fun onCreate(db: SQLiteDatabase) { - db.execSQL("CREATE TABLE items(id TEXT PRIMARY KEY, type TEXT NOT NULL, title TEXT NOT NULL, year INTEGER, rating TEXT NOT NULL, genres TEXT NOT NULL, studios TEXT NOT NULL, provider_ids TEXT NOT NULL, premiere TEXT NOT NULL, runtime_ms INTEGER)") + db.execSQL("CREATE TABLE items(id TEXT PRIMARY KEY, type TEXT NOT NULL, title TEXT NOT NULL, year INTEGER, rating TEXT NOT NULL, genres TEXT NOT NULL, studios TEXT NOT NULL, provider_ids TEXT NOT NULL, premiere TEXT NOT NULL, runtime_ms INTEGER, original_title TEXT NOT NULL, tags TEXT NOT NULL)") db.execSQL("CREATE INDEX items_type_title ON items(type, title)") db.execSQL("CREATE TABLE meta(key TEXT PRIMARY KEY, value TEXT NOT NULL)") } - override fun onUpgrade(db: SQLiteDatabase, oldVersion: Int, newVersion: Int) = Unit + /** Version 1 was a test import without years or ratings, so it restarts rather than staying partly filled. */ + override fun onUpgrade(db: SQLiteDatabase, oldVersion: Int, newVersion: Int) { + db.execSQL("DROP TABLE IF EXISTS items") + db.execSQL("DROP TABLE IF EXISTS meta") + onCreate(db) + } private fun meta(db: SQLiteDatabase, key: String): String? = db.rawQuery("SELECT value FROM meta WHERE key = ?", arrayOf(key)).use { if (it.moveToFirst()) it.getString(0) else null } private fun setMeta(db: SQLiteDatabase, key: String, value: String) { @@ -41,6 +46,7 @@ internal class CatalogueStore private constructor(context: Context) : CatalogueS put("id", r.id); put("type", r.type); put("title", r.title); put("year", r.year); put("rating", r.rating) put("genres", r.genres.joinToString("|")); put("studios", r.studios.joinToString("|")) put("provider_ids", JSONObject(r.providerIds as Map<*, *>).toString()); put("premiere", r.premiere); put("runtime_ms", r.runtimeMs) + put("original_title", r.originalTitle); put("tags", r.tags.joinToString("|")) }, SQLiteDatabase.CONFLICT_REPLACE) } setMeta(db, "next", nextStart.toString()) @@ -54,7 +60,7 @@ internal class CatalogueStore private constructor(context: Context) : CatalogueS fun count(where: String) = db.rawQuery("SELECT COUNT(*) FROM items WHERE $where", null).use { it.moveToFirst(); it.getLong(0) } return CatalogueSummary(count("type = 'Movie'"), count("type = 'Series'"), meta(db, "next")?.toLongOrNull() ?: 0, meta(db, "total")?.toLongOrNull(), meta(db, "finished") == "true", count("genres != ''"), count("provider_ids != '{}'"), - count("year IS NOT NULL"), count("rating != ''"), file.length()) + count("year IS NOT NULL"), count("rating != ''"), count("premiere != ''"), count("tags != ''"), file.length()) } @Synchronized fun clear() { close() diff --git a/android/app/src/main/java/tv/orbit/classic/EmbyClient.kt b/android/app/src/main/java/tv/orbit/classic/EmbyClient.kt index 514f5ec..e6dbc6c 100644 --- a/android/app/src/main/java/tv/orbit/classic/EmbyClient.kt +++ b/android/app/src/main/java/tv/orbit/classic/EmbyClient.kt @@ -41,7 +41,7 @@ internal class HttpsEmbyTransport(internal val deviceId: String = UUID.randomUUI connection.connectTimeout = 10_000 connection.readTimeout = if (request.path == "/Sessions/Logout") 5_000 else 15_000 connection.setRequestProperty("Accept", "application/json") - connection.setRequestProperty("X-Emby-Authorization", "Emby Client=\"Orbit TV\", Device=\"Orbit TV\", DeviceId=\"$deviceId\", Version=\"0.18.0\"") + connection.setRequestProperty("X-Emby-Authorization", "Emby Client=\"Orbit TV\", Device=\"Orbit TV\", DeviceId=\"$deviceId\", Version=\"0.19.0\"") request.token?.let { connection.setRequestProperty("X-Emby-Token", it) } if (request.method == "POST") { val bytes = request.body.orEmpty().toByteArray(Charsets.UTF_8) diff --git a/android/app/src/main/java/tv/orbit/classic/EmbyConnectionScreen.kt b/android/app/src/main/java/tv/orbit/classic/EmbyConnectionScreen.kt index a66e4e6..3227392 100644 --- a/android/app/src/main/java/tv/orbit/classic/EmbyConnectionScreen.kt +++ b/android/app/src/main/java/tv/orbit/classic/EmbyConnectionScreen.kt @@ -232,7 +232,7 @@ internal fun EmbyConnectionScreen(modifier: Modifier = Modifier, visible: Boolea catch (cancelled: CancellationException) { throw cancelled } catch (_: Exception) { message = "The library data could not be read." } } - CatalogueScreen(catalogueSummary, busy, !busy && session != null, message, catalogueRun, modifier, onRun = { + CatalogueScreen(catalogueSummary, busy, !busy && session != null, message, catalogueRun, modifier, onRun = { everything -> val active = session if (!busy && active != null) { busy = true; message = null; catalogueRun = null @@ -241,8 +241,11 @@ internal fun EmbyConnectionScreen(modifier: Modifier = Modifier, visible: Boolea try { val result = withContext(Dispatchers.IO) { val context = coroutineContext - active.importCatalogue(catalogue, pause = { Thread.sleep(CATALOGUE_PAUSE_MS) }, checkActive = { context.ensureActive() }, - onPage = { val latest = catalogue.summary(); scope.launch { catalogueSummary = latest } }) + var pages = 0 + active.importCatalogue(catalogue, if (everything) CATALOGUE_PAGES_FULL_RUN else CATALOGUE_PAGES_PER_RUN, + pause = { Thread.sleep(CATALOGUE_PAUSE_MS) }, checkActive = { context.ensureActive() }, + // The summary counts the whole table, so it refreshes every few pages, not on each one. + onPage = { if (++pages % 5 == 0) { val latest = catalogue.summary(); scope.launch { catalogueSummary = latest } } }) } catalogueRun = "Last run: ${result.records} titles in ${result.pages} pages, ${(System.currentTimeMillis() - started) / 1000} seconds." } catch (cancelled: CancellationException) { throw cancelled } diff --git a/android/app/src/main/java/tv/orbit/classic/EmbyPlaybackProtocol.kt b/android/app/src/main/java/tv/orbit/classic/EmbyPlaybackProtocol.kt index 79dba28..0128255 100644 --- a/android/app/src/main/java/tv/orbit/classic/EmbyPlaybackProtocol.kt +++ b/android/app/src/main/java/tv/orbit/classic/EmbyPlaybackProtocol.kt @@ -149,8 +149,8 @@ internal class EmbySampleSession(private val transport: EmbyTransport, private v fun checkMetadataCoverage(checkActive: () -> Unit = {}): List = auditMetadataCoverage(transport, userId, ::currentToken, checkActive) - fun importCatalogue(sink: CatalogueSink, pause: () -> Unit = {}, checkActive: () -> Unit = {}, onPage: () -> Unit = {}): CatalogueRun = - importCatalogue(transport, userId, ::currentToken, sink, pause = pause, checkActive = checkActive, onPage = onPage) + fun importCatalogue(sink: CatalogueSink, maxPages: Int = CATALOGUE_PAGES_PER_RUN, pause: () -> Unit = {}, checkActive: () -> Unit = {}, onPage: () -> Unit = {}): CatalogueRun = + importCatalogue(transport, userId, ::currentToken, sink, maxPages, pause, checkActive, onPage) fun checkAfterDarkMovies(genre: String): List { val found = discoverAfterDarkMovies(transport, userId, ::currentToken, genre) currentToken() diff --git a/android/app/src/main/java/tv/orbit/classic/MetadataAudit.kt b/android/app/src/main/java/tv/orbit/classic/MetadataAudit.kt index 01017d8..9aaa5e5 100644 --- a/android/app/src/main/java/tv/orbit/classic/MetadataAudit.kt +++ b/android/app/src/main/java/tv/orbit/classic/MetadataAudit.kt @@ -37,7 +37,7 @@ internal fun auditMetadataCoverage(transport: EmbyTransport, userId: String, tok require(EmbyClient.validId(userId)) return AuditTypes.map { (type, label) -> checkActive() - val reply = transport.execute(EmbyRequest("GET", "/Items?UserId=$userId&Recursive=true&IncludeItemTypes=$type&StartIndex=0&Limit=$AUDIT_LIMIT&SortBy=Random&Fields=Genres,Overview,People,ProviderIds,Studios&EnableImages=false&EnableUserData=false", token = token())) + val reply = transport.execute(EmbyRequest("GET", "/Items?UserId=$userId&Recursive=true&IncludeItemTypes=$type&StartIndex=0&Limit=$AUDIT_LIMIT&SortBy=Random&Fields=Genres,OfficialRating,OriginalTitle,Overview,People,PremiereDate,ProductionYear,ProviderIds,Studios,Tags&EnableImages=false&EnableUserData=false", token = token())) EmbyClient.expectSuccess(reply.status, false) val page = EmbyClient.parse(reply.body) val rows = page.optJSONArray("Items") ?: throw EmbyCheckException("The provider did not return a metadata sample.") diff --git a/android/app/src/main/java/tv/orbit/classic/MetadataCoverageScreen.kt b/android/app/src/main/java/tv/orbit/classic/MetadataCoverageScreen.kt index 6eb1f29..21320ba 100644 --- a/android/app/src/main/java/tv/orbit/classic/MetadataCoverageScreen.kt +++ b/android/app/src/main/java/tv/orbit/classic/MetadataCoverageScreen.kt @@ -30,9 +30,16 @@ internal fun MetadataCoverageScreen(coverage: List?, busy: Boo message?.let { Text(it, color = Yellow, fontSize = 15.sp, maxLines = 3) } if (coverage != null) { CoverageRow("Sample", coverage.map { sample -> "${sample.checked}${sample.total?.let { " of $it" }.orEmpty()}" }, Yellow) - CoverageRow("", coverage.map { it.label }, Yellow) - MetadataCoverageLabels.forEach { label -> - CoverageRow(label, coverage.map { sample -> sample.fields.firstOrNull { it.label == label }?.let { "${it.present}/${sample.checked}" } ?: "–" }, Color.White) + // Two tables side by side: one long list runs off the bottom of the TV with nothing to scroll it. + Row(horizontalArrangement = Arrangement.spacedBy(8.dp)) { + MetadataCoverageLabels.chunked((MetadataCoverageLabels.size + 1) / 2).forEach { labels -> + Column(Modifier.weight(1f), verticalArrangement = Arrangement.spacedBy(6.dp)) { + CoverageRow("", coverage.map { it.label }, Yellow) + labels.forEach { label -> + CoverageRow(label, coverage.map { sample -> sample.fields.firstOrNull { it.label == label }?.let { "${it.present}/${sample.checked}" } ?: "–" }, Color.White) + } + } + } } Text("This is a sample, not a library total. A zero can mean the provider didn't return that detail for this request.", color = Muted, fontSize = 13.sp) } diff --git a/android/app/src/main/java/tv/orbit/classic/SecureMediaDataSource.kt b/android/app/src/main/java/tv/orbit/classic/SecureMediaDataSource.kt index 3d65c5b..9fcbb27 100644 --- a/android/app/src/main/java/tv/orbit/classic/SecureMediaDataSource.kt +++ b/android/app/src/main/java/tv/orbit/classic/SecureMediaDataSource.kt @@ -33,7 +33,7 @@ internal class SecureMediaDataSource(private val session: EmbySampleSession) : B next.instanceFollowRedirects = false next.connectTimeout = 10_000 next.readTimeout = 20_000 - next.setRequestProperty("User-Agent", "OrbitTV/0.18.0") + next.setRequestProperty("User-Agent", "OrbitTV/0.19.0") next.setRequestProperty("Accept-Encoding", "identity") session.mediaHeaders(address).forEach { (key, value) -> next.setRequestProperty(key, value) } if (dataSpec.position != 0L || dataSpec.length != C.LENGTH_UNSET.toLong()) { diff --git a/android/app/src/test/java/tv/orbit/classic/CatalogueImportTest.kt b/android/app/src/test/java/tv/orbit/classic/CatalogueImportTest.kt index 9c4675c..0c5975b 100644 --- a/android/app/src/test/java/tv/orbit/classic/CatalogueImportTest.kt +++ b/android/app/src/test/java/tv/orbit/classic/CatalogueImportTest.kt @@ -37,10 +37,17 @@ class CatalogueImportTest { requests.forEach { r -> assertEquals("GET", r.method); assertEquals("fixture-token", r.token); assertFalse(r.path.contains("fixture-token")) assertTrue(r.path.contains("IncludeItemTypes=Movie,Series&")); assertTrue(r.path.contains("Limit=200&")) - assertTrue(r.path.contains("SortBy=DateCreated&SortOrder=Ascending")); assertFalse(r.path.contains("Episode")) + assertTrue(r.path.contains("SortBy=DateCreated&SortOrder=Ascending")); assertTrue(r.path.contains("ProductionYear")); assertTrue(r.path.contains("OfficialRating")); assertFalse(r.path.contains("Episode")) } assertEquals(1000, sink.records.size); assertEquals(1000L, sink.next); assertEquals(120_000L, sink.total); assertFalse(sink.finished) } + @Test fun aFullRunReachesTheEndOfALibraryThisSize() { + val requests = mutableListOf(); val sink = MemorySink(); var pauses = 0 + val result = importCatalogue(library(132_544, requests), "fixture-user", { "fixture-token" }, sink, CATALOGUE_PAGES_FULL_RUN, pause = { pauses++ }) + assertEquals(CatalogueRun(663, 132_544, true), result); assertEquals(662, pauses); assertEquals(663, requests.size) + assertTrue(sink.finished); assertEquals(132_544, sink.records.size) + assertEquals(requests.map(::startOf), requests.map(::startOf).distinct()) + } @Test fun aLaterRunResumesWhereTheLastStopped() { val requests = mutableListOf(); val sink = MemorySink(); val server = library(120_000, requests) run(server, sink); requests.clear() @@ -84,11 +91,12 @@ class CatalogueImportTest { val full = row("full").put("Name", "T".repeat(400)).put("ProductionYear", 1999).put("OfficialRating", "18").put("PremiereDate", "1999-01-01T00:00:00Z") .put("RunTimeTicks", 54_000_000_000L).put("Genres", JSONArray().put("Horror").put("Cult|Classic")) .put("Studios", JSONArray().put(JSONObject().put("Name", "Fixture Studio"))).put("ProviderIds", JSONObject().put("Imdb", "tt0").put("Empty", "")) + .put("OriginalTitle", "Fixture original").put("TagItems", JSONArray().put(JSONObject().put("Name", "slasher"))) val odd = row("odd").put("ProductionYear", "1999").put("OfficialRating", 7).put("Genres", "Horror").put("Studios", JSONObject()).put("ProviderIds", JSONArray()).put("RunTimeTicks", -5).put("PremiereDate", 9) val page = fetchCataloguePage(EmbyTransport { EmbyReply(200, JSONObject().put("Items", JSONArray().put(full).put(odd)).toString()) }, "fixture-user", { "fixture-token" }, 0) val first = page.records[0] assertEquals(160, first.title.length); assertEquals(1999, first.year); assertEquals("18", first.rating); assertEquals(5_400_000L, first.runtimeMs) - assertEquals(listOf("Horror", "Cult/Classic"), first.genres); assertEquals(listOf("Fixture Studio"), first.studios); assertEquals(mapOf("Imdb" to "tt0"), first.providerIds) + assertEquals(listOf("Horror", "Cult/Classic"), first.genres); assertEquals(listOf("Fixture Studio"), first.studios); assertEquals(mapOf("Imdb" to "tt0"), first.providerIds); assertEquals("Fixture original", first.originalTitle); assertEquals(listOf("slasher"), first.tags) val second = page.records[1] assertNull(second.year); assertEquals("", second.rating); assertTrue(second.genres.isEmpty()); assertTrue(second.studios.isEmpty()); assertTrue(second.providerIds.isEmpty()); assertNull(second.runtimeMs); assertEquals("", second.premiere) } diff --git a/android/app/src/test/java/tv/orbit/classic/MetadataAuditTest.kt b/android/app/src/test/java/tv/orbit/classic/MetadataAuditTest.kt index eb06861..41d3763 100644 --- a/android/app/src/test/java/tv/orbit/classic/MetadataAuditTest.kt +++ b/android/app/src/test/java/tv/orbit/classic/MetadataAuditTest.kt @@ -24,7 +24,7 @@ class MetadataAuditTest { requests.forEach { r -> assertEquals("GET", r.method); assertEquals("fixture-token", r.token) assertTrue(r.path.contains("Limit=20&")); assertTrue(r.path.contains("SortBy=Random")) - assertTrue(r.path.contains("Fields=Genres,Overview,People,ProviderIds,Studios&")) + assertTrue(r.path.contains("Fields=Genres,OfficialRating,OriginalTitle,Overview,People,PremiereDate,ProductionYear,ProviderIds,Studios,Tags&")) assertFalse(r.path.contains("fixture-token")) } } diff --git a/docs/metadata-and-matching.md b/docs/metadata-and-matching.md index fd7d5c6..893f75b 100644 --- a/docs/metadata-and-matching.md +++ b/docs/metadata-and-matching.md @@ -50,7 +50,7 @@ Documentary versus scripted, and traditional versus CGI, need explicit values in ## Library scale -The 0.17 check on Paul's account reported 98,441 films, 34,102 series and 1,053,298 episodes on 1 November 2026. At 200 titles a page, films and series take about 663 requests. Episodes would take over 5,000, which is why they aren't imported up front. The design has to hold at that size: +The checks on Paul's account reported 98,441 films, 34,103 series and 1,053,303 episodes on 1 November 2026. At 200 titles a page, films and series take about 663 requests. Episodes would take over 5,000, which is why they aren't imported up front. The design has to hold at that size: - **Storage:** an indexed on-device database, not the saved setup file. Android's built-in SQLite needs no new dependency. - **Import:** paged in the background, resumable, throttled, and stopped by a rate-limit reply. Films and series first. @@ -60,7 +60,11 @@ The 0.17 check on Paul's account reported 98,441 films, 34,102 series and 1,053, - **Review screens:** show a short ranked list, with paging for the rest. - **Playback checks:** only for titles going onto a channel, never for the whole library. -Real record sizes, box storage and provider limits still need measuring before these numbers are final. +Measured so far: 1,000 titles in 11 seconds with no provider errors over 4,000 titles, about 266 bytes per title, so about 35 MB in total, with 21 GB free on the box. A full run has not been tried yet. + +## What the metadata supports + +Early samples show genres and external IDs on nearly every film and series (3,998 and 3,997 of the first 4,000 imported), so stable-ID identification and genre matching look reliable. Studios and cast are usable but not complete, and episodes carry little of their own, so they should inherit from their series. Year, rating, date, original title and tags still need the 0.19 run, because earlier requests didn't ask for them. The first 4,000 titles and 20-item samples are not a full picture. ## Recommendation engine design