Skip to main content

Product specification: news.lihor.ro

Purpose​

A self-hosted, open-source news dashboard. It combines:

  1. News inbox — Clau/cron collects useful sources automatically via RSS, GitHub release feeds, trending feeds, and scraped pages.
  2. Reading tracker — records what was read, saved, skipped, and archived with timestamps.
  3. Summary archive — stores contextual summaries and "why this matters" blurbs per article card.
  4. Search — full-text search across titles, summaries, tags, and sources.
  5. Source health — each feed shows last check time, success/error state, and item counts.
  6. AI question layer — Ask AI answers over saved/read history with citations and optional graph context.
  7. Knowledge graph — entity and relationship exploration over recent visible articles.

Source taxonomy​

See backend/news_dashboard/sources.py for the canonical source list.

Sources are split by category:

  • python — Python Insider, Astral blog, Ruff/uv/mypy/pyright/scikit-learn/scipy/PyTorch/TensorFlow releases
  • ai-llm — Anthropic (scraped), OpenAI, Google AI, Hugging Face, Augment Code, Simon Willison, Latent Space, Import AI, InfoQ
  • agents — LangChain, LangGraph, Langfuse releases
  • cloud-infra — Kubernetes, Docker, AWS ML blog
  • engineering — Pragmatic Engineer, GitHub Changelog, GitHub Engineering
  • trending — Hacker News Best, Hacker News AI search
  • repositories — GitHub Trending (All, Python, TypeScript)

Source kinds:

  • rss_feed — standard RSS/Atom feed via feedparser
  • github_release_feed — GitHub releases Atom feed
  • trending_feed — HN/GitHub trending RSS
  • scraped_page — custom HTML scraper (stdlib urllib + html.parser, no extra deps)

Article status model​

new → read
new → saved → read
new → skipped
any → archived
archived → read  (restore)

Status transitions are tracked with timestamps (read_at, saved_at, skipped_at, archived_at).

Noise filtering​

Per-source limits are configured in ingest.py::NOISE_FILTERS:

  • Broad feeds (HN, GitHub trending): capped at 15–20 items per run
  • Newsletter feeds (Import AI, Latent Space): capped at 5 per run
  • Curated blog feeds: capped at 50 per run (feedparser default)

A keyword include-list hook is available per-source for future tighter filtering.

Summary / reason generation​

Summaries are the first 280 characters of feed description (after HTML cleaning).

The reason field is a contextual blurb generated by make_reason():

  • Release feeds → "New release vX.Y.Z from source."
  • Security content → "Security update from source — review recommended."
  • Tutorial/how-to → "How-to or deep-dive on category from source."
  • Trending (HN) → "Trending on Hacker News."
  • Trending (GitHub) → "Trending {Language} repository on GitHub today."
  • AI/agent content → "AI/agent development news from source."
  • Python content → "Python ecosystem update from source."
  • Fallback → "{Category} — {Source}."

Source health tracking​

Each source stores:

  • last_checked_at — timestamp of last fetch attempt
  • last_success_at — timestamp of last successful fetch
  • last_error — last error message (null if no error)
  • last_fetched_count — items found in last run
  • last_inserted_count — new items inserted in last run

The Sources tab in the UI shows a colored health badge (ok / stale / error) and truncated error message when present.

Non-goals for v1​

  • No public unauthenticated access.
  • No broad all-tech firehose.
  • No complex AI Q&A before article history exists.
  • No manual article creation as a primary workflow.

Future AI layer (v1.1+)​

After ≥ 100 saved/read articles exist:

Data model (ready now)​

  • Articles already store summary, tags, reason — sufficient for keyword search.
  • PostgreSQL full-text search (search_vector tsvector generated column and idx_articles_search GIN index) is created and indexes title/summary/reason/tags/source_name/body.
  • Backend /api/search?q=... endpoint is live.

Full-text extraction (v1.1)​

  • Add optional full_text column to articles table.
  • Fetch full article body for saved/read articles (via Trafilatura or goose3).
  • Run FTS over full_text when available.

Embeddings / semantic search (v1.2)​

  • Embed title + summary via the configured OpenAI-compatible embeddings provider.
  • Store embeddings in PostgreSQL-managed article data structures; runtime storage remains PostgreSQL only.
  • Enables semantic search and similarity grouping without adding a second runtime database.
  • Configure with FREE_LLM_API_KEY / FREE_LLM_BASE_URL (or the OPENAI_API_KEY / OPENAI_BASE_URL fallback); see README.md and backend/news_dashboard/embeddings.py.

Ask AI endpoint and knowledge graph​

Scope: questions over articles visible to the user. By default this is Starred and Done; users can include all non-archived articles.

Proposed API:

POST /api/ask
{ "question": "What LangGraph updates happened last month?", "limit": 20 }
→ { "answer": "...", "citations": [{ "id": 42, "title": "...", "url": "..." }] }

Implementation:

  1. Use pgvector in PostgreSQL to retrieve candidate articles.
  2. Bundle bounded article context into the prompt.
  3. If Neo4j is enabled, add bounded entity relationship context for those same candidate article IDs.
  4. Call the configured OpenAI-compatible LLM gateway and return article citations plus graph context provenance.
  5. Return answer + article IDs used as citations.

Privacy/security:

  • Endpoint requires the app authentication boundary, either local password sessions or optional Keycloak SSO.
  • No article content is sent to external APIs unless the relevant AI feature is configured with an API key.
  • OpenAI API key stored as an environment secret, never in source.
  • Briefing generation uses FREE_LLM_API_KEY / FREE_LLM_BASE_URL (or the OPENAI_API_KEY fallback) and OPENAI_BRIEFING_MODEL; see README.md and backend/news_dashboard/briefings.py.

Privacy/security boundaries​

  • No article content leaves the server unless the relevant feature-specific AI provider variables are configured.
  • Search index uses PostgreSQL full-text search.
  • Local password auth is built into the app, and production can enable Keycloak SSO while preserving local user_id authorization boundaries. See Authentication (Keycloak).
  • The TLS Ingress is the production application route. Caddy is retained only as a transitional boundary for the existing same-host Keycloak route and cannot bind public ports alongside the ingress controller.

Deployment​

Container​

Built via Dockerfile at repo root. Published to GHCR via GitHub Actions on every push to main.

ghcr.io/lihor-hub/news-dashboard:latest
ghcr.io/lihor-hub/news-dashboard:<sha>
ghcr.io/lihor-hub/news-dashboard@sha256:<digest>  # production runtime

Kubernetes (Helm)​

(
set -euo pipefail
IMAGE_DIGEST="${IMAGE_DIGEST:?set IMAGE_DIGEST to sha256:<64 lowercase hex>}"
: "${SESSION_SECRET:?set SESSION_SECRET}"
: "${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}"
: "${POSTGRES_HOST_PATH:?set POSTGRES_HOST_PATH}"
source ./scripts/production-deploy-lib.sh
production_cutover_enabled || { echo "Ingress cutover is not enabled" >&2; exit 2; }
prepare_production_helm_secret_files

helm upgrade --install news-dashboard ./helm/news-dashboard \
  --namespace news-dashboard --create-namespace \
  --values ./helm/news-dashboard/values-production.yaml \
  --set-string image.digest="${IMAGE_DIGEST}" \
  --set-string postgresql.persistence.hostPath="$POSTGRES_HOST_PATH" \
  --set-file app.auth.sessionSecret="$PRODUCTION_SESSION_SECRET_FILE" \
  --set-file postgresql.password="$PRODUCTION_POSTGRES_PASSWORD_FILE"
)

Production renders a TLS-enabled Ingress and a ClusterIP-only application Service. CI validates the rendered contract without appliance access; the self-hosted production job leaves the existing release untouched unless INGRESS_CUTOVER_ENABLED is exactly true, then verifies the public HTTPS hostname after rollout.

Caddy and Keycloak migration boundary​

deploy/Caddyfile no longer routes the application. Its remaining /keycloak proxy records the route that operators must preserve while moving ports 80 and 443 to the ingress controller. Prepare rollback to the previous Helm revision and saved live Caddy configuration before removing the old application route. Follow Ingress HTTPS and Caddy migration and record the appliance-only work in human rollout issue #1302.