skills v3

.ai/SKILL.md

@@ -0,0 +1,169 @@
+# Dune API Skill For OpenClaw Agents
+
+You are working with the Dune API, a Node.js Express service that imports Dune: Awakening data from Questlog into MongoDB and exposes it through simple REST endpoints for OpenClaw.
+
+## Primary Goal
+
+Help OpenClaw retrieve complete Dune: Awakening records with the least guesswork. Prefer the detailed singular datasets for client-facing lookups and search.
+
+## Public Base URL
+
+Use this production API base URL:
+
+```text
+https://dune.api.coppnic.cc
+```
+
+Swagger/OpenAPI is available here:
+
+```text
+https://dune.api.coppnic.cc/openapi.json
+https://ui.dune.api.coppnic.cc/docs
+```
+
+## Best Endpoints For OpenClaw
+
+Prefer singular datasets because they contain full Questlog single-record payloads in `raw`:
+
+```text
+GET /api/item
+GET /api/item/{id}
+GET /api/skill
+GET /api/skill/{id}
+GET /api/recipe
+GET /api/recipe/{id}
+GET /api/placeable
+GET /api/placeable/{id}
+GET /api/npc
+GET /api/npc/{id}
+```
+
+Useful examples:
+
+```text
+GET /api/item/LongRifle_Unique_Poison_03?language=en
+GET /api/item/Bloodsack_02?language=en
+GET /api/skill/skills_ability_poisonmine?language=en
+GET /api/recipe/Bloodsack_2_Recipe?language=en
+GET /api/placeable/Atre_Banner_Placeable?language=en
+GET /api/npc/bs43q?language=en
+```
+
+For text search:
+
+```text
+GET /api/search?q=rifle&datasets=item&language=en
+GET /api/search?q=poison&datasets=item,skill,recipe&language=en
+```
+
+Supported languages are `en` and `de`.
+
+## Dataset Rules
+
+Detailed singular datasets:
+
+| Dataset     | Mongo collection | Questlog detail method  |
+| ----------- | ---------------- | ----------------------- |
+| `item`      | `item`           | `database.getItem`      |
+| `skill`     | `skill`          | `database.getSkill`     |
+| `recipe`    | `recipe`         | `database.getRecipe`    |
+| `placeable` | `placeable`      | `database.getPlaceable` |
+| `npc`       | `npc`            | `database.getNpc`       |
+
+Summary plural datasets:
+
+| Dataset      | Mongo collection | Questlog page method     |
+| ------------ | ---------------- | ------------------------ |
+| `items`      | `items`          | `database.getItems`      |
+| `skills`     | `skills`         | `database.getSkills`     |
+| `recipes`    | `recipes`        | `database.getRecipes`    |
+| `placeables` | `placeables`     | `database.getPlaceables` |
+| `npcs`       | `npcs`           | `database.getNpcs`       |
+
+Use plural datasets only when a compact Questlog page-summary record is enough. If a plural lookup includes an id, for example `/api/items/LongRifle_Unique_Poison_03`, the API checks the matching singular collection first and falls back to the plural summary collection.
+
+## Response Shape
+
+Stored documents have normalized metadata and the original Questlog payload:
+
+```json
+{
+	"dataset": "item",
+	"language": "en",
+	"source": "questlog.gg",
+	"sourceMethod": "database.getItem",
+	"sourceId": "LongRifle_Unique_Poison_03",
+	"name": "Assassin's Rifle",
+	"raw": {}
+}
+```
+
+For OpenClaw, inspect `raw` for domain-specific fields:
+
+- Items: `raw.stats`, `raw.mainCategory`, `raw.subCategory`, recipe relationships, wearable/equip fields.
+- Skills: `raw.levels`, `raw.connections`, grid position, bonuses.
+- Recipes: `raw.recipeInputItems`, `raw.recipeOutputItems`, crafting requirements, crafting stations.
+- Placeables: production types, power/water fields, craftable recipe relationships.
+- NPCs: `raw.description`, `raw.npcTags`, category metadata.
+
+## Repository Map
+
+Look here when changing behavior:
+
+```text
+src/datasets.js                 Dataset keys, collections, Questlog methods, API allowlists.
+src/importer/questlogClient.js   Questlog URL building and TRPC response extraction.
+src/importer/importer.js         Import orchestration, detail fetches, Mongo upserts.
+src/db/indexes.js                Mongo indexes for summary and singular collections.
+src/routes/api.js                REST API routes, search, dataset lookup behavior.
+src/swagger/openapi.js           OpenAPI/Swagger documentation.
+src/app.js                       Express middleware and public top-level routes.
+scripts/import.js                CLI import entry point.
+```
+
+## Import Commands
+
+Run all imports:
+
+```powershell
+npm run import
+```
+
+Run a safe smoke import:
+
+```powershell
+npm run import:smoke
+```
+
+Import selected datasets:
+
+```powershell
+node scripts/import.js --datasets=items,skills,recipes,placeables,npcs --languages=en,de
+```
+
+The importer writes both plural summaries and singular detailed records when a dataset has a detail endpoint.
+
+## Validation
+
+Use the existing syntax check:
+
+```powershell
+npm run check
+```
+
+Useful endpoint checks:
+
+```text
+GET /health
+GET /api/datasets
+GET /api/item/LongRifle_Unique_Poison_03?language=en
+GET /api/search?q=poison&datasets=item,skill,recipe&language=en
+GET /SKILL.md
+```
+
+## Important Constraints
+
+- Do not expose secrets in public documentation or API responses.
+- Keep `.env` values private; production configuration comes from environment variables.
+- Keep the singular datasets first-class for OpenClaw.
+- Keep changes small and consistent with the existing CommonJS/Express style.

.dockerignore

@@ -2,6 +2,8 @@ node_modules
 npm-debug.log*
 .env
 .git
-.idea
+.idea/*
+!.idea/
+!.idea/SKILL.md
 coverage
 dist

.env.example

@@ -1,5 +1,8 @@
 PORT=3030
 API_HOST_PORT=8030
+PUBLIC_API_URL=https://dune.api.coppnic.cc
+PUBLIC_UI_URL=https://ui.dune.api.coppnic.cc
+FORCE_HTTPS=true
 MONGODB_URI=mongodb://root:change-me@37.60.245.70:27017
 MONGODB_DB=duneawa
 MAX_PAGES=

.gitignore

@@ -6,4 +6,6 @@ npm-debug.log*
 coverage/
 dist/
 .DS_Store
-.idea
+.idea/*
+!.idea/
+!.idea/SKILL.md

.idea/SKILL.md

@@ -0,0 +1,241 @@
+# Dune API Skill For OpenClaw Agents
+
+You are working with the Dune API, a Node.js Express service that imports Dune: Awakening data from Questlog into MongoDB and exposes it through REST endpoints for OpenClaw.
+
+## Primary Goal
+
+Help OpenClaw retrieve complete Dune: Awakening records with the least guesswork. Prefer the detailed singular datasets for client-facing lookups and search.
+
+## Public Base URL
+
+Use this production API base URL:
+
+```text
+https://dune.api.coppnic.cc
+```
+
+Swagger/OpenAPI is available here:
+
+```text
+https://dune.api.coppnic.cc/openapi.json
+https://ui.dune.api.coppnic.cc/docs
+```
+
+## Best Endpoints For OpenClaw
+
+Prefer singular datasets because they contain full Questlog single-record payloads in `raw`:
+
+```text
+GET /api/item
+GET /api/item/{id}
+GET /api/skill
+GET /api/skill/{id}
+GET /api/recipe
+GET /api/recipe/{id}
+GET /api/placeable
+GET /api/placeable/{id}
+GET /api/npc
+GET /api/npc/{id}
+```
+
+Useful examples:
+
+```text
+GET /api/item/LongRifle_Unique_Poison_03?language=en
+GET /api/item/Bloodsack_02?language=en
+GET /api/skill/skills_ability_poisonmine?language=en
+GET /api/recipe/Bloodsack_2_Recipe?language=en
+GET /api/placeable/Atre_Banner_Placeable?language=en
+GET /api/npc/bs43q?language=en
+```
+
+For text search:
+
+```text
+GET /api/search?q=rifle&datasets=item&language=en
+GET /api/search?q=poison&datasets=item,skill,recipe&language=en
+```
+
+Supported languages are `en` and `de`.
+
+## Dataset Rules
+
+Detailed singular datasets:
+
+| Dataset     | Mongo collection | Questlog detail method  |
+| ----------- | ---------------- | ----------------------- |
+| `item`      | `item`           | `database.getItem`      |
+| `skill`     | `skill`          | `database.getSkill`     |
+| `recipe`    | `recipe`         | `database.getRecipe`    |
+| `placeable` | `placeable`      | `database.getPlaceable` |
+| `npc`       | `npc`            | `database.getNpc`       |
+
+Summary plural datasets:
+
+| Dataset      | Mongo collection | Questlog page method     |
+| ------------ | ---------------- | ------------------------ |
+| `items`      | `items`          | `database.getItems`      |
+| `skills`     | `skills`         | `database.getSkills`     |
+| `recipes`    | `recipes`        | `database.getRecipes`    |
+| `placeables` | `placeables`     | `database.getPlaceables` |
+| `npcs`       | `npcs`           | `database.getNpcs`       |
+
+Use plural datasets only when a compact Questlog page-summary record is enough. If a plural lookup includes an id, for example `/api/items/LongRifle_Unique_Poison_03`, the API checks the matching singular collection first and falls back to the plural summary collection.
+
+## Response Shape
+
+Stored documents have normalized metadata and the original Questlog payload:
+
+```json
+{
+	"dataset": "item",
+	"language": "en",
+	"source": "questlog.gg",
+	"sourceMethod": "database.getItem",
+	"sourceId": "LongRifle_Unique_Poison_03",
+	"name": "Assassin's Rifle",
+	"raw": {}
+}
+```
+
+For OpenClaw, inspect `raw` for domain-specific fields:
+
+- Items: `raw.stats`, `raw.mainCategory`, `raw.subCategory`, recipe relationships, wearable/equip fields.
+- Skills: `raw.levels`, `raw.connections`, grid position, bonuses.
+- Recipes: `raw.recipeInputItems`, `raw.recipeOutputItems`, crafting requirements, crafting stations.
+- Placeables: production types, power/water fields, craftable recipe relationships.
+- NPCs: `raw.description`, `raw.npcTags`, category metadata.
+
+## Discord Markdown Output Contract
+
+Rich Discord embeds are not available yet. When OpenClaw asks for Discord output, return a plain Discord message string formatted with Discord-supported Markdown only. Do not return embed JSON.
+
+Use supported Markdown:
+
+- Headings with `#`, `##`, or `###` for short section titles.
+- Bold with asterisks, for example `**Assassin's Rifle**`. Do not use underscores for bold.
+- Italic sparingly for secondary text.
+- Unordered lists with `-` for fields and search results.
+- Ordered lists for ranked results.
+- Inline code for ids, dataset names, stat keys, and short values.
+- Fenced code blocks for compact JSON or command examples.
+- Links with `[label](url)` when a clean label helps.
+- Blockquotes with `>` or `>>>` for quoted descriptions.
+- Strikethrough only when it adds useful meaning.
+
+Avoid unsupported or unreliable formatting:
+
+- No Markdown tables.
+- No HTML.
+- No horizontal rules.
+- No task lists.
+- No footnotes, heading ids, definition lists, subscript, superscript, or highlight syntax.
+- No images in Markdown.
+- Do not rely on Markdown paragraph syntax or special line-break syntax; use normal newline-separated lines.
+
+Default Discord result format:
+
+```text
+## Assassin's Rifle
+**Type:** `item`
+**Language:** `en`
+**Source ID:** `LongRifle_Unique_Poison_03`
+**Category:** `weapon` / `rifle`
+
+**Stats**
+- Damage: `128.25`
+- Accuracy: `1.2`
+- Clip Size: `2`
+
+[Open in Dune API](https://dune.api.coppnic.cc/api/item/LongRifle_Unique_Poison_03?language=en)
+```
+
+Search result format:
+
+```text
+## Search Results: poison
+1. **Poison Mine** (`skill`) - `skills_ability_poisonmine`
+2. **Assassin's Rifle** (`item`) - `LongRifle_Unique_Poison_03`
+
+Use `/api/{dataset}/{id}?language=en` for details.
+```
+
+Discord response rules:
+
+- Keep responses concise enough for Discord message limits.
+- Show at most 10 compact search results unless the user asks for more.
+- For detailed records, prefer the most useful 5 to 8 fields.
+- Do not dump the full `raw` object.
+- Omit empty, null, unknown, or noisy fields.
+- Include a Dune API link for single-record answers.
+- Prefer singular datasets: `item`, `skill`, `recipe`, `placeable`, `npc`.
+
+Dataset field ideas:
+
+- `item`: grade, category, subcategory, weapon stats, armor stats, fillable stats.
+- `skill`: category, subcategory, max skill level, level bonuses.
+- `recipe`: output items, input items, crafting time, required stations.
+- `placeable`: category, power, water, supported production types.
+- `npc`: category, tags, description.
+
+## Repository Map
+
+Look here when changing behavior:
+
+```text
+src/datasets.js                 Dataset keys, collections, Questlog methods, API allowlists.
+src/importer/questlogClient.js   Questlog URL building and TRPC response extraction.
+src/importer/importer.js         Import orchestration, detail fetches, Mongo upserts.
+src/db/indexes.js                Mongo indexes for summary and singular collections.
+src/routes/api.js                REST API routes, search, dataset lookup behavior.
+src/swagger/openapi.js           OpenAPI/Swagger documentation.
+src/app.js                       Express middleware and public top-level routes.
+scripts/import.js                CLI import entry point.
+```
+
+## Import Commands
+
+Run all imports:
+
+```powershell
+npm run import
+```
+
+Run a safe smoke import:
+
+```powershell
+npm run import:smoke
+```
+
+Import selected datasets:
+
+```powershell
+node scripts/import.js --datasets=items,skills,recipes,placeables,npcs --languages=en,de
+```
+
+The importer writes both plural summaries and singular detailed records when a dataset has a detail endpoint.
+
+## Validation
+
+Use the existing syntax check:
+
+```powershell
+npm run check
+```
+
+Useful endpoint checks:
+
+```text
+GET /health
+GET /api/datasets
+GET /api/item/LongRifle_Unique_Poison_03?language=en
+GET /api/search?q=poison&datasets=item,skill,recipe&language=en
+GET /SKILL.md
+```
+
+## Important Constraints
+
+- Do not expose secrets in public documentation or API responses.
+- Keep `.env` values private; production configuration comes from environment variables.
+- Keep the singular datasets first-class for OpenClaw.
+- Keep changes small and consistent with the existing CommonJS/Express style.

Dockerfile

@@ -14,6 +14,9 @@ WORKDIR /app
 ENV NODE_ENV=production
 
 COPY --from=dependencies /app/node_modules ./node_modules
+COPY SKILL.md ./SKILL.md
+COPY .idea/SKILL.md ./.idea/SKILL.md
+COPY public ./public
 COPY src ./src
 COPY scripts ./scripts
 

README.md

@@ -10,7 +10,7 @@ npm run import:smoke
 npm start
 ```
 
-Open Swagger UI at `http://localhost:3030/docs`.
+Open Swagger UI locally at `http://localhost:3030/docs`.
 
 ## Docker
 
@@ -18,7 +18,26 @@ Open Swagger UI at `http://localhost:3030/docs`.
 docker compose up --build
 ```
 
-The API listens on `http://localhost:8030` by default when run through Docker Compose. Set `API_HOST_PORT=3031` if your machine needs the alternate host port.
+The container listens on `3030` and Docker Compose exposes it on `8030` by default. Set `API_HOST_PORT=3031` if your machine needs the alternate host port.
+
+## Public HTTPS Domains
+
+This app is configured for Dokploy/Traefik HTTPS by default. Traefik should terminate TLS and route both public HTTPS domains to the app container:
+
+```text
+https://dune.api.coppnic.cc    -> container port 3030
+https://ui.dune.api.coppnic.cc -> container port 3030
+```
+
+The public API URL is `https://dune.api.coppnic.cc`. Swagger UI is available at `https://ui.dune.api.coppnic.cc/docs`.
+
+The OpenAPI document advertises the HTTPS API domain by default, so Swagger requests go to the right public API host. These values can be adjusted through environment variables:
+
+```env
+PUBLIC_API_URL=https://dune.api.coppnic.cc
+PUBLIC_UI_URL=https://ui.dune.api.coppnic.cc
+FORCE_HTTPS=true
+```
 
 ## Import All Data
 
@@ -45,10 +64,38 @@ The importer pulls every page for all configured datasets and both supported lan
 - `GET /health`
 - `GET /docs`
 - `GET /openapi.json`
+- `GET /api/item`
+- `GET /api/item/{id}`
+- `GET /api/skill`
+- `GET /api/skill/{id}`
+- `GET /api/recipe`
+- `GET /api/recipe/{id}`
+- `GET /api/placeable`
+- `GET /api/placeable/{id}`
+- `GET /api/npc`
+- `GET /api/npc/{id}`
 - `GET /api/{dataset}`
 - `GET /api/{dataset}/{id}`
 - `GET /api/search?q=...`
 - `POST /api/import`
 - `GET /api/import/status`
 
-Datasets: `items`, `skills`, `recipes`, `placeables`, `npcs`.
+Use singular datasets for detailed records. These collections store the full Questlog single-record payloads. For example, `item` includes item-specific `raw.stats` structures such as `weaponStats`, `fillableStats`, and wearable stats.
+
+```text
+GET /api/item?language=en&limit=25
+GET /api/item/LongRifle_Unique_Poison_03?language=en
+GET /api/items/LongRifle_Unique_Poison_03?language=en
+GET /api/item/Bloodsack_02?language=de
+GET /api/skill/skills_ability_poisonmine?language=en
+GET /api/recipe/Bloodsack_2_Recipe?language=en
+GET /api/placeable/Atre_Banner_Placeable?language=en
+GET /api/npc/bs43q?language=en
+GET /api/search?q=rifle&datasets=item,skill,recipe&language=en
+```
+
+Public API datasets: `item`, `skill`, `recipe`, `placeable`, `npc`, `items`, `skills`, `recipes`, `placeables`, `npcs`.
+
+Plural datasets are the older paginated Questlog summary collections. For OpenClaw and other clients that need complete stats and relationships, prefer the singular datasets.
+
+For convenience, `GET /api/{pluralDataset}/{id}` checks the matching detailed singular collection first when one exists, then falls back to the older summary record.

SKILL.md

@@ -0,0 +1,134 @@
+# Dune API Skill For OpenClaw Agents
+
+Use `https://dune.api.coppnic.cc` as the API base URL.
+
+Prefer detailed singular datasets for OpenClaw because they contain complete Questlog single-record payloads in `raw`:
+
+```text
+GET /api/item/{id}
+GET /api/skill/{id}
+GET /api/recipe/{id}
+GET /api/placeable/{id}
+GET /api/npc/{id}
+```
+
+Examples:
+
+```text
+GET /api/item/LongRifle_Unique_Poison_03?language=en
+GET /api/item/Bloodsack_02?language=en
+GET /api/skill/skills_ability_poisonmine?language=en
+GET /api/recipe/Bloodsack_2_Recipe?language=en
+GET /api/placeable/Atre_Banner_Placeable?language=en
+GET /api/npc/bs43q?language=en
+GET /api/search?q=poison&datasets=item,skill,recipe&language=en
+```
+
+Supported languages are `en` and `de`.
+
+Detailed public datasets:
+
+```text
+item, skill, recipe, placeable, npc
+```
+
+Plural summary datasets:
+
+```text
+items, skills, recipes, placeables, npcs
+```
+
+Use singular datasets for complete stats and relationships. Use plural datasets only when a compact Questlog page-summary record is enough.
+
+## Discord Markdown Output Contract
+
+Rich Discord embeds are not available yet. When OpenClaw asks for Discord output, return a plain Discord message string formatted with Discord-supported Markdown only. Do not return embed JSON.
+
+Use supported Markdown:
+
+- Headings with `#`, `##`, or `###` for short section titles.
+- Bold with asterisks, for example `**Assassin's Rifle**`. Do not use underscores for bold.
+- Italic sparingly for secondary text.
+- Unordered lists with `-` for fields and search results.
+- Ordered lists for ranked results.
+- Inline code for ids, dataset names, stat keys, and short values.
+- Fenced code blocks for compact JSON or command examples.
+- Links with `[label](url)` when a clean label helps.
+- Blockquotes with `>` or `>>>` for quoted descriptions.
+- Strikethrough only when it adds useful meaning.
+
+Avoid unsupported or unreliable formatting:
+
+- No Markdown tables.
+- No HTML.
+- No horizontal rules.
+- No task lists.
+- No footnotes, heading ids, definition lists, subscript, superscript, or highlight syntax.
+- No images in Markdown.
+- Do not rely on Markdown paragraph syntax or special line-break syntax; use normal newline-separated lines.
+
+Default Discord result format:
+
+```text
+## Assassin's Rifle
+**Type:** `item`
+**Language:** `en`
+**Source ID:** `LongRifle_Unique_Poison_03`
+**Category:** `weapon` / `rifle`
+
+**Stats**
+- Damage: `128.25`
+- Accuracy: `1.2`
+- Clip Size: `2`
+
+[Open in Dune API](https://dune.api.coppnic.cc/api/item/LongRifle_Unique_Poison_03?language=en)
+```
+
+Search result format:
+
+```text
+## Search Results: poison
+1. **Poison Mine** (`skill`) - `skills_ability_poisonmine`
+2. **Assassin's Rifle** (`item`) - `LongRifle_Unique_Poison_03`
+
+Use `/api/{dataset}/{id}?language=en` for details.
+```
+
+Discord response rules:
+
+- Keep responses concise enough for Discord message limits.
+- Show at most 10 compact search results unless the user asks for more.
+- For detailed records, prefer the most useful 5 to 8 fields.
+- Do not dump the full `raw` object.
+- Omit empty, null, unknown, or noisy fields.
+- Include a Dune API link for single-record answers.
+- Prefer singular datasets: `item`, `skill`, `recipe`, `placeable`, `npc`.
+
+Dataset field ideas:
+
+- `item`: grade, category, subcategory, weapon stats, armor stats, fillable stats.
+- `skill`: category, subcategory, max skill level, level bonuses.
+- `recipe`: output items, input items, crafting time, required stations.
+- `placeable`: category, power, water, supported production types.
+- `npc`: category, tags, description.
+
+Repository guide:
+
+```text
+src/datasets.js                 Dataset keys, collections, Questlog methods, API allowlists.
+src/importer/questlogClient.js   Questlog URL building and TRPC response extraction.
+src/importer/importer.js         Import orchestration, detail fetches, Mongo upserts.
+src/db/indexes.js                Mongo indexes for summary and singular collections.
+src/routes/api.js                REST API routes, search, dataset lookup behavior.
+src/swagger/openapi.js           OpenAPI/Swagger documentation.
+src/app.js                       Express middleware and public top-level routes.
+scripts/import.js                CLI import entry point.
+```
+
+Useful checks:
+
+```text
+GET /health
+GET /api/datasets
+GET /SKILL.md
+```

docker-compose.yml

@@ -7,6 +7,9 @@ services:
       - "${API_HOST_PORT:-8030}:${PORT:-3030}"
     environment:
       PORT: ${PORT:-3030}
+      PUBLIC_API_URL: ${PUBLIC_API_URL:-https://dune.api.coppnic.cc}
+      PUBLIC_UI_URL: ${PUBLIC_UI_URL:-https://ui.dune.api.coppnic.cc}
+      FORCE_HTTPS: ${FORCE_HTTPS:-true}
       MONGODB_URI: ${MONGODB_URI}
       MONGODB_DB: ${MONGODB_DB:-duneawa}
       MAX_PAGES: ${MAX_PAGES:-}

public/SKILL.md

@@ -0,0 +1,213 @@
+# Dune API Skill For OpenClaw Agents
+
+You are working with the Dune API, a Node.js Express service that imports Dune: Awakening data from Questlog into MongoDB and exposes it through REST endpoints for OpenClaw.
+
+## Primary Goal
+
+Help OpenClaw retrieve complete Dune: Awakening records with the least guesswork. Prefer the detailed singular datasets for client-facing lookups and search.
+
+## Public Base URL
+
+Use this production API base URL:
+
+```text
+https://dune.api.coppnic.cc
+```
+
+Swagger/OpenAPI is available here:
+
+```text
+https://dune.api.coppnic.cc/openapi.json
+https://ui.dune.api.coppnic.cc/docs
+```
+
+## Best Endpoints For OpenClaw
+
+Prefer singular datasets because they contain full Questlog single-record payloads in `raw`:
+
+```text
+GET /api/item
+GET /api/item/{id}
+GET /api/skill
+GET /api/skill/{id}
+GET /api/recipe
+GET /api/recipe/{id}
+GET /api/placeable
+GET /api/placeable/{id}
+GET /api/npc
+GET /api/npc/{id}
+```
+
+Useful examples:
+
+```text
+GET /api/item/LongRifle_Unique_Poison_03?language=en
+GET /api/item/Bloodsack_02?language=en
+GET /api/skill/skills_ability_poisonmine?language=en
+GET /api/recipe/Bloodsack_2_Recipe?language=en
+GET /api/placeable/Atre_Banner_Placeable?language=en
+GET /api/npc/bs43q?language=en
+```
+
+For text search:
+
+```text
+GET /api/search?q=rifle&datasets=item&language=en
+GET /api/search?q=poison&datasets=item,skill,recipe&language=en
+```
+
+Supported languages are `en` and `de`.
+
+## Dataset Rules
+
+Detailed singular datasets:
+
+| Dataset     | Mongo collection | Questlog detail method  |
+| ----------- | ---------------- | ----------------------- |
+| `item`      | `item`           | `database.getItem`      |
+| `skill`     | `skill`          | `database.getSkill`     |
+| `recipe`    | `recipe`         | `database.getRecipe`    |
+| `placeable` | `placeable`      | `database.getPlaceable` |
+| `npc`       | `npc`            | `database.getNpc`       |
+
+Summary plural datasets:
+
+| Dataset      | Mongo collection | Questlog page method     |
+| ------------ | ---------------- | ------------------------ |
+| `items`      | `items`          | `database.getItems`      |
+| `skills`     | `skills`         | `database.getSkills`     |
+| `recipes`    | `recipes`        | `database.getRecipes`    |
+| `placeables` | `placeables`     | `database.getPlaceables` |
+| `npcs`       | `npcs`           | `database.getNpcs`       |
+
+Use plural datasets only when a compact Questlog page-summary record is enough. If a plural lookup includes an id, for example `/api/items/LongRifle_Unique_Poison_03`, the API checks the matching singular collection first and falls back to the plural summary collection.
+
+## Response Shape
+
+Stored documents have normalized metadata and the original Questlog payload:
+
+```json
+{
+	"dataset": "item",
+	"language": "en",
+	"source": "questlog.gg",
+	"sourceMethod": "database.getItem",
+	"sourceId": "LongRifle_Unique_Poison_03",
+	"name": "Assassin's Rifle",
+	"raw": {}
+}
+```
+
+For OpenClaw, inspect `raw` for domain-specific fields:
+
+- Items: `raw.stats`, `raw.mainCategory`, `raw.subCategory`, recipe relationships, wearable/equip fields.
+- Skills: `raw.levels`, `raw.connections`, grid position, bonuses.
+- Recipes: `raw.recipeInputItems`, `raw.recipeOutputItems`, crafting requirements, crafting stations.
+- Placeables: production types, power/water fields, craftable recipe relationships.
+- NPCs: `raw.description`, `raw.npcTags`, category metadata.
+
+## Discord Markdown Output Contract
+
+Rich Discord embeds are not available yet. When OpenClaw asks for Discord output, return a plain Discord message string formatted with Discord-supported Markdown only. Do not return embed JSON.
+
+Use supported Markdown:
+
+- Headings with `#`, `##`, or `###` for short section titles.
+- Bold with asterisks, for example `**Assassin's Rifle**`. Do not use underscores for bold.
+- Italic sparingly for secondary text.
+- Unordered lists with `-` for fields and search results.
+- Ordered lists for ranked results.
+- Inline code for ids, dataset names, stat keys, and short values.
+- Fenced code blocks for compact JSON or command examples.
+- Links with `[label](url)` when a clean label helps.
+- Blockquotes with `>` or `>>>` for quoted descriptions.
+- Strikethrough only when it adds useful meaning.
+
+Avoid unsupported or unreliable formatting:
+
+- No Markdown tables.
+- No HTML.
+- No horizontal rules.
+- No task lists.
+- No footnotes, heading ids, definition lists, subscript, superscript, or highlight syntax.
+- No images in Markdown.
+- Do not rely on Markdown paragraph syntax or special line-break syntax; use normal newline-separated lines.
+
+Default Discord result format:
+
+```text
+## Assassin's Rifle
+**Type:** `item`
+**Language:** `en`
+**Source ID:** `LongRifle_Unique_Poison_03`
+**Category:** `weapon` / `rifle`
+
+**Stats**
+- Damage: `128.25`
+- Accuracy: `1.2`
+- Clip Size: `2`
+
+[Open in Dune API](https://dune.api.coppnic.cc/api/item/LongRifle_Unique_Poison_03?language=en)
+```
+
+Search result format:
+
+```text
+## Search Results: poison
+1. **Poison Mine** (`skill`) - `skills_ability_poisonmine`
+2. **Assassin's Rifle** (`item`) - `LongRifle_Unique_Poison_03`
+
+Use `/api/{dataset}/{id}?language=en` for details.
+```
+
+Discord response rules:
+
+- Keep responses concise enough for Discord message limits.
+- Show at most 10 compact search results unless the user asks for more.
+- For detailed records, prefer the most useful 5 to 8 fields.
+- Do not dump the full `raw` object.
+- Omit empty, null, unknown, or noisy fields.
+- Include a Dune API link for single-record answers.
+- Prefer singular datasets: `item`, `skill`, `recipe`, `placeable`, `npc`.
+
+Dataset field ideas:
+
+- `item`: grade, category, subcategory, weapon stats, armor stats, fillable stats.
+- `skill`: category, subcategory, max skill level, level bonuses.
+- `recipe`: output items, input items, crafting time, required stations.
+- `placeable`: category, power, water, supported production types.
+- `npc`: category, tags, description.
+
+## Repository Map
+
+Look here when changing behavior:
+
+```text
+src/datasets.js                 Dataset keys, collections, Questlog methods, API allowlists.
+src/importer/questlogClient.js   Questlog URL building and TRPC response extraction.
+src/importer/importer.js         Import orchestration, detail fetches, Mongo upserts.
+src/db/indexes.js                Mongo indexes for summary and singular collections.
+src/routes/api.js                REST API routes, search, dataset lookup behavior.
+src/swagger/openapi.js           OpenAPI/Swagger documentation.
+src/app.js                       Express middleware and public top-level routes.
+scripts/import.js                CLI import entry point.
+```
+
+## Validation
+
+Useful endpoint checks:
+
+```text
+GET /health
+GET /api/datasets
+GET /api/item/LongRifle_Unique_Poison_03?language=en
+GET /api/search?q=poison&datasets=item,skill,recipe&language=en
+GET /SKILL.md
+```
+
+## Important Constraints
+
+- Do not expose secrets in public documentation or API responses.
+- Keep `.env` values private; production configuration comes from environment variables.
+- Keep the singular datasets first-class for OpenClaw.
+- Keep changes small and consistent with the existing CommonJS/Express style.

src/SKILL.md

@@ -0,0 +1,241 @@
+# Dune API Skill For OpenClaw Agents
+
+You are working with the Dune API, a Node.js Express service that imports Dune: Awakening data from Questlog into MongoDB and exposes it through REST endpoints for OpenClaw.
+
+## Primary Goal
+
+Help OpenClaw retrieve complete Dune: Awakening records with the least guesswork. Prefer the detailed singular datasets for client-facing lookups and search.
+
+## Public Base URL
+
+Use this production API base URL:
+
+```text
+https://dune.api.coppnic.cc
+```
+
+Swagger/OpenAPI is available here:
+
+```text
+https://dune.api.coppnic.cc/openapi.json
+https://ui.dune.api.coppnic.cc/docs
+```
+
+## Best Endpoints For OpenClaw
+
+Prefer singular datasets because they contain full Questlog single-record payloads in `raw`:
+
+```text
+GET /api/item
+GET /api/item/{id}
+GET /api/skill
+GET /api/skill/{id}
+GET /api/recipe
+GET /api/recipe/{id}
+GET /api/placeable
+GET /api/placeable/{id}
+GET /api/npc
+GET /api/npc/{id}
+```
+
+Useful examples:
+
+```text
+GET /api/item/LongRifle_Unique_Poison_03?language=en
+GET /api/item/Bloodsack_02?language=en
+GET /api/skill/skills_ability_poisonmine?language=en
+GET /api/recipe/Bloodsack_2_Recipe?language=en
+GET /api/placeable/Atre_Banner_Placeable?language=en
+GET /api/npc/bs43q?language=en
+```
+
+For text search:
+
+```text
+GET /api/search?q=rifle&datasets=item&language=en
+GET /api/search?q=poison&datasets=item,skill,recipe&language=en
+```
+
+Supported languages are `en` and `de`.
+
+## Dataset Rules
+
+Detailed singular datasets:
+
+| Dataset     | Mongo collection | Questlog detail method  |
+| ----------- | ---------------- | ----------------------- |
+| `item`      | `item`           | `database.getItem`      |
+| `skill`     | `skill`          | `database.getSkill`     |
+| `recipe`    | `recipe`         | `database.getRecipe`    |
+| `placeable` | `placeable`      | `database.getPlaceable` |
+| `npc`       | `npc`            | `database.getNpc`       |
+
+Summary plural datasets:
+
+| Dataset      | Mongo collection | Questlog page method     |
+| ------------ | ---------------- | ------------------------ |
+| `items`      | `items`          | `database.getItems`      |
+| `skills`     | `skills`         | `database.getSkills`     |
+| `recipes`    | `recipes`        | `database.getRecipes`    |
+| `placeables` | `placeables`     | `database.getPlaceables` |
+| `npcs`       | `npcs`           | `database.getNpcs`       |
+
+Use plural datasets only when a compact Questlog page-summary record is enough. If a plural lookup includes an id, for example `/api/items/LongRifle_Unique_Poison_03`, the API checks the matching singular collection first and falls back to the plural summary collection.
+
+## Response Shape
+
+Stored documents have normalized metadata and the original Questlog payload:
+
+```json
+{
+	"dataset": "item",
+	"language": "en",
+	"source": "questlog.gg",
+	"sourceMethod": "database.getItem",
+	"sourceId": "LongRifle_Unique_Poison_03",
+	"name": "Assassin's Rifle",
+	"raw": {}
+}
+```
+
+For OpenClaw, inspect `raw` for domain-specific fields:
+
+- Items: `raw.stats`, `raw.mainCategory`, `raw.subCategory`, recipe relationships, wearable/equip fields.
+- Skills: `raw.levels`, `raw.connections`, grid position, bonuses.
+- Recipes: `raw.recipeInputItems`, `raw.recipeOutputItems`, crafting requirements, crafting stations.
+- Placeables: production types, power/water fields, craftable recipe relationships.
+- NPCs: `raw.description`, `raw.npcTags`, category metadata.
+
+## Discord Markdown Output Contract
+
+Rich Discord embeds are not available yet. When OpenClaw asks for Discord output, return a plain Discord message string formatted with Discord-supported Markdown only. Do not return embed JSON.
+
+Use supported Markdown:
+
+- Headings with `#`, `##`, or `###` for short section titles.
+- Bold with asterisks, for example `**Assassin's Rifle**`. Do not use underscores for bold.
+- Italic sparingly for secondary text.
+- Unordered lists with `-` for fields and search results.
+- Ordered lists for ranked results.
+- Inline code for ids, dataset names, stat keys, and short values.
+- Fenced code blocks for compact JSON or command examples.
+- Links with `[label](url)` when a clean label helps.
+- Blockquotes with `>` or `>>>` for quoted descriptions.
+- Strikethrough only when it adds useful meaning.
+
+Avoid unsupported or unreliable formatting:
+
+- No Markdown tables.
+- No HTML.
+- No horizontal rules.
+- No task lists.
+- No footnotes, heading ids, definition lists, subscript, superscript, or highlight syntax.
+- No images in Markdown.
+- Do not rely on Markdown paragraph syntax or special line-break syntax; use normal newline-separated lines.
+
+Default Discord result format:
+
+```text
+## Assassin's Rifle
+**Type:** `item`
+**Language:** `en`
+**Source ID:** `LongRifle_Unique_Poison_03`
+**Category:** `weapon` / `rifle`
+
+**Stats**
+- Damage: `128.25`
+- Accuracy: `1.2`
+- Clip Size: `2`
+
+[Open in Dune API](https://dune.api.coppnic.cc/api/item/LongRifle_Unique_Poison_03?language=en)
+```
+
+Search result format:
+
+```text
+## Search Results: poison
+1. **Poison Mine** (`skill`) - `skills_ability_poisonmine`
+2. **Assassin's Rifle** (`item`) - `LongRifle_Unique_Poison_03`
+
+Use `/api/{dataset}/{id}?language=en` for details.
+```
+
+Discord response rules:
+
+- Keep responses concise enough for Discord message limits.
+- Show at most 10 compact search results unless the user asks for more.
+- For detailed records, prefer the most useful 5 to 8 fields.
+- Do not dump the full `raw` object.
+- Omit empty, null, unknown, or noisy fields.
+- Include a Dune API link for single-record answers.
+- Prefer singular datasets: `item`, `skill`, `recipe`, `placeable`, `npc`.
+
+Dataset field ideas:
+
+- `item`: grade, category, subcategory, weapon stats, armor stats, fillable stats.
+- `skill`: category, subcategory, max skill level, level bonuses.
+- `recipe`: output items, input items, crafting time, required stations.
+- `placeable`: category, power, water, supported production types.
+- `npc`: category, tags, description.
+
+## Repository Map
+
+Look here when changing behavior:
+
+```text
+src/datasets.js                 Dataset keys, collections, Questlog methods, API allowlists.
+src/importer/questlogClient.js   Questlog URL building and TRPC response extraction.
+src/importer/importer.js         Import orchestration, detail fetches, Mongo upserts.
+src/db/indexes.js                Mongo indexes for summary and singular collections.
+src/routes/api.js                REST API routes, search, dataset lookup behavior.
+src/swagger/openapi.js           OpenAPI/Swagger documentation.
+src/app.js                       Express middleware and public top-level routes.
+scripts/import.js                CLI import entry point.
+```
+
+## Import Commands
+
+Run all imports:
+
+```powershell
+npm run import
+```
+
+Run a safe smoke import:
+
+```powershell
+npm run import:smoke
+```
+
+Import selected datasets:
+
+```powershell
+node scripts/import.js --datasets=items,skills,recipes,placeables,npcs --languages=en,de
+```
+
+The importer writes both plural summaries and singular detailed records when a dataset has a detail endpoint.
+
+## Validation
+
+Use the existing syntax check:
+
+```powershell
+npm run check
+```
+
+Useful endpoint checks:
+
+```text
+GET /health
+GET /api/datasets
+GET /api/item/LongRifle_Unique_Poison_03?language=en
+GET /api/search?q=poison&datasets=item,skill,recipe&language=en
+GET /SKILL.md
+```
+
+## Important Constraints
+
+- Do not expose secrets in public documentation or API responses.
+- Keep `.env` values private; production configuration comes from environment variables.
+- Keep the singular datasets first-class for OpenClaw.
+- Keep changes small and consistent with the existing CommonJS/Express style.

src/app.js

@@ -1,18 +1,81 @@
 const cors = require("cors");
 const express = require("express");
+const fs = require("fs");
+const path = require("path");
 const swaggerUi = require("swagger-ui-express");
+const { config } = require("./config");
 const { pingMongo } = require("./db/client");
 const { router: apiRouter } = require("./routes/api");
 const { openApiDocument } = require("./swagger/openapi");
 
+const publicDir = path.join(__dirname, "..", "public");
+const skillFilePaths = [
+	path.join(__dirname, "..", "SKILL.md"),
+	path.join(publicDir, "SKILL.md"),
+	path.join(__dirname, "SKILL.md"),
+	path.join(__dirname, "..", ".idea", "SKILL.md"),
+	path.join(__dirname, "..", ".ai", "SKILL.md"),
+];
+
+const fallbackSkillMarkdown = `# Dune API Skill For OpenClaw Agents
+
+Use https://dune.api.coppnic.cc as the API base URL.
+
+Prefer detailed singular datasets for OpenClaw:
+
+- GET /api/item/{id}
+- GET /api/skill/{id}
+- GET /api/recipe/{id}
+- GET /api/placeable/{id}
+- GET /api/npc/{id}
+
+Examples:
+
+- GET /api/item/LongRifle_Unique_Poison_03?language=en
+- GET /api/skill/skills_ability_poisonmine?language=en
+- GET /api/recipe/Bloodsack_2_Recipe?language=en
+- GET /api/placeable/Atre_Banner_Placeable?language=en
+- GET /api/npc/bs43q?language=en
+
+Search example:
+
+- GET /api/search?q=poison&datasets=item,skill,recipe&language=en
+
+Supported languages: en, de.
+`;
+
+function getSkillFilePath() {
+	return skillFilePaths.find((filePath) => fs.existsSync(filePath));
+}
+
 function createApp() {
 	const app = express();
 
+	app.set("trust proxy", true);
 	app.use(cors());
 	app.use(express.json({ limit: "2mb" }));
+	app.use(express.static(publicDir));
+	app.use((request, response, next) => {
+		const forwardedProto = request.get("x-forwarded-proto");
+		const isSecure = request.secure || forwardedProto === "https";
+		const isLocalhost = ["localhost", "127.0.0.1", "::1"].includes(
+			request.hostname,
+		);
+
+		if (config.public.forceHttps && !isSecure && !isLocalhost) {
+			const baseUrl =
+				request.hostname === "ui.dune.api.coppnic.cc"
+					? config.public.uiUrl
+					: config.public.apiUrl;
+			response.redirect(308, new URL(request.originalUrl, baseUrl).toString());
+			return;
+		}
+
+		next();
+	});
 
 	app.get("/", (request, response) => {
-		response.redirect("/docs");
+		response.redirect(308, `${config.public.uiUrl}/docs`);
 	});
 
 	app.get("/health", async (request, response) => {
@@ -30,6 +93,18 @@ function createApp() {
 		response.json(openApiDocument);
 	});
 
+	app.get(["/SKILL.md", "/skill.md"], (request, response) => {
+		const skillFilePath = getSkillFilePath();
+		response.type("text/markdown");
+
+		if (skillFilePath) {
+			response.send(fs.readFileSync(skillFilePath, "utf8"));
+			return;
+		}
+
+		response.send(fallbackSkillMarkdown);
+	});
+
 	app.use("/docs", swaggerUi.serve, swaggerUi.setup(openApiDocument));
 	app.use("/api", apiRouter);
 

src/config.js

@@ -15,8 +15,21 @@ function parseOptionalPositiveInteger(value, name) {
 	return parsed;
 }
 
+function parseBoolean(value, fallback = false) {
+	if (value === undefined || value === null || value === "") {
+		return fallback;
+	}
+
+	return ["1", "true", "yes", "on"].includes(String(value).toLowerCase());
+}
+
 const config = {
 	port: parseOptionalPositiveInteger(process.env.PORT, "PORT") || 3030,
+	public: {
+		apiUrl: process.env.PUBLIC_API_URL || "https://dune.api.coppnic.cc",
+		uiUrl: process.env.PUBLIC_UI_URL || "https://ui.dune.api.coppnic.cc",
+		forceHttps: parseBoolean(process.env.FORCE_HTTPS, true),
+	},
 	mongodb: {
 		uri:
 			process.env.MONGODB_URI || "mongodb://root:63eba009@37.60.245.70:27017",
@@ -32,4 +45,4 @@ const config = {
 	},
 };
 
-module.exports = { config, parseOptionalPositiveInteger };
+module.exports = { config, parseBoolean, parseOptionalPositiveInteger };

src/datasets.js

@@ -4,39 +4,90 @@ const DATASETS = {
 		collection: "items",
 		method: "database.getItems",
 		singular: "item",
+		detail: {
+			key: "item",
+			collection: "item",
+			method: "database.getItem",
+		},
 	},
 	skills: {
 		key: "skills",
 		collection: "skills",
 		method: "database.getSkills",
 		singular: "skill",
+		detail: {
+			key: "skill",
+			collection: "skill",
+			method: "database.getSkill",
+		},
 	},
 	recipes: {
 		key: "recipes",
 		collection: "recipes",
 		method: "database.getRecipes",
 		singular: "recipe",
+		detail: {
+			key: "recipe",
+			collection: "recipe",
+			method: "database.getRecipe",
+		},
 	},
 	placeables: {
 		key: "placeables",
 		collection: "placeables",
 		method: "database.getPlaceables",
 		singular: "placeable",
+		detail: {
+			key: "placeable",
+			collection: "placeable",
+			method: "database.getPlaceable",
+		},
 	},
 	npcs: {
 		key: "npcs",
 		collection: "npcs",
 		method: "database.getNpcs",
 		singular: "npc",
+		detail: {
+			key: "npc",
+			collection: "npc",
+			method: "database.getNpc",
+		},
 	},
 };
 
+const DETAIL_DATASETS = Object.fromEntries(
+	Object.values(DATASETS)
+		.filter((dataset) => dataset.detail)
+		.map((dataset) => [
+			dataset.detail.key,
+			{
+				...dataset.detail,
+				singular: dataset.singular,
+				description: `Detailed ${dataset.singular} records from Questlog single-record data.`,
+			},
+		]),
+);
+
+const API_DATASETS = {
+	...DETAIL_DATASETS,
+	...DATASETS,
+};
+
+const DEFAULT_API_DATASET_KEYS = Object.keys(API_DATASETS).filter(
+	(datasetKey) => !DATASETS[datasetKey]?.detail,
+);
+
 const LANGUAGES = ["en", "de"];
 
 function getDataset(key) {
 	return DATASETS[key];
 }
 
+function getApiDataset(key) {
+	return API_DATASETS[key];
+}
+
 function assertDataset(key) {
 	const dataset = getDataset(key);
 	if (!dataset) {
@@ -50,6 +101,19 @@ function assertDataset(key) {
 	return dataset;
 }
 
+function assertApiDataset(key) {
+	const dataset = getApiDataset(key);
+	if (!dataset) {
+		const allowed = Object.keys(API_DATASETS).join(", ");
+		const error = new Error(
+			`Unknown dataset "${key}". Allowed datasets: ${allowed}`,
+		);
+		error.status = 400;
+		throw error;
+	}
+	return dataset;
+}
+
 function normalizeDatasetList(values) {
 	if (!values || values.length === 0) {
 		return Object.keys(DATASETS);
@@ -59,6 +123,15 @@ function normalizeDatasetList(values) {
 	return list.map((value) => assertDataset(String(value).trim()).key);
 }
 
+function normalizeApiDatasetList(values) {
+	if (!values || values.length === 0) {
+		return DEFAULT_API_DATASET_KEYS;
+	}
+
+	const list = Array.isArray(values) ? values : String(values).split(",");
+	return list.map((value) => assertApiDataset(String(value).trim()).key);
+}
+
 function normalizeLanguageList(values) {
 	if (!values || values.length === 0) {
 		return LANGUAGES;
@@ -82,9 +155,13 @@ function normalizeLanguageList(values) {
 }
 
 module.exports = {
+	API_DATASETS,
 	DATASETS,
+	DEFAULT_API_DATASET_KEYS,
 	LANGUAGES,
+	assertApiDataset,
 	assertDataset,
+	normalizeApiDatasetList,
 	normalizeDatasetList,
 	normalizeLanguageList,
 };

src/db/indexes.js

@@ -1,8 +1,17 @@
 const { DATASETS } = require("../datasets");
 
+function getIndexedCollections() {
+	return [
+		...Object.values(DATASETS),
+		...Object.values(DATASETS)
+			.map((dataset) => dataset.detail)
+			.filter(Boolean),
+	];
+}
+
 async function ensureIndexes(db) {
 	await Promise.all(
-		Object.values(DATASETS).map(async (dataset) => {
+		getIndexedCollections().map(async (dataset) => {
 			const collection = db.collection(dataset.collection);
 			await collection.createIndex(
 				{ language: 1, sourceId: 1 },

src/importer/importer.js

@@ -7,7 +7,7 @@ const {
 } = require("../datasets");
 const { connectToMongo } = require("../db/client");
 const { ensureIndexes } = require("../db/indexes");
-const { fetchQuestlogPage } = require("./questlogClient");
+const { fetchQuestlogDetail, fetchQuestlogPage } = require("./questlogClient");
 
 const importStatus = {
 	running: false,
@@ -18,6 +18,8 @@ const importStatus = {
 	totals: {},
 };
 
+const DETAIL_CONCURRENCY = 6;
+
 function stableJsonHash(value) {
 	return crypto.createHash("sha1").update(JSON.stringify(value)).digest("hex");
 }
@@ -88,6 +90,67 @@ async function upsertRecords(db, dataset, language, records) {
 	};
 }
 
+async function mapWithConcurrency(values, limit, iteratee) {
+	const results = new Array(values.length);
+	let nextIndex = 0;
+
+	async function worker() {
+		while (nextIndex < values.length) {
+			const currentIndex = nextIndex;
+			nextIndex += 1;
+			results[currentIndex] = await iteratee(
+				values[currentIndex],
+				currentIndex,
+			);
+		}
+	}
+
+	const workerCount = Math.min(limit, values.length);
+	await Promise.all(Array.from({ length: workerCount }, worker));
+	return results;
+}
+
+function extractDetailId(record, dataset) {
+	if (record?.id) {
+		return String(record.id);
+	}
+
+	if (record?.compoundId) {
+		return String(record.compoundId).replace(
+			new RegExp(`^${dataset.singular}-`),
+			"",
+		);
+	}
+
+	return undefined;
+}
+
+async function fetchDetailRecords(dataset, records, language) {
+	const detailDataset = dataset.detail;
+	return mapWithConcurrency(records, DETAIL_CONCURRENCY, async (record) => {
+		const id = extractDetailId(record, dataset);
+		if (!id) {
+			throw new Error(
+				`Could not determine Questlog ${dataset.singular} detail id for ${JSON.stringify(record)}`,
+			);
+		}
+
+		return fetchQuestlogDetail(detailDataset.method, id, language);
+	});
+}
+
+async function importDetails(db, dataset, language, page, records) {
+	const detailDataset = dataset.detail;
+	importStatus.current = {
+		dataset: detailDataset.key,
+		language,
+		page,
+		records: records.length,
+	};
+	const details = await fetchDetailRecords(dataset, records, language);
+	return upsertRecords(db, detailDataset, language, details);
+}
+
 function resetStatus() {
 	importStatus.running = true;
 	importStatus.startedAt = new Date().toISOString();
@@ -133,6 +196,22 @@ async function importDatasetLanguage(db, dataset, language, maxPages) {
 		);
 		recordTotals(dataset.key, language, pageResult, payload.records.length);
 
+		if (dataset.detail) {
+			const detailResult = await importDetails(
+				db,
+				dataset,
+				language,
+				page,
+				payload.records,
+			);
+			recordTotals(
+				dataset.detail.key,
+				language,
+				detailResult,
+				payload.records.length,
+			);
+		}
+
 		const reachedKnownEnd = payload.pageCount && page >= payload.pageCount;
 		const reachedConfiguredLimit = maxPages && page >= maxPages;
 		if (reachedKnownEnd || reachedConfiguredLimit) {

src/importer/questlogClient.js

@@ -11,6 +11,11 @@ function buildQuestlogUrl(method, language, page) {
 	return `${config.questlog.baseUrl}/${method}?input=${encodeURIComponent(input)}`;
 }
 
+function buildQuestlogDetailUrl(method, id, language) {
+	const input = JSON.stringify({ id, language });
+	return `${config.questlog.baseUrl}/${method}?input=${encodeURIComponent(input)}`;
+}
+
 function findFirstArray(value) {
 	if (Array.isArray(value)) {
 		return value;
@@ -46,8 +51,8 @@ function findFirstArray(value) {
 
 function extractPagePayload(payload) {
 	const data =
-		payload?.result?.data ||
 		payload?.result?.data?.json ||
+		payload?.result?.data ||
 		payload?.data ||
 		payload;
 	const records = findFirstArray(data);
@@ -69,6 +74,24 @@ function extractPagePayload(payload) {
 	return { records, pageCount, currentPage };
 }
 
+function extractDetailPayload(payload) {
+	const data =
+		payload?.result?.data?.json ||
+		payload?.result?.data ||
+		payload?.data ||
+		payload;
+
+	if (!data || typeof data !== "object" || Array.isArray(data)) {
+		const topLevelKeys =
+			payload && typeof payload === "object" ? Object.keys(payload) : [];
+		throw new Error(
+			`Could not find detail object in Questlog response. Top-level keys: ${topLevelKeys.join(", ")}`,
+		);
+	}
+
+	return data;
+}
+
 async function fetchQuestlogPage(dataset, language, page) {
 	const url = buildQuestlogUrl(dataset.method, language, page);
 	const response = await fetch(url, {
@@ -88,8 +111,30 @@ async function fetchQuestlogPage(dataset, language, page) {
 	return extractPagePayload(payload);
 }
 
+async function fetchQuestlogDetail(method, id, language) {
+	const url = buildQuestlogDetailUrl(method, id, language);
+	const response = await fetch(url, {
+		headers: {
+			accept: "application/json",
+			"user-agent": "dune-api-importer/1.0",
+		},
+	});
+
+	if (!response.ok) {
+		throw new Error(
+			`Questlog detail request failed for ${method}/${language}/${id}: ${response.status} ${response.statusText}`,
+		);
+	}
+
+	const payload = await response.json();
+	return extractDetailPayload(payload);
+}
+
 module.exports = {
+	buildQuestlogDetailUrl,
 	buildQuestlogUrl,
+	extractDetailPayload,
 	extractPagePayload,
+	fetchQuestlogDetail,
 	fetchQuestlogPage,
 };

src/routes/api.js

@@ -1,8 +1,9 @@
 const express = require("express");
 const { ObjectId } = require("mongodb");
 const {
-	assertDataset,
+	API_DATASETS,
-	DATASETS,
+	assertApiDataset,
+	normalizeApiDatasetList,
 	normalizeDatasetList,
 	normalizeLanguageList,
 } = require("../datasets");
@@ -57,8 +58,24 @@ function buildSearchFilter(query) {
 	return { $text: { $search: String(query) } };
 }
 
+async function findDatasetRecord(db, dataset, filter) {
+	const datasets = dataset.detail ? [dataset.detail, dataset] : [dataset];
+
+	for (const candidateDataset of datasets) {
+		const document = await db
+			.collection(candidateDataset.collection)
+			.findOne(filter, { projection: { searchText: 0 } });
+
+		if (document) {
+			return document;
+		}
+	}
+
+	return undefined;
+}
+
 router.get("/datasets", (request, response) => {
-	response.json({ datasets: Object.values(DATASETS) });
+	response.json({ datasets: Object.values(API_DATASETS) });
 });
 
 router.get("/import/status", (request, response) => {
@@ -92,14 +109,14 @@ router.get("/search", async (request, response, next) => {
 			throw error;
 		}
 
-		const datasetKeys = normalizeDatasetList(request.query.datasets);
+		const datasetKeys = normalizeApiDatasetList(request.query.datasets);
 		const languageFilter = buildLanguageFilter(request.query.language);
 		const limit = parseLimit(request.query.limit, 10, 50);
 		const db = getDb();
 
 		const results = {};
 		for (const datasetKey of datasetKeys) {
-			const dataset = DATASETS[datasetKey];
+			const dataset = API_DATASETS[datasetKey];
 			results[datasetKey] = await db
 				.collection(dataset.collection)
 				.find({ ...languageFilter, ...buildSearchFilter(query) })
@@ -116,7 +133,7 @@ router.get("/search", async (request, response, next) => {
 
 router.get("/:dataset", async (request, response, next) => {
 	try {
-		const dataset = assertDataset(request.params.dataset);
+		const dataset = assertApiDataset(request.params.dataset);
 		const page = parsePage(request.query.page);
 		const limit = parseLimit(request.query.limit, 25, 100);
 		const skip = (page - 1) * limit;
@@ -143,18 +160,16 @@ router.get("/:dataset", async (request, response, next) => {
 
 router.get("/:dataset/:id", async (request, response, next) => {
 	try {
-		const dataset = assertDataset(request.params.dataset);
+		const dataset = assertApiDataset(request.params.dataset);
 		const id = request.params.id;
 		const languageFilter = buildLanguageFilter(request.query.language);
 		const idFilter = ObjectId.isValid(id)
 			? { $or: [{ _id: new ObjectId(id) }, { sourceId: id }] }
 			: { sourceId: id };
-		const document = await getDb()
+		const document = await findDatasetRecord(getDb(), dataset, {
-			.collection(dataset.collection)
+			...languageFilter,
-			.findOne(
+			...idFilter,
-				{ ...languageFilter, ...idFilter },
+		});
-				{ projection: { searchText: 0 } },
-			);
 
 		if (!document) {
 			response.status(404).json({ error: "Not found" });

src/server.js

@@ -10,7 +10,8 @@ async function start() {
 	const app = createApp();
 	const server = app.listen(config.port, () => {
 		console.log(`Dune API listening on http://localhost:${config.port}`);
-		console.log(`Swagger UI available at http://localhost:${config.port}/docs`);
+		console.log(`Public API URL: ${config.public.apiUrl}`);
+		console.log(`Public Swagger UI URL: ${config.public.uiUrl}/docs`);
 	});
 
 	async function shutdown(signal) {

src/swagger/openapi.js

@@ -1,15 +1,125 @@
-const { DATASETS, LANGUAGES } = require("../datasets");
+const {
+	API_DATASETS,
+	DATASETS,
+	DEFAULT_API_DATASET_KEYS,
+	LANGUAGES,
+} = require("../datasets");
+const { config } = require("../config");
 
-const datasetKeys = Object.keys(DATASETS);
+const apiDatasetKeys = Object.keys(API_DATASETS);
+const importDatasetKeys = Object.keys(DATASETS);
+const detailDatasetExamples = {
+	item: {
+		id: "LongRifle_Unique_Poison_03",
+		summary: "List detailed item records",
+		description:
+			"Returns detailed Questlog item records from the singular item collection. These records include raw item payloads with stats such as weaponStats, fillableStats, wearableStats, and other item-specific structures.",
+	},
+	skill: {
+		id: "skills_ability_poisonmine",
+		summary: "List detailed skill records",
+		description:
+			"Returns detailed Questlog skill records from the singular skill collection, including levels and skill tree connections.",
+	},
+	recipe: {
+		id: "Bloodsack_2_Recipe",
+		summary: "List detailed recipe records",
+		description:
+			"Returns detailed Questlog recipe records from the singular recipe collection, including inputs, outputs, crafting requirements, and crafting stations.",
+	},
+	placeable: {
+		id: "Atre_Banner_Placeable",
+		summary: "List detailed placeable records",
+		description:
+			"Returns detailed Questlog placeable records from the singular placeable collection, including production, power, water, and crafting relationships.",
+	},
+	npc: {
+		id: "bs43q",
+		summary: "List detailed NPC records",
+		description:
+			"Returns detailed Questlog NPC records from the singular npc collection, including descriptions and NPC tags when available.",
+	},
+};
+
+function createListParameters() {
+	return [
+		{
+			name: "language",
+			in: "query",
+			schema: { type: "string", enum: LANGUAGES },
+		},
+		{ name: "q", in: "query", schema: { type: "string" } },
+		{
+			name: "page",
+			in: "query",
+			schema: { type: "integer", minimum: 1, default: 1 },
+		},
+		{
+			name: "limit",
+			in: "query",
+			schema: { type: "integer", minimum: 1, maximum: 100, default: 25 },
+		},
+	];
+}
+
+function buildDetailDatasetPaths() {
+	return Object.fromEntries(
+		Object.entries(detailDatasetExamples).flatMap(([datasetKey, example]) => [
+			[
+				`/api/${datasetKey}`,
+				{
+					get: {
+						tags: ["Data"],
+						summary: example.summary,
+						description: example.description,
+						parameters: createListParameters(),
+						responses: {
+							200: { description: `Paged detailed ${datasetKey} records` },
+						},
+					},
+				},
+			],
+			[
+				`/api/${datasetKey}/{id}`,
+				{
+					get: {
+						tags: ["Data"],
+						summary: `Get one detailed ${datasetKey} record`,
+						description: `Get a single detailed ${datasetKey} by MongoDB id or Questlog source id, for example ${example.id}.`,
+						parameters: [
+							{
+								name: "id",
+								in: "path",
+								required: true,
+								schema: { type: "string" },
+								example: example.id,
+							},
+							{
+								name: "language",
+								in: "query",
+								schema: { type: "string", enum: LANGUAGES },
+							},
+						],
+						responses: {
+							200: { description: `Detailed ${datasetKey} record` },
+							404: { description: "Record was not found" },
+						},
+					},
+				},
+			],
+		]),
+	);
+}
 
 const openApiDocument = {
 	openapi: "3.0.3",
 	info: {
 		title: "Dune Awakening API",
 		version: "1.0.0",
-		description: "API for Dune: Awakening Questlog data stored in MongoDB.",
+		description:
+			"API for Dune: Awakening Questlog data stored in MongoDB. Use singular datasets like /api/item, /api/skill, /api/recipe, /api/placeable, and /api/npc for detailed records.",
 	},
-	servers: [{ url: "/" }],
+	servers: [{ url: config.public.apiUrl }],
 	tags: [{ name: "Health" }, { name: "Data" }, { name: "Import" }],
 	paths: {
 		"/health": {
@@ -25,20 +135,23 @@ const openApiDocument = {
 		"/api/datasets": {
 			get: {
 				tags: ["Data"],
-				summary: "List supported datasets",
+				summary: "List supported public API datasets",
 				responses: { 200: { description: "Supported datasets" } },
 			},
 		},
+		...buildDetailDatasetPaths(),
 		"/api/{dataset}": {
 			get: {
 				tags: ["Data"],
 				summary: "List records for a dataset",
+				description:
+					"Generic dataset listing. Prefer singular datasets like /api/item, /api/skill, /api/recipe, /api/placeable, and /api/npc for detailed data used by OpenClaw; plural datasets remain available for older summary records.",
 				parameters: [
 					{
 						name: "dataset",
 						in: "path",
 						required: true,
-						schema: { type: "string", enum: datasetKeys },
+						schema: { type: "string", enum: apiDatasetKeys },
 					},
 					{
 						name: "language",
@@ -64,12 +177,14 @@ const openApiDocument = {
 			get: {
 				tags: ["Data"],
 				summary: "Get one record by MongoDB id or Questlog source id",
+				description:
+					"Generic dataset lookup. When a plural dataset has a singular detail collection, this returns the detailed singular record when available, then falls back to the older summary record.",
 				parameters: [
 					{
 						name: "dataset",
 						in: "path",
 						required: true,
-						schema: { type: "string", enum: datasetKeys },
+						schema: { type: "string", enum: apiDatasetKeys },
 					},
 					{
 						name: "id",
@@ -92,7 +207,8 @@ const openApiDocument = {
 		"/api/search": {
 			get: {
 				tags: ["Data"],
-				summary: "Search across datasets",
+				summary: "Search across public API datasets",
+				description: `Searches detailed singular records by default. Default datasets: ${DEFAULT_API_DATASET_KEYS.join(", ")}. Pass a plural dataset like datasets=items only if you want older summary records.`,
 				parameters: [
 					{
 						name: "q",
@@ -108,7 +224,7 @@ const openApiDocument = {
 					{
 						name: "datasets",
 						in: "query",
-						schema: { type: "string", example: "items,skills" },
+						schema: { type: "string", example: "item,skill,recipe" },
 					},
 					{
 						name: "limit",
@@ -134,7 +250,7 @@ const openApiDocument = {
 								properties: {
 									datasets: {
 										type: "array",
-										items: { type: "string", enum: datasetKeys },
+										items: { type: "string", enum: importDatasetKeys },
 									},
 									languages: {
 										type: "array",