malloydata/publisher/malloy-document

Documenting a Malloy Model

Add #(doc) tags to describe sources and fields in plain language so they are easy to find and understand:

#(doc) is a standard Malloy annotation. It documents a field or source with a human-readable description that downstream tools can surface and search against.

#(doc) Tag

Add before any source, dimension, measure, view, or join. When multiple fields share a keyword, use it once as a block header. Tags and field names are indented under the keyword; tags go on the line(s) directly above the field they annotate.

Tag ordering (when a field has multiple tags): #(doc) → render tags (# currency, # label, etc.) → field name. Separate each field group with a blank line:

#(doc) Customer who placed the order
join_one: users with user_id
 
dimension:
  #(doc) Date the order was placed (UTC)
  order_date is created_at::date
 
measure:
  #(doc) Total revenue from all orders in USD
  # currency
  revenue is sum(total)

Writing Doc Strings for Retrieval

Doc strings power natural-language search: users type plain-English questions and the system matches against your #(doc) strings. Write descriptions that match how analysts would search:

• Include business meaning, not code mechanics: what it represents, not how it's implemented
• Include units (USD, count, percentage): a unit is part of what a number means. For counts, name the unit being counted and say whether it counts distinct entities or events: "total students enrolled" on a subject×term-grain measure counts enrolments, not students, and a student taking four subjects counts four times. If the model cannot answer the distinct-entity version, say so in the doc.
• List a categorical field's values only while the list stays short (roughly ten or fewer). A handful of values makes a description concrete; past that, say what the field captures instead, because the dump crowds out the meaning and goes stale the moment someone adds a value. Treat ten as a rule of thumb, not a hard cap.
• Avoid Malloy jargon: never use "filterable", "groupable", "dimension", "measure", "aggregation"

Good examples:

• #(doc) Total revenue from completed orders in USD matches "what was our revenue?"
• #(doc) Customer signup date (UTC) matches "when did the customer join?"
• #(doc) Order status: pending, processing, shipped, delivered, cancelled matches "what are the order statuses?"

Bad examples:

• #(doc) Filterable dimension for order status: no analyst searches for "filterable"
• #(doc) Groupable by region: "groupable" is a system concept
• #(doc) Aggregation of total sales: "aggregation" doesn't match natural queries

Mark conventions as conventions

A #(doc) must let a reader tell a measured fact from a choice someone made. Any dimension or measure encoding a threshold, bucket boundary, or business definition that the user did not explicitly confirm must say so in its own doc string:

// WRONG - a chosen cutoff stated as fact
#(doc) Popularity band: Hit (70+), Popular (40-69), Moderate (15-39), Obscure (<15)
 
// RIGHT - the choice is visible and auditable
#(doc) Popularity band: Hit (70+), Popular (40-69), Moderate (15-39), Obscure (<15).
#(doc) 70 follows the source dataset's own high-popularity cutoff; 40 and 15 are
#(doc) working boundaries for this model, not settled by the data.

These are governed models: a threshold nobody confirmed is an assumption, and an unlabeled assumption reads as a fact to everyone downstream, including the agents that answer questions from these docs. The hedge in the #(doc) is the artifact-time record; the "Flag Ambiguous Descriptions" table below is the conversation-time surface for getting them confirmed, and modeling-notes.md's "Open decisions" section (see skill:malloy-modeling) is where they wait for a subject-matter expert.

Do not hedge measured facts: avg_energy is avg(energy) needs no caveat. Hedge only where a domain expert could reasonably choose differently.

#(filter): deprecated, see malloy-model

#(filter) is deprecated. Never add a #(filter) annotation: every use, including required, implicit, and date/number ranges, has a given: form. See skill:malloy-model § Parameterizing sources with given:.

#(filter) is also a #(...)-shaped annotation, but unlike #(doc) it's a runtime/modeling construct: it shapes governance, query latency, and correctness, not discoverability. How to read and migrate an existing one lives in malloy-model § Legacy: reading an existing #(filter) model.

One rule worth knowing here: a filter the model reads lives on the source, never on the consumer. Ad-hoc reports and notebooks that import a source inherit its givens automatically, and import them by name rather than re-declaring them. A notebook or dashboard may declare a given: of its own for a control only its own tiles read, and nothing else.

internal: and private:: column-level access in a source

#(doc) describes what's exposed. Two access modifiers control what's exposed in the first place, and both live inside a source's include {} block. They are about the source's public API and data sensitivity, not about documentation, so reach for them when curating which columns callers can pick.

In one sentence: `internal:` and `private:` shape what's inside a source's public API; `#(doc)` describes the fields you do expose.

#(classification): what a field _is_, in the policy's words

When a package has an access or data-handling policy, every published dimension and measure should say which class of data it carries, so an agent sees the class before reading the field and a review can check each surface against the policy. One annotation on the field, after the closing parenthesis: one quoted class, or a quoted, comma-separated list when the field sits under several regimes:

#(doc) Social security number.
#(classification) "phi.direct_identifier"
dimension: ssn is SSN
 
#(classification) "none"
measure: order_count is count()
 
#(classification) ["phi.direct_identifier", "gdpr.personal"]
dimension: member_ssn is SSN

• The vocabulary is the policy's, not Malloy's: use the class names the policy document uses (phi.direct_identifier, confidential, pci.pan), and none for a field the policy has nothing to say about. Publisher checks only the shape: quoted, and a list in ["a", "b"] form like every other list-valued annotation. One #(classification) per field; an unquoted or space-delimited value warns classification-malformed.
• Write the value after the route. #(classification "phi.clinical") compiles with a malformed-route warning and declares nothing.
• Classify a field where it is declared; the classes ride every join to it, survive extend and rename:, and a field derived from it (dimension: x is y, concat(y, …), count(y)) inherits them, as the union over everything it reads. Declare classes on a derived field only to override the inherited set (a year derived from a birth date is a quasi-identifier, not a direct one).
• #(carries) ["<class>", …] on a source line lists every class a caller may reach from that source. A field with any class off the list, or no class, is a leak, and a leak refuses the package load, so write it on each entry point next to its #(authorize) and classify everything it exposes. It is also checked per query: a caller query that reads a field with no class or a class off the list (through extend, a join it writes, or a source it declares) is refused with a 400, and a caller cannot write #(classification) or #(carries) in a query.
• GET …/packages/{pkg}/classification returns each published entry point's #(carries) and its field count by class, for a scorecard or a CI check.
• Once any field in a package is classified, Publisher warns (classification-coverage) about every published field that is not, so classify a package all at once rather than one source at a time. A field under private: is never reachable, and an internal: field is not demanded one at load; but under #(carries) a caller query that reads an internal: field through extend is refused unless its class is on the list, so classify any internal: field a caller could read.
• It changes nothing about what a query returns. Pair it with private: to hide a column and with #(authorize) / #(access_filter) (see skill:malloy-model) to control who reads a source.

Example

A base source pulled from a messy raw table often uses internal: to drop raw fields from the public API, while documenting the curated columns with #(doc).

// orders_base.malloy
#(doc) Raw orders. Use orders.malloy as the entry point for analysis.
source: orders_base is conn.table('orders_raw')
  include {
    public: id, customer_id, order_date, total
    internal: raw_json_payload, deprecated_status_code, _temp_dedup_marker
  }
  extend {
    primary_key: id
  }

// orders.malloy
import "orders_base.malloy"