================================================================================
SitRepMapper v1.18.0 — Data File Reference

Designed and developed by Stephen Clay McGehee, KN4AM
Coding done using Claude Code AI
See SitRepNet.com for more information
================================================================================


PURPOSE AND THE READ-ONLY RULE
------------------------------
A companion program is a separate tool that adds a feature by reading
SitRepMapper's data, with no change to SitRepMapper's code. This document
lists the data a companion can use and the rules that keep it safe.

THE ONE RULE: companions read SitRepMapper's live data files; they never
write to them. SitRepMapper loads these files at startup and rewrites them
whenever its data changes. A file edited by another program while
SitRepMapper runs will be overwritten, or will confuse its merge logic.

To get data INTO SitRepMapper, a companion writes an import file in a
folder of its own. The user then loads it from SitRepMapper's normal
Import menu (see SENDING DATA BACK below).

To show something on SitRepMapper's screen, a companion writes a small
display file in the companions folder (see SHOWING THINGS ON SCREEN).


RULES FOR COMPANION AUTHORS
---------------------------
If you build a companion with an AI coding assistant, give it this whole
file. These rules come first; the sections below give the details.

  1. Don't change SitRepMapper. A companion works from outside it. If
     something seems to need a change to SitRepMapper, stop and ask the
     user instead of making it.
  2. Read the live files; never write to them. Data goes back to
     SitRepMapper only through import files that the user loads (see
     SENDING DATA BACK). In the config folder, the only files a
     companion writes are its own display files in companions/ (see
     SHOWING THINGS ON SCREEN).
  3. Use the SAFE READING RECIPE: retry if the JSON won't parse, check
     "format" and "version", and re-read a file only when its modified
     time or size changes.
  4. Don't transmit unless the user has explicitly chosen that. Sending
     from two programs at once can collide with SitRepMapper's Send and
     Expect features.
  5. Open the JS8Spotter database read-only (mode=ro).
  6. Leave expect.expect.json and send_drafts.json alone. They are not
     part of the companion contract.
  7. Ignore fields you don't recognize, and give every config.json key
     a default.
  8. SitRepMapper's Python helpers are a convenience, not a promise.
     Recheck them after each SitRepMapper update.
  9. Keep the load light. Many users run on a Raspberry Pi shared with
     SitRepMapper, JS8Call and JS8Spotter.
 10. Test against a real ~/.config/sitrepmapper folder, not just
     made-up samples.
 11. If the companion acts on what stations send, give it callsign
     lists (see STATIONS A COMPANION RESPONDS TO).
 12. Ship an info sheet with it (see COMPANION INFO SHEET).


WHERE THE FILES LIVE
--------------------
All live data sits in one folder, ~/.config/sitrepmapper/

  Linux / Raspberry Pi OS:  /home/<user>/.config/sitrepmapper/
  Windows:                  C:\Users\<user>\.config\sitrepmapper\

In Python, os.path.expanduser('~/.config/sitrepmapper') gives the right
path on both.

  File                   Holds                                Companion use
  ---------------------  -----------------------------------  -------------
  reports.reports.json   Report Library: every stored F!/     Read
                         CommStat report, plus deletion
                         records
  flags.flags.json       Flag pins, plus deletion records     Read
  areas.areas.json       Drawn areas, plus deletion records   Read
  config.json            User settings: data source paths,    Read
                         station call and grid, filters
  statusboard.           Status Board summary: latest         Read
    statusboard.json     status per station, sub-optimal
                         flags, totals
  expect.expect.json     Expect auto-reply entries            Leave alone
  send_drafts.json       Saved Send-form drafts               Leave alone
  companions/            Display files that companions        Write your
                         write                                own only
  companions.log         Problems with display files          Read

The exports/ folder under the program directory is the default
destination for exports. A companion may read files there too, but the
user decides what goes in it.


REPORTS.REPORTS.JSON
--------------------
This file is the Report Library: reports the user has imported. IT DOES
NOT HOLD LIVE ON-AIR TRAFFIC. SitRepMapper reads live reports straight
from the JS8Spotter database or JS8Call's DIRECTED.TXT each time and
never copies them here. A companion that needs live traffic should read
those sources the same way (see OTHER DATA SOURCES below). A companion
that only needs the overall picture can read STATUSBOARD.STATUSBOARD.JSON
instead.

Top level:

  {
    "format": "sitrepmapper_reports",
    "version": "1.0",
    "exported_at": "2026-08-28T02:47:59Z",
    "reports": [ ... ],
    "deleted": { "<sha256 identity hash>": "2026-08-13T16:06:44Z" }
  }

Each entry in "reports":

  {
    "id": "6a0a2db3-b9e6-49b6-82c6-f8e68cda0004",
    "form_type": "f505",
    "raw_text": "@SITREP MSG F!505 33333333333333333333333333333333 ST[OK] GR[EM15] CT[CANADIAN] #HFV0",
    "callsign": "KJ5MIW",
    "timestamp_str": "2026-08-06 21:52:00",
    "source": "directed_log",
    "stored_at": "2026-08-13T15:31:28Z"
  }

  id             UUID assigned when stored
  form_type      f104, f309, f500, f505 or commstat
  raw_text       The message exactly as sent on air. All decoded values
                 come from this.
  callsign       Sending station, upper case
  timestamp_str  Receive time, YYYY-MM-DD HH:MM:SS, UTC WITH NO ZONE
                 SUFFIX. May be empty for some F!104/309/505 reports with
                 no datecode.
  source         Where it came from: directed_log, js8spotter_db, file, ...
  stored_at      When it entered the Library, ISO 8601 UTC

"deleted" maps a content hash to its deletion time. It only prevents a
deleted report from coming back on re-import; a companion can ignore it,
since deleted reports are already removed from "reports".

Decoding raw_text: don't write a second parser. A Python companion can
call the existing ones (see SAFE READING RECIPE below):
report_store.report_from_envelope(entry) returns the decoded report
object. Other languages need their own parser (see DECODING F! FORMS
below).


FLAGS.FLAGS.JSON
----------------
This file holds every flag pin on the map. It uses the same format as a
.flags.json export, so a companion that can read one can read both.

  {
    "format": "sitrep_flags",
    "version": "1.0",
    "exported_at": "2026-08-14T13:56:02Z",
    "flags": [
      {
        "id": "a064bad5-c58a-472e-95f1-555a2f44f75f",
        "description": "GMAG Member LJ",
        "grid": "EL99HC84MC",
        "lat": 29.100495416699804,
        "lon": -81.34573866097013,
        "datetime": "2026-07-17T18:35:20Z",
        "source": "GMAG",
        "color": "#00FF00",
        "blink": false,
        "severity": "",
        "status": "",
        "group": "GMAG Member",
        "updated_at": "2026-07-23T02:04:46Z"
      }
    ],
    "deleted": [
      { "id": "2179c825-84f8-4b62-9762-d88ac6c2ad64",
        "deleted_at": "2026-07-21T12:49:05Z" }
    ]
  }

  id                UUID; the key used to merge on import
  description       Label shown on the pin
  grid              Maidenhead locator, 4 to 10 characters
  lat, lon          Decimal degrees. The map places the pin by these, not
                    by grid.
  datetime          Event time the user entered, ISO 8601 UTC
  source            Who placed it; defaults to the station callsign
  color             Hex color, #RRGGBB
  blink             Whether the pin flashes
  severity, status  Optional free-text tags, often empty
  group             Group name for View > Flag Pin Groups; empty means
                    no group
  updated_at        Last change, ISO 8601 UTC. On import, the newer copy
                    wins.

"deleted" lists pins the user removed. SitRepMapper already removes those
pins from "flags", so a companion only needs to read "flags".


AREAS.AREAS.JSON
----------------
This file holds every drawn area, in the same layout as the flags file:
format "sitrep_areas", an "areas" list and a "deleted" list of
{id, deleted_at}.

  {
    "id": "5803884c-67b9-4bbe-9a77-9102d6995f19",
    "type": "circle",
    "points": [[28.65111315468583, -80.65114287567138],
               [28.6505559645154, -80.64146165628706], ...],
    "description": "Zone 4 - Moderate Blast Damage (5 psi)",
    "color": "#FF0000",
    "line_width": 2,
    "show_title": false,
    "title_size": 14,
    "show_center": true,
    "group": "Nuke Damage Zone",
    "updated_at": "2026-08-12T00:40:34Z"
  }

  type                   polygon, rectangle or circle
  points                 Outline as [lat, lon] pairs in decimal degrees.
                         Note the order: latitude first.
  description            The area's title
  color, line_width      Outline color #RRGGBB and width in pixels
  show_title, title_size Whether the title is drawn on the map, and its
                         font size
  show_center            Whether a center marker is drawn (on by default
                         for circles)
  group                  Group name for View > Area Groups
  id, updated_at         Same meaning as for flags: merge key and
                         newest-wins timestamp

CIRCLES HAVE NO CENTER OR RADIUS FIELD. They are stored as 48 outline
points. The center is the average of the points, and the radius is the
average distance from it. area_manager.centroid_latlon() and
area_manager.circle_radius_m() compute both, and circle_points() builds
the 48 points for a new circle.


CONFIG.JSON
-----------
This file is a flat JSON object of user settings. Only keys the user has
changed are guaranteed to be there, so a companion must supply its own
default for every key it reads. The keys most useful to a companion:

  report_source           Live source in use: js8spotter_db or
                          directed_log (default js8spotter_db)
  last_db_path            Path to the JS8Spotter database
  last_directed_log_path  Path to JS8Call's DIRECTED.TXT
  station_callsign        The user's callsign
  station_grid            The user's home grid
  js8call_host,           JS8Call API address (default 127.0.0.1, 2442)
  js8call_port
  duplicate_handling      F!505 repeat reports: latest or average
                          (default latest; see WHAT THE MAP SHOWS)
  rounding                How averaged F!505 scores round: round, ceil
                          or floor (default round)
  filter_mode,            The map's Date Filter, so a companion can match
  filter_n_days,          what the map shows (defaults none, 14, empty,
  filter_from,            empty)
  filter_to
  hidden_flag_groups,     Group names hidden on the map (default [])
  hidden_area_groups
  export_dir              Save to Export Folder destination; empty means
                          <program folder>/exports

The full list, with comments, is DEFAULTS in config.py. Read
report_source and the two path keys to find the same live data
SitRepMapper is using.


STATUSBOARD.STATUSBOARD.JSON
----------------------------
The Status Board's processed picture: each station's latest status for
every report type, which stations are flagged sub-optimal and why, and
totals. Use it when a companion needs to know what's going on overall,
rather than what each report said. Available in SitRepMapper v1.18.0
and later.

It holds the same results that Reports > Status Board > Station Status
Roster and Sub-Optimal Conditions Digest show for the same number of
days, so a companion and the screen can't disagree.

Which reports it covers:

  - Steps 1 to 5 of WHAT THE MAP SHOWS: live reports plus the Report
    Library, minus IGNORE.TXT, repeat reports and reports with no grid.
  - NOT the map's Date Filter or Time Lapse. Instead, the Status
    Board's saved "Days to include" (bigpicture_n_days in config.json),
    always ending today in UTC. An end date chosen in the Status Board
    window is ignored here.
  - Flagged means flagged by Settings > Sub-Optimal Thresholds. The
    values in use are copied into the file. An F!309 that is Resolved
    or past its own Expiration is never flagged (see "ended" below).

When it's written:

  - Within about 5 seconds of any change to the reports or settings,
    and rebuilt at least once a minute, so the window moves forward at
    0000Z even with no new traffic.
  - Only when its contents change, AND at least every 10 minutes even
    when nothing has changed. So if "generated_at" (or the file's
    modified time) is more than about 12 minutes old, SitRepMapper
    isn't running. A quiet net still gets a fresh file.
  - The file doesn't exist until SitRepMapper v1.18.0 or later has run
    once. It stays in place when SitRepMapper closes.

Top level:

  {
    "format": "sitrepmapper_statusboard",
    "version": "1.0",
    "generated_at": "2026-11-09T18:42:10Z",
    "window": {
      "days": 7,
      "from": "2026-11-03",
      "to": "2026-11-09",
      "date_source": "received_time"
    },
    "thresholds": {
      "f104_degraded": true,
      "f104_problems": true,
      "f309_min_severity": "2",
      "f500_supply_tier": "moderate",
      "f500_mood_tier": "moderate",
      "f505_avg_below": 2.0,
      "commstat_yellow": true,
      "commstat_red": true
    },
    "totals": {
      "stations": 23,
      "flagged": 4,
      "reports_by_type": {"f104": 40, "f309": 3, "f505": 9, "f500": 6,
                          "commstat": 12},
      "flagged_by_type": {"f104": 2, "f505": 1, "commstat": 1}
    },
    "stations": [ ... ]
  }

  generated_at     When SitRepMapper wrote the file, ISO 8601 UTC
  window           Which reports were counted: "days" ending on "to"
                   (both dates UTC, YYYY-MM-DD), and whether a report's
                   date is its receive time or its prepared time
                   (date_source: received_time or timestamp)
  thresholds       The Sub-Optimal Thresholds in use (config.json keys
                   bigpicture_thresh_*, without that prefix)
  totals           stations: how many stations reported in the window
                   flagged: how many have at least one flagged type
                   reports_by_type: reports counted, every type listed
                   flagged_by_type: stations flagged for each type; a
                   type with none is left out

Each entry in "stations":

  {
    "callsign": "KJ5MIW",
    "grid": "EM15",
    "latest": "2026-11-09T17:58:00Z",
    "flagged_count": 2,
    "types": {
      "f104": {
        "time": "2026-11-09T17:58:00Z",
        "grid": "EM15",
        "status": "Problems",
        "flagged": true,
        "reason": "Problems",
        "code": "3"
      },
      "f505": {
        "time": "2026-11-07T19:30:00Z",
        "grid": "EM15",
        "status": "2.40 Degraded but Functional",
        "flagged": false,
        "reason": "",
        "score": 2.4
      },
      "commstat": {
        "time": "2026-11-09T02:10:00Z",
        "grid": "EM15",
        "status": "Red",
        "flagged": true,
        "reason": "Red",
        "code": "3"
      }
    }
  }

  callsign       Upper case. Empty ("") when the reports carried no
                 callsign and none could be matched by grid.
  grid           The station's grid square: the first grid found in
                 its reports (F!104 first, then F!309, F!505, F!500,
                 CommStat). It doesn't follow later reports, so for
                 where one report says it is, use that report's own
                 "grid" below. This matters most for F!309, whose grid
                 is the incident's location, not the station's.
  latest         Time of its most recent report of any type, ISO 8601
                 UTC, or null if none of its reports has a time
  flagged_count  How many of its report types are flagged (0 to 5)
  types          One entry for each type the station sent in the
                 window, holding only its LATEST report of that type.
                 Types it didn't send are left out.
    time         That report's time, ISO 8601 UTC, or null
    grid         That report's own grid, upper case, "" if none
    status       The Station Status Roster's text for that report
    flagged      true if it crosses the Sub-Optimal Thresholds
    reason       The Digest's explanation when flagged, else ""
    (values)     The type's own machine-readable value(s), below

"status" and "reason" are for showing to people; their wording may be
improved in later versions. Make decisions from "flagged" and the
values below, which won't change meaning within version 1.x.

Values for each type, each on that type's own scale:

  f104      code            "1" Normal, "2" Degraded, "3" Problems,
                            "" if missing
  commstat  code            "1" Green, "2" Yellow, "3" Red,
                            "4" Unknown, "" if the status couldn't be
                            read
  f309      severity        "1" Minor, "2" Moderate, "3" Severe,
                            "4" Major, "5" Unknown
            category        The form's category code ("1"-"9", "A"-"R");
                            "status" gives its name
            time_frame      "1" General/unknown, "2" Anticipated (week/
                            month), "3" Imminent (hours to days),
                            "4" Currently happening, "5" Chronic, ""
            situation       The form's Current Status: "1" New,
                            "2" Ongoing, Deteriorating, "3" Ongoing,
                            Improving, "4" Resolved, "5" Unknown, ""
            expires         When the report's own Expiration runs out
                            (from its datecode, else when received),
                            ISO 8601 UTC, or null if it doesn't expire
            ended           "Resolved" (situation "4") or "Expired"
                            (past "expires"), else "". An ended F!309
                            is never flagged, whatever its severity,
                            and its "status" ends " (Resolved)" or
                            " (Expired)". "ended" can turn to "Expired"
                            with no new report, when the time passes.
  f505      score           Average of the known scores, 0.00 (worst)
                            to 3.00 (fully operational), or null if
                            every score was unknown
  f500      worst_question  Key of the most severe answered question:
                            mood, eggs, milk, bread, water, gasoline,
                            diesel, propane, batteries, containers,
                            ammo, materials. null if none was answered
                            (an "Unknown" answer doesn't count). A tie
                            goes to the first in that list.
            worst_severity  That answer from 0.0 (normal) to 1.0 (the
                            question's worst answer), or null

There is deliberately no single severity number shared across types:
the five forms don't measure the same things on the same scale. Use
"flagged" and "flagged_count" to compare stations across types.

"stations" is sorted worst first: by flagged_count, highest first, then
by callsign (or grid, when there's no callsign). To list the stations
that need attention, take entries until flagged_count is 0.

This file has only each station's latest report of each type. For
history or individual reports, read the reports themselves (see
REPORTS.REPORTS.JSON and OTHER DATA SOURCES).


OTHER DATA SOURCES
------------------
Live reports don't come from SitRepMapper's files. They come from the
same upstream sources SitRepMapper reads, and a companion can read those
sources directly.

JS8Spotter database (path in last_db_path). This is SQLite. Always open
it read-only, the way SitRepMapper does, so it is safe while JS8Spotter
is running:

  conn = sqlite3.connect(f'file:{db_path}?mode=ro', uri=True)

  Table      Holds                         Columns SitRepMapper uses
  ---------  ----------------------------  ------------------------------
  forms      F!104, F!309, F!500, F!505    id, typeid, fromcall, msgtxt,
             messages                      responses, timesig, lm
  csstatrep  CommStat StatReps             id, cssr_from, cssr_group,
                                           cssr_grid, cssr_prio,
                                           cssr_msgid, cssr_status,
                                           cssr_notes, cssr_timestamp

To poll for new rows only, remember the highest id seen and query
WHERE id > ?. js8spotter_db.JS8SpotterDB already does this and returns
decoded reports.

JS8Call DIRECTED.TXT (path in last_directed_log_path). This is a
tab-separated text log that JS8Call appends to: time (UTC), dial
frequency, offset, SNR, then CALLSIGN: message. Read new lines from the
last file position; never write to it. directed_log.parse_line() and
route_entry() turn a line into a report.

JS8Call TCP API (js8call_host:js8call_port). This is JSON over TCP, for
live events and for sending. A companion should only listen unless the
user has explicitly chosen to let it transmit. Sending from two programs
at once can collide with SitRepMapper's Send and Expect features, which
check whether JS8Call is busy before they transmit.


WHAT THE MAP SHOWS
------------------
No file holds the reports exactly as the map shows them. SitRepMapper
builds that view each time, in this order (statusboard.statusboard.json
follows steps 1 to 5 but uses its own window instead of step 6):

  1. Live reports from the source named in report_source.
  2. Plus every report in the Report Library (reports.reports.json).
  3. Minus callsigns listed in IGNORE.TXT, in SitRepMapper's program
     folder (not the config folder). One callsign per line, any case;
     lines starting with '#' are comments. No file means nobody is
     ignored. It applies to all five report types from every source.
  4. Minus repeat reports: several reports from the same callsign in
     the same grid square become one.
       F!505            follows duplicate_handling: "latest" keeps the
                        most recent; "average" makes one report with
                        averaged scores, rounded as set by rounding
       F!104, F!500,    always keep the most recent
       CommStat
       F!309            never merged (each is a separate incident)
     "Most recent" means received time, or prepared time if there is
     none. Different callsigns in the same grid are never merged.
  5. Minus reports with no grid square (they can't be placed).
  6. Trimmed by the Date Filter (filter_mode and the other filter_
     keys). A report with no date passes the filter.

A companion doesn't have to match the map. A trend companion may want
every report, repeats included. But one that claims to show "what the
map shows" should follow these steps.


DECODING F! FORMS
-----------------
Python companions should use SitRepMapper's own parsers (see the
REPORTS.REPORTS.JSON). Companions in other languages need two things:

  What each answer means: the form spec files that ship with
  JS8Spotter, in <JS8Spotter folder>/forms/ (MCF104.txt, MCF309.txt,
  MCF500.txt, MCF505.txt), plus its "MCForms Documentation" folder.

  How the answers are laid out in the message (below).

Each form is one line. Tags in [brackets] other than the form number
and answer string may be missing. A checksum in {HEX} or [CRC:...]
form may follow; strip it before parsing.

  F!104 <3 chars> ST[XX] GR[grid] FM[text] <comment> #XXXX
      Char 1: 1 = Green, 2 = Yellow, 3 = Red. Char 2: situation report
      code. Char 3: 0 = no alert, E = Awareness/Alert Report active.
  F!309 <6 chars> DS[description] GR[grid] #XXXX
      One character per field: Affected Area, Category, Severity, Time
      Frame, Status, Expiration, using the @ codes in MCF309.txt.
  F!500 <13 digits> ST[XX] GR[grid] CT[city/county] #XXXX
      One digit (1-9) per question, in the order of MCF500.txt.
  F!505 <32 chars> ST[XX] GR[grid] CT[city/county] #XXXX
      One score per field, 1 to 32: 0, 1, 2 or 3, or X for unknown.

ST is a 2-letter state, GR a Maidenhead grid of 4 to 10 characters.

#XXXX is a datecode: the time the form was prepared, in UTC. It is not
a checksum. Each of the 4 characters is one part:

  Month   A-L = January to December
  Day     A-Z = 1 to 26; 0-4 = 27 to 31
  Hour    A-X = 0 to 23
  Minute  A-Z = 0, 2, 4 ... 50; 0-3 = 52, 54, 56, 58

The year isn't sent; use the current UTC year. Example: #KCMA is
November 3, 12:00 UTC.

CommStat StatReps come in more than one valid format. The easiest
route is the JS8Spotter database's csstatrep table, where the fields
are already split into columns (see OTHER DATA SOURCES).


SENDING DATA BACK
-----------------
A companion adds data by writing an import file in its own folder, which
the user then loads from the File menu. SitRepMapper's existing merge
rules handle duplicates, so importing the same file twice does no harm.

  To add      Write this file                  User loads it with
  ----------  -------------------------------  ----------------------------
  Flag pins   <name>.flags.json, same layout   File > Import Flag Pins...
              as flags.flags.json
  Areas       <name>.areas.json, same layout   File > Import Areas...
              as areas.areas.json
  Reports     A sitrepmapper_reports file,     File > Import/Export
              same layout as                   Reports > Import Report
              reports.reports.json (id and     Library...
              stored_at may be omitted;
              "deleted" is not needed)

Rules for files a companion writes:

  - Give each new flag or area a fresh UUID in "id", and set
    "updated_at" to the current UTC time.
  - To change a pin that already exists, reuse its "id" with a newer
    "updated_at". The newer copy replaces the old one.
  - Leave out "deleted" unless the companion really means to remove
    items. An entry there deletes that id on import.
  - Write all times in UTC, in the same formats the live files use.


SHOWING THINGS ON SCREEN
------------------------
Available in SitRepMapper v1.18.0 and later.

A companion can put a message on a banner under the toolbar, or a color
block at the right end of the toolbar, by keeping a small text file in
~/.config/sitrepmapper/companions/. SitRepMapper checks that folder
every 5 seconds. It only reads these files; it never moves, changes or
deletes them. Writing them is optional: a companion can have its own
window, or no window at all.

SitRepMapper creates the folder when it starts. A companion can start
first, so it should create the folder too if it's missing.

File names

  <CODE>.banner      Text for the banner
  <CODE>.indicator   A color block

  CODE is exactly 3 characters, A-Z or 0-9, upper case. Each companion
  picks its own, and two companions using the same code will overwrite
  each other's files. There's no central list. SRM (SitRepMapper) and
  TST (testing) are reserved. Anything else in the folder is ignored,
  including *.tmp files.

  Keep one file per display and replace it to update the display.
  Delete the file to clear the display.

File layout

  #DET
  ACTION: BANNER
  MODE: SCROLL
  MAXAGE: 120

  Bridge out on Route 9 near mile 14
  Shelter open at Lincoln School

  #DET
  ACTION: INDICATOR
  COLOR: YELLOW
  TEXT: Caution

  Line 1 is # plus the code, with nothing else on the line. It must
  match the file name exactly. Then come "LABEL: value" lines, in any
  order. On a banner, a blank line and then the text follow. Each
  non-blank line of text is one banner item.

  Label    Used on     Required?  Values                     Default
  -------  ----------  ---------  -------------------------  -------
  ACTION   Both        Yes        BANNER or INDICATOR, must  None
                                  match the file name
  MAXAGE   Both        No         Whole minutes, 1 or more   60
  MODE     Banner      No         STATIC, SCROLL, ALERT      STATIC
  COLOR    Indicator   Yes        GREEN, YELLOW, RED, GRAY   None
  TEXT     Indicator   No         Up to 10 characters        A letter

Format rules

  - Labels and values can be upper or lower case, and spaces around
    them are ignored. The #CODE must be upper case. Banner text and
    TEXT stay exactly as written.
  - If a label is repeated, the last one wins.
  - Unknown labels are ignored, so new ones can be added later.
  - An unknown value on a known label makes the file unreadable
    (COLOR: PURPLE, MAXAGE: soon). So does a line above the blank line
    with no colon, a code or action that disagrees with the file name,
    or a banner with no text.
  - Files must be UTF-8 and no larger than 64 KB. Windows line endings
    are fine.
  - Age is measured from the file's modified time. Past MAXAGE, the
    file counts as expired.
  - One action per file.

How a banner behaves

  - All companions' items join into one line with a bullet between
    them. The line scrolls if any item is SCROLL, or if it's too long
    to fit the window.
  - Text is limited to 500 characters per file. Past that it's cut off
    and logged, and the rest still shows.
  - An expired or unreadable file doesn't show.
  - The strip shows 10 items at most, dropping those from the oldest
    files first.
  - Any ALERT file turns the strip red, with alert items first and
    other messages after them. Alert items are never dropped by the
    10-item limit. They stay until they expire or the file is deleted,
    and the user can't close them, so use ALERT sparingly and delete
    the file when the alert is over.
  - The user can right-click a non-alert banner and choose Close. It
    stays closed until a message appears that wasn't there when it was
    closed, so rewriting the same text doesn't reopen it.
  - A message replaced between two checks may never appear. Don't use
    a banner for anything that must be seen.

How an indicator behaves

  - Each companion gets one block, shown in code order, up to three.
    Any others are logged and not shown.
  - TEXT is a word or short phrase on the block. Past 10 characters
    it's cut off and logged. Wide words shrink to fit. With no TEXT,
    the block shows G, Y or R, or a dot on gray.
  - An expired or unreadable file shows gray with a dot, never the old
    word. This is the one meaning SitRepMapper itself gives a color: a
    block that isn't being updated. A companion that sets COLOR: GRAY
    keeps its TEXT.
  - Hovering shows the code and the file's age.
  - What each color or word means is up to the companion. Explain it
    on the info sheet (see COMPANION INFO SHEET).

Writing the files safely

Write to a temporary name, then rename, so SitRepMapper never catches
a half-written file. Rewrite an indicator on a timer even if nothing
has changed, with MAXAGE at least twice that interval. Then the block
turns gray if the companion stops, which is how the user can tell.

  import os

  COMPANIONS_DIR = os.path.expanduser('~/.config/sitrepmapper/companions')


  def write_display_file(name, text):
      """Replace one display file in a single step."""
      os.makedirs(COMPANIONS_DIR, exist_ok=True)
      path = os.path.join(COMPANIONS_DIR, name)
      tmp = path + '.tmp'
      with open(tmp, 'w', encoding='utf-8') as f:
          f.write(text)
      os.replace(tmp, path)


  write_display_file('DET.indicator',
                     '#DET\nACTION: INDICATOR\nCOLOR: GREEN\n'
                     'TEXT: All Clear\nMAXAGE: 5\n')

  write_display_file('DET.banner',
                     '#DET\nACTION: BANNER\nMAXAGE: 30\n\n'
                     'Shelter open at Lincoln School\n')

Finding problems

~/.config/sitrepmapper/companions.log records unreadable files with the
reason, banner text changes, cut-off text and extra indicators. It's
only written when something changes, it keeps only recent entries, and
its wording may change. Read it while testing; don't build on it.

Starting and stopping

SitRepMapper doesn't start or stop companions. A companion can start
from desktop autostart on Linux and the Pi, or the Startup folder on
Windows, and can run while SitRepMapper is closed. One with a window
can simply be closed. One without a window needs a way to stop, such as
a Stop button in its settings using a PID file.


STATIONS A COMPANION RESPONDS TO
--------------------------------
A companion that acts on what stations send keeps two callsign lists in
its own settings. SitRepMapper has no setting for this.

  Always Ignore       Never act on these stations.
  Ignore all except   Act only on these stations. * means everyone.

  - An empty "Ignore all except" list means nobody can trigger the
    companion. Say so in its settings and on its info sheet.
  - Always Ignore wins if a callsign is on both lists.
  - JS8 callsigns aren't verified. A list keeps out noise and mistakes,
    but it can't prove who sent something.
  - IGNORE.TXT isn't applied for you. Read it yourself to honor it (see
    WHAT THE MAP SHOWS).
  - Requests that arrive as E? pulls are already covered by each Expect
    entry's Allowed field and the Expect Blocklist.

  def may_trigger(callsign, always_ignore, ignore_all_except):
      """Both lists hold upper-case callsigns. An empty
      ignore_all_except means nobody."""
      callsign = callsign.upper()
      if callsign in always_ignore:
          return False
      return '*' in ignore_all_except or callsign in ignore_all_except


COMPANION INFO SHEET
--------------------
Each companion ships a short text file named <ProgramName>_<DocType>.txt,
for example FlagWatch_Info.txt, so the user knows what it does before
running it. SitRepMapper has no list of installed companions. Copy this
template:

  <Program name> v<version>
  By <author>

  Purpose:        What it does, in a sentence or two.
  Code:           Its 3-character code, and any Expect codes the user
                  must know.
  Display:        What each color, word and banner means.
  Transmits:      Whether it transmits, when, and how the user turns
                  that on.
  Callsign lists: Where they're set. An empty "Ignore all except" list
                  means nobody can trigger it.
  Reads:          The files it reads.
  Writes:         The files it writes.
  Start:          How to start it, and whether it starts by itself.
  Stop:           How to stop it.
  Remove:         How to remove it, including any files it leaves.


SAFE READING RECIPE
-------------------
SitRepMapper saves each data file by writing a temporary <file>.tmp
and then swapping it into place, so a reader normally sees either the
complete old file or the complete new one. On Windows, if another
program has the file open at that moment, the save falls back to
writing over the file directly, so a companion can still very rarely
catch a file half-written. The fix: if the JSON fails to parse, wait a
moment and try again.

  import json
  import os
  import time

  CONFIG_DIR = os.path.expanduser('~/.config/sitrepmapper')


  def read_sitrep_file(name, expected_format, retries=3):
      """Read one SitRepMapper data file. Returns None if it can't be
      read right now: missing, still mid-write after a few tries, or a
      format this companion doesn't know."""
      path = os.path.join(CONFIG_DIR, name)
      for _ in range(retries):
          try:
              with open(path, encoding='utf-8') as f:
                  data = json.load(f)
          except FileNotFoundError:
              return None
          except (json.JSONDecodeError, OSError):
              time.sleep(0.5)      # caught it mid-save; try again
              continue
          if data.get('format') != expected_format:
              return None
          if not str(data.get('version', '')).startswith('1.'):
              return None          # a format this companion doesn't know
          return data
      return None


  def file_signature(name):
      """Modified time plus size. Re-read the file only when this
      changes. The size catches two quick saves that land in the same
      2-second slot on FAT/exFAT USB drives."""
      try:
          st = os.stat(os.path.join(CONFIG_DIR, name))
      except FileNotFoundError:
          return None
      return (st.st_mtime_ns, st.st_size)


  data = read_sitrep_file('flags.flags.json', 'sitrep_flags')
  if data is not None:
      flags = data.get('flags', [])

The same call reads ('areas.areas.json', 'sitrep_areas') with the key
'areas', ('reports.reports.json', 'sitrepmapper_reports') with the
key 'reports', and ('statusboard.statusboard.json',
'sitrepmapper_statusboard') with the key 'stations'. None isn't the same as an empty list: keep the last good
data, or a failed read will look as if everything was deleted.

To decode library reports with SitRepMapper's own parsers, add its
program folder to the import path:

  import sys
  sys.path.insert(0, '/path/to/sitrepmapper')
  import report_store

  report = report_store.report_from_envelope(reports[0])

Other habits that keep a companion well-behaved:

  - Checking file_signature() is nearly free, so every 5 to 10 seconds
    is fine. Re-read a file only when its signature changes. Don't hold
    files open.
  - To find what's new, keep the set of "id" values already seen.
    Reports, flags and areas all have one.
  - Never write to the live files, and ignore any *.tmp files in the
    config folder. They are SitRepMapper's saves in progress.
  - A file that stops changing may mean no new traffic, or that
    SitRepMapper is closed. Showing how long ago the file last changed
    lets the user tell which. statusboard.statusboard.json settles it:
    it's rewritten at least every 10 minutes while SitRepMapper runs.
  - Open the JS8Spotter database read-only (mode=ro) every time.
  - Check the companion's own load on a Raspberry Pi. It shares the CPU
    with SitRepMapper, JS8Call and JS8Spotter.


STABILITY AND COMPATIBILITY
---------------------------
The file formats are the stable part. They double as SitRepMapper's
export formats and are shared with other tools, so they won't change
lightly. If they do change, the "version" field changes and
SitRepMapper_CHANGELOG.txt says so.

  format names, field names, version 1.0
      Stable. Check "format" before reading.

  New fields appearing in records
      Possible. Ignore fields you don't recognize. Existing fields
      won't be removed or renamed within version 1.x.

  config.json keys
      Mostly stable, but keys can be added or retired between versions.
      Always supply a default.

  statusboard.statusboard.json
      Stable, like the other data files. "status" and "reason" are
      display text whose wording may change; the coded values don't.

  Display files in companions/
      Stable. Labels and values are only added to; existing ones won't
      change or be removed. companions.log isn't part of this.

  Python helpers (report_from_envelope, JS8SpotterDB, parse_line,
  centroid_latlon)
      Internal code, not a promise. Usually unchanged, but recheck after
      a SitRepMapper update.

  expect.expect.json, send_drafts.json
      Not documented for companions; may change at any time.
