# USMT Dash

A real-time migration dashboard for Windows device-refresh projects. It tracks every old-to-new
PC migration driven by ConfigMgr (SCCM) and USMT: who's online, how much disk is free, where
each backup and restore is, and which specialty apps still need to go onto the new machine.

It's written in **PowerShell** end to end: an `HttpListener` web server with a JSON API, a
polling worker that queries SCCM's SQL database, Active Directory and the devices themselves,
and a single-file HTML/JS front end. It needs no web framework, database server or runtime
beyond PowerShell.

> **Live demo:** the hosted demo runs the real server and worker in a container against a
> simulated 200-device fleet. Open the **Terminals** panel at the bottom of the page to watch
> them work.

![Overview page](docs/screenshots/overview.png)

## Why it exists

On a large hardware refresh, a technician checking one device in the ConfigMgr console (the
association, whether the backup started, whether the box is even on the network) took about a
minute. USMT Dash brings that down to under a second for every device at once, and gives a
team a shared view of the whole refresh.

Built and run during a real enterprise Windows 11 refresh, where the team's daily build rate
went from about 10 to 25 devices a day and the project closed out about 1,900 machines against
a 1,300-device scope.

## Features

- **Overview**: the landing page, built to drive a refresh's open issues to zero. Counters for
  offline sources, not started, *Bad Client* and *Broken WMI* tags, devices missing from AD /
  SCCM / DNS, low disk, backup errors and undocumented apps; overall progress; devices per
  Backup Status; a building x floor map of what's left; and a calendar of the deployment-date
  tags showing how far each day's devices have got. Every number opens the devices behind it.
- **Tracked Devices**: technicians register devices under tags (a name, a date batch, a
  department). The worker only sweeps tagged devices. Tags can be flagged *Skip USMT* for
  inventory-only tracking, and `SOURCE, DEST` lines create manual pairs for hardware exceptions.
- **USMT Status**: every migration grouped by state (*Not Started, Backup In Progress, Backup
  Completed, Restore In Progress, Completed, Not Associated*), with ping, free space and the
  latest scanstate/loadstate log line. Each column can be sorted, filtered and copied.
- **Smart status engine**: combines the ConfigMgr `MigrationStatus` with whether the USMT
  progress log in `SMSTSLog` is actively being written. Completed pairs are cached and skip
  network checks on later cycles. A reverted association resets the cache.
- **Network checks that tell the truth**: resolves *all* A records, pings each one, and
  validates reverse DNS. This separates *Online*, *Timed Out*, *Not Found* and *DNS Issue*
  (a stale record pointing at a different machine).
- **Missing software**: compares installed software on source and destination, filters out the
  baseline image using a pattern list, groups multi-part products into suites, and lists the
  specialty apps not yet on the new device.
- **Device details**: AD OU and groups, hardware, disk, RAM, monitors, client health,
  location, department, users, for both sides of a migration. A blank says why: *Not in AD*,
  *Not in SCCM*, *No DNS record*, or *Waiting for worker...* if it hasn't been looked up yet.
  The dialog refreshes in place when the worker finishes a cycle.
- **Presence**: every ping is recorded, so each device shows when it was last online and how
  many days it has been offline. Completed pairs and Skip USMT devices, which skip network checks
  in the sweep, are pinged hourly during business hours and at midnight to keep theirs current.
- **Snapshots**: the last free space and scanstate / loadstate lines seen while a device was
  reachable are kept with their timestamps, and shown for completed migrations and offline
  devices instead of blanks.
- **Excel-style filters** on every table: sort, search and tick values; list columns match
  *any*, *only* or *exactly* the ticked items. On software columns, one click shows devices
  whose apps all have a documentation article, and *Easiest first* sorts by how many apps
  still lack one.
- **Custom View**: build your own table from any field (including days offline, last online,
  found in AD/SCCM/DNS and monitors), with drag-to-reorder, resizable and Excel-style
  filterable columns. Saved per browser.
- **Workbench**: personal "stacks" of devices for the cart in front of you.
- **Knowledge base**: Markdown articles with an editor, image manager, badges, alerts and
  click-to-copy links. Each article has its own address (`/docs/Adobe Acrobat Pro`); the list
  is searchable, counted and split into *Applications* and *Troubleshooting & Guides*; PDF
  export and print. Renaming an image rewrites every article that links to it. Articles named
  after an app are linked from that app in the software view.
- **Business-hours scheduling**: the worker cycles every few minutes during working hours,
  sleeps overnight, and makes a midnight "pitstop" for the daily software refresh.
  *Force Update* and *Force Full Refresh* skip the wait. The dashboard updates live while a cycle runs
  (swept rows are published every few seconds), newly tracked devices are swept and
  published first, the sync dialog shows how long each kind of device (SCCM association,
  manual pair, completed snapshot...) took.

| | |
|---|---|
| ![USMT Status](docs/screenshots/usmt-status.png) | ![Terminals](docs/screenshots/terminals.png) |
| ![Missing software](docs/screenshots/missing-software.png) | ![Custom View](docs/screenshots/custom-view.png) |
| ![Device details](docs/screenshots/device-details.png) | ![Documentation](docs/screenshots/documentation.png) |

## Architecture

![Architecture](Documentation/images/USMT%20Dash%20Architecture.png)

The server and worker never talk to each other directly. `Data\` is the entire interface:
the worker writes JSON state files, the server serves them, and the dashboard's "force
refresh" buttons drop flag files that the worker watches for.

| Component | What it does |
|---|---|
| `USMT-Server.ps1` | `System.Net.HttpListener` server: static files, a JSON API (17 routes), IP allowlist, gzip |
| `USMT-Worker.ps1` | Polling loop: builds the working set from the tags, sweeps each device, works out its status, writes `Data\*.json` |
| `Get-Software.ps1` | Per-device installed-software lookup with a 24-hour cache (used by the worker and the API) |
| `Providers\SCCM.ps1` | Real data source: ConfigMgr module, site SQL database, ADSI, DNS/ICMP, admin shares |
| `Providers\Demo.ps1` | Simulated data source used by the demo (see below) |
| `Config\usmt-dash.psd1` | Every environment-specific setting |
| `web\index.html` | The dashboard (single-file HTML/JS, about 4,500 lines) |
| `web\demo\demo.js` | Terminal panel, plus the demo-mode browser sandbox |
| `web\overview.js` | Overview page (issue counters, progress, locations, deployment calendar) |
| `web\device-info.js` | Shared device helpers: missing-data reasons, presence, snapshot markers |
| `web\docs.js` | Knowledge base: permalinks, search, categories, PDF / print |

### API

| Method & path | Purpose |
|---|---|
| `GET /` and static files | Dashboard, assets, `Data\` state files, `Documentation\` |
| `GET /tracked` · `/workbench` · `/status` · `/devices` · `/custom` · `/docs` | The dashboard, opened on that page (the logo and name go to `/`, the Overview) |
| `GET /docs/<title>` | The dashboard, opened on that knowledge-base article |
| `GET /get-config` | Mode flags for the front end |
| `GET /get-logs?source=server\|worker&after=<id>` | Tail of the service console output (terminal panel) |
| `POST /force-update` · `POST /force-full` | Wake the worker now (full = also refresh software/inventory caches) |
| `GET /get-blacklist` | Baseline-app pattern list |
| `GET /get-all-tags` · `GET /get-tracked?tech=` · `GET /get-tag-meta?tech=` | Read tracking tags |
| `POST /update-tracked` | Set a tag's devices and *Skip USMT* flag |
| `GET /get-software?name=` | Installed software for one device |
| `GET /get-docs-list` · `POST /save-doc` · `POST /delete-doc` | Knowledge base (delete archives to `Documentation\Archive`) |
| `GET /list-images` · `POST /upload-image` · `POST /rename-image` · `POST /delete-image` | Knowledge-base images |

## Demo mode

The public demo has to show the whole dashboard working without real infrastructure, and
without letting strangers change what other visitors see.

- **It's the real code.** The container runs the unmodified `USMT-Server.ps1` and
  `USMT-Worker.ps1` under PowerShell 7 with `-Demo`. The worker loads `Providers\Demo.ps1`
  instead of `Providers\SCCM.ps1`, and that's the only difference.
- **A believable fleet.** The demo provider derives each device's hardware, users, location,
  department and software from a hash of its name. Each migration moves through every stage
  on a 24-hour clock, so the dashboard keeps changing. Devices go offline, have stale DNS,
  run low on disk, and fail backups at realistic rates.
- **Read-only API.** In demo mode the server refuses every route that writes (and
  `get-software`, which takes free-text input) with `403`. The terminal output is the real
  console of both processes, mirrored to `Data\Logs` by a `Write-Host` proxy.
- **Visitor changes stay in their browser.** `web\demo\demo.js` intercepts the dashboard's
  write requests and keeps the visitor's tags, articles and images in IndexedDB, merged over
  the server's data. Their actions print in the terminal panel exactly as the services would,
  marked with a small dot. Nobody else sees them, and nothing a visitor types reaches the
  server, so there's nothing to moderate or inject. **DEMO > Reset my changes** clears it.

Without `-Demo`, none of this applies: `demo.js` passes every request straight through and
the dashboard works against the real backend.

## Running it

### Demo (Docker / Coolify)

```bash
docker compose up -d --build        # http://localhost:8080
```

On **Coolify**, create a resource from this repository using the *Dockerfile* (or *Docker
Compose*) build pack, set the exposed port to **8080**, and attach a domain. The container
reseeds the simulated fleet on every start and needs no volumes or secrets. Set `TZ` to show
times in your timezone.

### Demo (local PowerShell 7)

```powershell
pwsh ./Demo/New-DemoData.ps1 -Force
pwsh ./USMT-Worker.ps1 -Demo        # terminal 1
pwsh ./USMT-Server.ps1 -Demo        # terminal 2 -> http://localhost:8080
```

### Live, against your environment

Requirements: Windows PowerShell 5.1, the ConfigMgr console (for the `ConfigurationManager`
module), read access to the site database, AD read access, and admin-share access to devices.

1. Edit `Config\usmt-dash.psd1`: site code, SMS Provider, SQL server and database, and (if
   you extend hardware inventory with an asset/location class) the view and column map
   under `AssetInventory`.
2. Start the worker, then the server:
   ```powershell
   .\USMT-Worker.ps1          # add -v for per-device tracing
   .\USMT-Server.ps1          # binding http://*:8080/ may need: netsh http add urlacl url=http://*:8080/ user=DOMAIN\account
   ```
3. Optionally restrict access by listing allowed IPs or hostnames in `Data\Whitelisted-IPs.txt`.
4. Put baseline-image apps in `Data\Blacklisted-Apps.txt` so only specialty software shows as
   missing. See `Demo\Blacklisted-Apps.txt` for the syntax (plain patterns, `Suite = a*, b*`,
   `Rule: DEVICE* = app*`, `Combo: Label = a* && b*`).

### Adapting it to a different toolset

All infrastructure access goes through a provider. To support another platform (Intune, a
different CMDB, PDQ, ...), add `Providers\<Name>.ps1`, implement these functions, and set
`Provider = '<Name>'` in the config:

| Function | Returns |
|---|---|
| `Initialize-Provider` | One-time setup (import modules, ...) |
| `Open-ProviderCycle` / `Close-ProviderCycle` | Per-cycle session (connections) |
| `pingDevice <name>` | `Online`, `Timed Out`, `Not Found` or `DNS Issue` |
| `Resolve-DeviceIP <name>` | Current IP, or `$null` |
| `Get-DiskSpace <name>` | `@{ Total = '238 GB'; Free = '80 GB' }` |
| `Get-LogInfo <name> <logName>` | Latest USMT progress log line (via the shared parser) |
| `Get-DeviceAssociations <name>` | Objects with `SourceComputer`, `DestinationComputer`, `MigrationStatus` (0/1/2) |
| `Get-DeviceInventory <name> <session>` | Hashtable of detail fields (OU, Groups, Model, RAM, MonitorCount, Monitors, Location, ...), plus `InAD` / `InSCCM` (`$true` / `$false`) |
| `Get-InstalledSoftware <name>` | Rows of `Name`, `Version`, `Publisher`, `Source`, `InstallDate` |
| `Get-DeviceLastSeen <name> <session>` *(optional)* | When the device was last active, used to backfill "last online" the first time it's tracked |

`Providers\Demo.ps1` is a complete working example that needs no infrastructure.

## Project layout

```
USMT-Server.ps1          web server + JSON API
USMT-Worker.ps1          polling worker
Get-Software.ps1         per-device software lookup + cache
Config/usmt-dash.psd1    configuration
Lib/Common.ps1           config loader, console-log mirror, shared I/O
Providers/SCCM.ps1       ConfigMgr / SQL / AD / SMB data source
Providers/Demo.ps1       simulated fleet
Demo/                    demo seed script + sample app blacklist
Documentation/           knowledge-base articles and images
web/                     dashboard, terminal panel / demo sandbox, vendored libraries
Dockerfile, docker/      container for the hosted demo
Data/                    runtime state (created on first run, not committed)
```

## Third-party code

Vendored in `web/vendor/` under their own licenses: [marked](https://github.com/markedjs/marked)
(MIT), [highlight.js](https://github.com/highlightjs/highlight.js) (BSD-3-Clause),
[html2pdf.js](https://github.com/eKoopmans/html2pdf.js) (MIT).