Johan Sandoval
f72e39892c
- Every nav page has its own address (/tracked, /status, /devices, ...); the logo and name go to the Overview; back/forward work. - List filters gain an "Only ticked" mode; software columns get a one-click "devices with only documented apps" pick and an easiest-first sort. - Overview: the device list uses the dashboard's table header (sticky, with Excel filters); what opened it is clearly highlighted; locations colored by how much is done (red to green); stages read red / amber / green. - Details: one-line title bar; presence and monitors only in Advanced View. - Presence sweep pings only completed pairs (everything else is pinged by the sweep itself), hourly and at midnight. - Custom View scrollbar smaller, in the table header blue. - Docs: no sideways scroll in the article list; no blank lines around code.
231 lines
14 KiB
Markdown
231 lines
14 KiB
Markdown
# USMT Dash
|
|
|
|
A real-time migration dashboard for Windows device-refresh projects. It tracks every old-to-new
|
|
PC migration driven by ConfigMgr (SCCM) and USMT: who's online, how much disk is free, where
|
|
each backup and restore is, and which specialty apps still need to go onto the new machine.
|
|
|
|
It's written in **PowerShell** end to end: an `HttpListener` web server with a JSON API, a
|
|
polling worker that queries SCCM's SQL database, Active Directory and the devices themselves,
|
|
and a single-file HTML/JS front end. It needs no web framework, database server or runtime
|
|
beyond PowerShell.
|
|
|
|
> **Live demo:** the hosted demo runs the real server and worker in a container against a
|
|
> simulated 200-device fleet. Open the **Terminals** panel at the bottom of the page to watch
|
|
> them work.
|
|
|
|
|
|
|
|
## Why it exists
|
|
|
|
On a large hardware refresh, a technician checking one device in the ConfigMgr console (the
|
|
association, whether the backup started, whether the box is even on the network) took about a
|
|
minute. USMT Dash brings that down to under a second for every device at once, and gives a
|
|
team a shared view of the whole refresh.
|
|
|
|
Built and run during a real enterprise Windows 11 refresh, where the team's daily build rate
|
|
went from about 10 to 25 devices a day and the project closed out about 1,900 machines against
|
|
a 1,300-device scope.
|
|
|
|
## Features
|
|
|
|
- **Overview**: the landing page, built to drive a refresh's open issues to zero. Counters for
|
|
offline sources, not started, *Bad Client* and *Broken WMI* tags, devices missing from AD /
|
|
SCCM / DNS, low disk, backup errors and undocumented apps; overall progress; devices per
|
|
Backup Status; a building x floor map of what's left; and a calendar of the deployment-date
|
|
tags showing how far each day's devices have got. Every number opens the devices behind it.
|
|
- **Tracked Devices**: technicians register devices under tags (a name, a date batch, a
|
|
department). The worker only sweeps tagged devices. Tags can be flagged *Skip USMT* for
|
|
inventory-only tracking, and `SOURCE, DEST` lines create manual pairs for hardware exceptions.
|
|
- **USMT Status**: every migration grouped by state (*Not Started, Backup In Progress, Backup
|
|
Completed, Restore In Progress, Completed, Not Associated*), with ping, free space and the
|
|
latest scanstate/loadstate log line. Each column can be sorted, filtered and copied.
|
|
- **Smart status engine**: combines the ConfigMgr `MigrationStatus` with whether the USMT
|
|
progress log in `SMSTSLog` is actively being written. Completed pairs are cached and skip
|
|
network checks on later cycles. A reverted association resets the cache.
|
|
- **Network checks that tell the truth**: resolves *all* A records, pings each one, and
|
|
validates reverse DNS. This separates *Online*, *Timed Out*, *Not Found* and *DNS Issue*
|
|
(a stale record pointing at a different machine).
|
|
- **Missing software**: compares installed software on source and destination, filters out the
|
|
baseline image using a pattern list, groups multi-part products into suites, and lists the
|
|
specialty apps not yet on the new device.
|
|
- **Device details**: AD OU and groups, hardware, disk, RAM, monitors, client health,
|
|
location, department, users, for both sides of a migration. A blank says why: *Not in AD*,
|
|
*Not in SCCM*, *No DNS record*, or *Waiting for worker...* if it hasn't been looked up yet.
|
|
The dialog refreshes in place when the worker finishes a cycle.
|
|
- **Presence**: every ping is recorded, so each device shows when it was last online and how
|
|
many days it has been offline. Completed pairs, which skip network checks in the sweep, are
|
|
pinged hourly during business hours and at midnight to keep theirs current.
|
|
- **Snapshots**: the last free space and scanstate / loadstate lines seen while a device was
|
|
reachable are kept with their timestamps, and shown for completed migrations and offline
|
|
devices instead of blanks.
|
|
- **Excel-style filters** on every table: sort, search and tick values; list columns match
|
|
*any*, *only* or *exactly* the ticked items. On software columns, one click shows devices
|
|
whose apps all have a documentation article, and *Easiest first* sorts by how many apps
|
|
still lack one.
|
|
- **Custom View**: build your own table from any field (including days offline, last online,
|
|
found in AD/SCCM/DNS and monitors), with drag-to-reorder, resizable and Excel-style
|
|
filterable columns. Saved per browser.
|
|
- **Workbench**: personal "stacks" of devices for the cart in front of you.
|
|
- **Knowledge base**: Markdown articles with an editor, image manager, badges, alerts and
|
|
click-to-copy links. Each article has its own address (`/docs/Adobe Acrobat Pro`); the list
|
|
is searchable, counted and split into *Applications* and *Troubleshooting & Guides*; PDF
|
|
export and print. Renaming an image rewrites every article that links to it. Articles named
|
|
after an app are linked from that app in the software view.
|
|
- **Business-hours scheduling**: the worker cycles every few minutes during working hours,
|
|
sleeps overnight, and makes a midnight "pitstop" for the daily software refresh.
|
|
*Force Update* and *Force Full Refresh* skip the wait. Newly tracked devices are swept and
|
|
published first; the sync dialog shows how long each kind of device (SCCM association,
|
|
manual pair, completed snapshot...) took.
|
|
|
|
| | |
|
|
|---|---|
|
|
|  |  |
|
|
|  |  |
|
|
|  |  |
|
|
|
|
## Architecture
|
|
|
|
|
|
|
|
The server and worker never talk to each other directly. `Data\` is the entire interface:
|
|
the worker writes JSON state files, the server serves them, and the dashboard's "force
|
|
refresh" buttons drop flag files that the worker watches for.
|
|
|
|
| Component | What it does |
|
|
|---|---|
|
|
| `USMT-Server.ps1` | `System.Net.HttpListener` server: static files, a JSON API (17 routes), IP allowlist, gzip |
|
|
| `USMT-Worker.ps1` | Polling loop: builds the working set from the tags, sweeps each device, works out its status, writes `Data\*.json` |
|
|
| `Get-Software.ps1` | Per-device installed-software lookup with a 24-hour cache (used by the worker and the API) |
|
|
| `Providers\SCCM.ps1` | Real data source: ConfigMgr module, site SQL database, ADSI, DNS/ICMP, admin shares |
|
|
| `Providers\Demo.ps1` | Simulated data source used by the demo (see below) |
|
|
| `Config\usmt-dash.psd1` | Every environment-specific setting |
|
|
| `web\index.html` | The dashboard (single-file HTML/JS, about 4,500 lines) |
|
|
| `web\demo\demo.js` | Terminal panel, plus the demo-mode browser sandbox |
|
|
| `web\overview.js` | Overview page (issue counters, progress, locations, deployment calendar) |
|
|
| `web\device-info.js` | Shared device helpers: missing-data reasons, presence, snapshot markers |
|
|
| `web\docs.js` | Knowledge base: permalinks, search, categories, PDF / print |
|
|
|
|
### API
|
|
|
|
| Method & path | Purpose |
|
|
|---|---|
|
|
| `GET /` and static files | Dashboard, assets, `Data\` state files, `Documentation\` |
|
|
| `GET /tracked` · `/workbench` · `/status` · `/devices` · `/custom` · `/docs` | The dashboard, opened on that page (the logo and name go to `/`, the Overview) |
|
|
| `GET /docs/<title>` | The dashboard, opened on that knowledge-base article |
|
|
| `GET /get-config` | Mode flags for the front end |
|
|
| `GET /get-logs?source=server\|worker&after=<id>` | Tail of the service console output (terminal panel) |
|
|
| `POST /force-update` · `POST /force-full` | Wake the worker now (full = also refresh software/inventory caches) |
|
|
| `GET /get-blacklist` | Baseline-app pattern list |
|
|
| `GET /get-all-tags` · `GET /get-tracked?tech=` · `GET /get-tag-meta?tech=` | Read tracking tags |
|
|
| `POST /update-tracked` | Set a tag's devices and *Skip USMT* flag |
|
|
| `GET /get-software?name=` | Installed software for one device |
|
|
| `GET /get-docs-list` · `POST /save-doc` · `POST /delete-doc` | Knowledge base (delete archives to `Documentation\Archive`) |
|
|
| `GET /list-images` · `POST /upload-image` · `POST /rename-image` · `POST /delete-image` | Knowledge-base images |
|
|
|
|
## Demo mode
|
|
|
|
The public demo has to show the whole dashboard working without real infrastructure, and
|
|
without letting strangers change what other visitors see.
|
|
|
|
- **It's the real code.** The container runs the unmodified `USMT-Server.ps1` and
|
|
`USMT-Worker.ps1` under PowerShell 7 with `-Demo`. The worker loads `Providers\Demo.ps1`
|
|
instead of `Providers\SCCM.ps1`, and that's the only difference.
|
|
- **A believable fleet.** The demo provider derives each device's hardware, users, location,
|
|
department and software from a hash of its name. Each migration moves through every stage
|
|
on a 24-hour clock, so the dashboard keeps changing. Devices go offline, have stale DNS,
|
|
run low on disk, and fail backups at realistic rates.
|
|
- **Read-only API.** In demo mode the server refuses every route that writes (and
|
|
`get-software`, which takes free-text input) with `403`. The terminal output is the real
|
|
console of both processes, mirrored to `Data\Logs` by a `Write-Host` proxy.
|
|
- **Visitor changes stay in their browser.** `web\demo\demo.js` intercepts the dashboard's
|
|
write requests and keeps the visitor's tags, articles and images in IndexedDB, merged over
|
|
the server's data. Their actions print in the terminal panel exactly as the services would,
|
|
marked with a small dot. Nobody else sees them, and nothing a visitor types reaches the
|
|
server, so there's nothing to moderate or inject. **DEMO > Reset my changes** clears it.
|
|
|
|
Without `-Demo`, none of this applies: `demo.js` passes every request straight through and
|
|
the dashboard works against the real backend.
|
|
|
|
## Running it
|
|
|
|
### Demo (Docker / Coolify)
|
|
|
|
```bash
|
|
docker compose up -d --build # http://localhost:8080
|
|
```
|
|
|
|
On **Coolify**, create a resource from this repository using the *Dockerfile* (or *Docker
|
|
Compose*) build pack, set the exposed port to **8080**, and attach a domain. The container
|
|
reseeds the simulated fleet on every start and needs no volumes or secrets. Set `TZ` to show
|
|
times in your timezone.
|
|
|
|
### Demo (local PowerShell 7)
|
|
|
|
```powershell
|
|
pwsh ./Demo/New-DemoData.ps1 -Force
|
|
pwsh ./USMT-Worker.ps1 -Demo # terminal 1
|
|
pwsh ./USMT-Server.ps1 -Demo # terminal 2 -> http://localhost:8080
|
|
```
|
|
|
|
### Live, against your environment
|
|
|
|
Requirements: Windows PowerShell 5.1, the ConfigMgr console (for the `ConfigurationManager`
|
|
module), read access to the site database, AD read access, and admin-share access to devices.
|
|
|
|
1. Edit `Config\usmt-dash.psd1`: site code, SMS Provider, SQL server and database, and (if
|
|
you extend hardware inventory with an asset/location class) the view and column map
|
|
under `AssetInventory`.
|
|
2. Start the worker, then the server:
|
|
```powershell
|
|
.\USMT-Worker.ps1 # add -v for per-device tracing
|
|
.\USMT-Server.ps1 # binding http://*:8080/ may need: netsh http add urlacl url=http://*:8080/ user=DOMAIN\account
|
|
```
|
|
3. Optionally restrict access by listing allowed IPs or hostnames in `Data\Whitelisted-IPs.txt`.
|
|
4. Put baseline-image apps in `Data\Blacklisted-Apps.txt` so only specialty software shows as
|
|
missing. See `Demo\Blacklisted-Apps.txt` for the syntax (plain patterns, `Suite = a*, b*`,
|
|
`Rule: DEVICE* = app*`, `Combo: Label = a* && b*`).
|
|
|
|
### Adapting it to a different toolset
|
|
|
|
All infrastructure access goes through a provider. To support another platform (Intune, a
|
|
different CMDB, PDQ, ...), add `Providers\<Name>.ps1`, implement these functions, and set
|
|
`Provider = '<Name>'` in the config:
|
|
|
|
| Function | Returns |
|
|
|---|---|
|
|
| `Initialize-Provider` | One-time setup (import modules, ...) |
|
|
| `Open-ProviderCycle` / `Close-ProviderCycle` | Per-cycle session (connections) |
|
|
| `pingDevice <name>` | `Online`, `Timed Out`, `Not Found` or `DNS Issue` |
|
|
| `Resolve-DeviceIP <name>` | Current IP, or `$null` |
|
|
| `Get-DiskSpace <name>` | `@{ Total = '238 GB'; Free = '80 GB' }` |
|
|
| `Get-LogInfo <name> <logName>` | Latest USMT progress log line (via the shared parser) |
|
|
| `Get-DeviceAssociations <name>` | Objects with `SourceComputer`, `DestinationComputer`, `MigrationStatus` (0/1/2) |
|
|
| `Get-DeviceInventory <name> <session>` | Hashtable of detail fields (OU, Groups, Model, RAM, MonitorCount, Monitors, Location, ...), plus `InAD` / `InSCCM` (`$true` / `$false`) |
|
|
| `Get-InstalledSoftware <name>` | Rows of `Name`, `Version`, `Publisher`, `Source`, `InstallDate` |
|
|
| `Get-DeviceLastSeen <name> <session>` *(optional)* | When the device was last active, used to backfill "last online" the first time it's tracked |
|
|
|
|
`Providers\Demo.ps1` is a complete working example that needs no infrastructure.
|
|
|
|
## Project layout
|
|
|
|
```
|
|
USMT-Server.ps1 web server + JSON API
|
|
USMT-Worker.ps1 polling worker
|
|
Get-Software.ps1 per-device software lookup + cache
|
|
Config/usmt-dash.psd1 configuration
|
|
Lib/Common.ps1 config loader, console-log mirror, shared I/O
|
|
Providers/SCCM.ps1 ConfigMgr / SQL / AD / SMB data source
|
|
Providers/Demo.ps1 simulated fleet
|
|
Demo/ demo seed script + sample app blacklist
|
|
Documentation/ knowledge-base articles and images
|
|
web/ dashboard, terminal panel / demo sandbox, vendored libraries
|
|
Dockerfile, docker/ container for the hosted demo
|
|
Data/ runtime state (created on first run, not committed)
|
|
```
|
|
|
|
## Third-party code
|
|
|
|
Vendored in `web/vendor/` under their own licenses: [marked](https://github.com/markedjs/marked)
|
|
(MIT), [highlight.js](https://github.com/highlightjs/highlight.js) (BSD-3-Clause),
|
|
[html2pdf.js](https://github.com/eKoopmans/html2pdf.js) (MIT).
|