Johan Sandoval
18dc8961e2
- README: configuration table, every web module, corrected filter and live update wording, Data/ file list. - Using USMT Dash article rewritten as a user guide: pages and addresses, tracking, device details, filtering, phones and tablets, tips. - Screenshots retaken from the current build.
262 lines
16 KiB
Markdown
262 lines
16 KiB
Markdown
# USMT Dash
|
|
|
|
A real-time migration dashboard for Windows device-refresh projects. It tracks every old-to-new
|
|
PC migration driven by ConfigMgr (SCCM) and USMT: who's online, how much disk is free, where
|
|
each backup and restore is, and which specialty apps still need to go onto the new machine.
|
|
|
|
It's written in **PowerShell** end to end: an `HttpListener` web server with a JSON API, a
|
|
polling worker that queries SCCM's SQL database, Active Directory and the devices themselves,
|
|
and a single-file HTML/JS front end. It needs no web framework, database server or runtime
|
|
beyond PowerShell.
|
|
|
|
> **Live demo:** the hosted demo runs the real server and worker in a container against a
|
|
> simulated 200-device fleet. Open the **Terminals** panel at the bottom of the page to watch
|
|
> them work.
|
|
|
|
|
|
|
|
## Why it exists
|
|
|
|
On a large hardware refresh, a technician checking one device in the ConfigMgr console (the
|
|
association, whether the backup started, whether the box is even on the network) took about a
|
|
minute. USMT Dash brings that down to under a second for every device at once, and gives a
|
|
team a shared view of the whole refresh.
|
|
|
|
Built and run during a real enterprise Windows 11 refresh, where the team's daily build rate
|
|
went from about 10 to 25 devices a day and the project closed out about 1,900 machines against
|
|
a 1,300-device scope.
|
|
|
|
## Features
|
|
|
|
- **Overview**: the landing page, built to drive a refresh's open issues to zero. Counters for
|
|
offline sources, not started, *Bad Client* and *Broken WMI* tags, devices missing from AD /
|
|
SCCM / DNS, low disk, backup errors and undocumented apps; overall progress; devices per
|
|
Backup Status; a building x floor map of what's left; and a calendar of the deployment-date
|
|
tags showing how far each day's devices have got. Every number opens the devices behind it.
|
|
- **Tracked Devices**: technicians register devices under tags (a name, a date batch, a
|
|
department). The worker only sweeps tagged devices. Tags can be flagged *Skip USMT* for
|
|
inventory-only tracking, and `SOURCE, DEST` lines create manual pairs for hardware exceptions.
|
|
- **USMT Status**: every migration grouped by state (*Not Started, Backup In Progress, Backup
|
|
Completed, Restore In Progress, Completed, Not Associated*), with ping, free space and the
|
|
latest scanstate/loadstate log line. Each column can be sorted, filtered and copied.
|
|
- **Smart status engine**: combines the ConfigMgr `MigrationStatus` with whether the USMT
|
|
progress log in `SMSTSLog` is actively being written. Completed pairs are cached and skip
|
|
network checks on later cycles. A reverted association resets the cache.
|
|
- **Network checks that tell the truth**: resolves *all* A records, pings each one, and
|
|
validates reverse DNS. This separates *Online*, *Timed Out*, *Not Found* and *DNS Issue*
|
|
(a stale record pointing at a different machine).
|
|
- **Missing software**: compares installed software on source and destination, filters out the
|
|
baseline image using a pattern list, groups multi-part products into suites, and lists the
|
|
specialty apps not yet on the new device.
|
|
- **Device details**: AD OU and groups, hardware, disk, RAM, monitors, client health,
|
|
location, department, users, for both sides of a migration. A blank says why: *Not in AD*,
|
|
*Not in SCCM*, *No DNS record*, or *Waiting for worker...* if it hasn't been looked up yet.
|
|
Open dialogs refresh in place as new data arrives.
|
|
- **Presence**: every ping is recorded, so each device shows when it was last online and how
|
|
many days it has been offline. Completed pairs and Skip USMT devices, which skip network checks
|
|
in the sweep, are pinged hourly during business hours and at midnight to keep theirs current.
|
|
- **Snapshots**: the last free space and scanstate / loadstate lines seen while a device was
|
|
reachable are kept with their timestamps, and shown for completed migrations and offline
|
|
devices instead of blanks.
|
|
- **Excel-style filters** on every table: sort, search and tick values; list columns match
|
|
*any*, *only* or *exactly* the ticked items. On software columns, *Documented* + *Only
|
|
ticked* shows devices whose apps all have a documentation article, and *Easiest first*
|
|
sorts by how many apps still lack one.
|
|
- **Phones and tablets**: tables become cards with one *Filter & sort* button, and a floating
|
|
*Jump* button finds a device by name (scrolls to its card without filtering) or jumps to any
|
|
section, the top or the bottom of the page.
|
|
- **Custom View**: build your own table from any field (including days offline, last online,
|
|
found in AD/SCCM/DNS and monitors), with drag-to-reorder, resizable and Excel-style
|
|
filterable columns. Saved per browser.
|
|
- **Workbench**: personal "stacks" of devices for the cart in front of you.
|
|
- **Knowledge base**: Markdown articles with an editor, image manager, badges, alerts and
|
|
click-to-copy links. Each article has its own address (`/docs/Adobe Acrobat Pro`); the list
|
|
is searchable, counted and split into *Applications* and *Troubleshooting & Guides*; PDF
|
|
export and print. Renaming an image rewrites every article that links to it. Articles named
|
|
after an app are linked from that app in the software view.
|
|
- **Live, on a schedule**: the worker cycles every few minutes during business hours, sleeps
|
|
overnight, and makes a midnight "pitstop" for the daily software refresh. While a cycle
|
|
runs, the rows swept so far are published every few seconds, so the dashboard updates live;
|
|
newly tracked devices are swept and published first. *Force Update* and *Force Full
|
|
Refresh* skip the wait, and the sync dialog shows how long each kind of device (SCCM
|
|
association, manual pair, completed snapshot...) took.
|
|
- **Every page has an address** (`/status`, `/devices`, `/docs/Adobe Acrobat Pro`, ...), so
|
|
links, bookmarks and the back button work. The logo goes home to the Overview.
|
|
|
|
| | |
|
|
|---|---|
|
|
|  |  |
|
|
|  |  |
|
|
|  |  |
|
|
|
|
## Architecture
|
|
|
|
|
|
|
|
The server and worker never talk to each other directly. `Data\` is the entire interface:
|
|
the worker writes JSON state files, the server serves them, and the dashboard's "force
|
|
refresh" buttons drop flag files that the worker watches for.
|
|
|
|
| Component | What it does |
|
|
|---|---|
|
|
| `USMT-Server.ps1` | `System.Net.HttpListener` server: static files, a JSON API (17 routes), IP allowlist, gzip |
|
|
| `USMT-Worker.ps1` | Polling loop: builds the working set from the tags, sweeps each device, works out its status, writes `Data\*.json` |
|
|
| `Get-Software.ps1` | Per-device installed-software lookup with a 24-hour cache (used by the worker and the API) |
|
|
| `Providers\SCCM.ps1` | Real data source: ConfigMgr module, site SQL database, ADSI, DNS/ICMP, admin shares |
|
|
| `Providers\Demo.ps1` | Simulated data source used by the demo (see below) |
|
|
| `Config\usmt-dash.psd1` | Every environment-specific setting |
|
|
| `web\index.html` | The dashboard (HTML/JS, about 4,600 lines) |
|
|
| `web\demo\demo.js` | Terminal panel, plus the demo-mode browser sandbox |
|
|
| `web\overview.js` | Overview page (issue counters, progress, locations, deployment calendar) |
|
|
| `web\device-info.js` | Shared device helpers: missing-data reasons, presence, snapshot markers |
|
|
| `web\excel-filter.js` | Excel-style sort / filter menus on every table; phone *Filter & sort* |
|
|
| `web\layout.js` / `layout.css` | Sticky headers, card view on phones and tablets, Custom View scrolling and column drag |
|
|
| `web\docs.js` | Knowledge base: permalinks, search, categories, PDF / print |
|
|
| `web\router.js` | Page addresses (`/status`, `/docs/...`) and back / forward |
|
|
| `web\jump.js` | Phones and tablets: jump to a device, a section, the top, middle or bottom |
|
|
|
|
### API
|
|
|
|
| Method & path | Purpose |
|
|
|---|---|
|
|
| `GET /` and static files | Dashboard, assets, `Data\` state files, `Documentation\` |
|
|
| `GET /tracked` · `/workbench` · `/status` · `/devices` · `/custom` · `/docs` | The dashboard, opened on that page (the logo and name go to `/`, the Overview) |
|
|
| `GET /docs/<title>` | The dashboard, opened on that knowledge-base article |
|
|
| `GET /get-config` | Mode flags for the front end |
|
|
| `GET /get-logs?source=server\|worker&after=<id>` | Tail of the service console output (terminal panel) |
|
|
| `POST /force-update` · `POST /force-full` | Wake the worker now (full = also refresh software/inventory caches) |
|
|
| `GET /get-blacklist` | Baseline-app pattern list |
|
|
| `GET /get-all-tags` · `GET /get-tracked?tech=` · `GET /get-tag-meta?tech=` | Read tracking tags |
|
|
| `POST /update-tracked` | Set a tag's devices and *Skip USMT* flag |
|
|
| `GET /get-software?name=` | Installed software for one device |
|
|
| `GET /get-docs-list` · `POST /save-doc` · `POST /delete-doc` | Knowledge base (delete archives to `Documentation\Archive`) |
|
|
| `GET /list-images` · `POST /upload-image` · `POST /rename-image` · `POST /delete-image` | Knowledge-base images |
|
|
|
|
## Demo mode
|
|
|
|
The public demo has to show the whole dashboard working without real infrastructure, and
|
|
without letting strangers change what other visitors see.
|
|
|
|
- **It's the real code.** The container runs the unmodified `USMT-Server.ps1` and
|
|
`USMT-Worker.ps1` under PowerShell 7 with `-Demo`. The worker loads `Providers\Demo.ps1`
|
|
instead of `Providers\SCCM.ps1`, and that's the only difference.
|
|
- **A believable fleet.** The demo provider derives each device's hardware, users, location,
|
|
department and software from a hash of its name. Each migration moves through every stage
|
|
on a 24-hour clock, so the dashboard keeps changing. Devices go offline, have stale DNS,
|
|
run low on disk, and fail backups at realistic rates.
|
|
- **Read-only API.** In demo mode the server refuses every route that writes (and
|
|
`get-software`, which takes free-text input) with `403`. The terminal output is the real
|
|
console of both processes, mirrored to `Data\Logs` by a `Write-Host` proxy.
|
|
- **Visitor changes stay in their browser.** `web\demo\demo.js` intercepts the dashboard's
|
|
write requests and keeps the visitor's tags, articles and images in IndexedDB, merged over
|
|
the server's data. Their actions print in the terminal panel exactly as the services would,
|
|
marked with a small dot. Nobody else sees them, and nothing a visitor types reaches the
|
|
server, so there's nothing to moderate or inject. **DEMO > Reset my changes** clears it.
|
|
|
|
Without `-Demo`, none of this applies: `demo.js` passes every request straight through and
|
|
the dashboard works against the real backend.
|
|
|
|
## Running it
|
|
|
|
### Demo (Docker / Coolify)
|
|
|
|
```bash
|
|
docker compose up -d --build # http://localhost:8080
|
|
```
|
|
|
|
On **Coolify**, create a resource from this repository using the *Dockerfile* (or *Docker
|
|
Compose*) build pack, set the exposed port to **8080**, and attach a domain. The container
|
|
reseeds the simulated fleet on every start and needs no volumes or secrets. Times are shown in
|
|
`TZ` (default `America/Chicago`); set it to your own timezone. The demo worker runs
|
|
6 AM to 9 PM every day (`Demo.BusinessHours` in the config) and sleeps overnight.
|
|
|
|
### Demo (local PowerShell 7)
|
|
|
|
```powershell
|
|
pwsh ./Demo/New-DemoData.ps1 -Force
|
|
pwsh ./USMT-Worker.ps1 -Demo # terminal 1
|
|
pwsh ./USMT-Server.ps1 -Demo # terminal 2 -> http://localhost:8080
|
|
```
|
|
|
|
### Live, against your environment
|
|
|
|
Requirements: Windows PowerShell 5.1, the ConfigMgr console (for the `ConfigurationManager`
|
|
module), read access to the site database, AD read access, and admin-share access to devices.
|
|
|
|
1. Edit `Config\usmt-dash.psd1`: site code, SMS Provider, SQL server and database, and (if
|
|
you extend hardware inventory with an asset/location class) the view and column map
|
|
under `AssetInventory`.
|
|
2. Start the worker, then the server:
|
|
```powershell
|
|
.\USMT-Worker.ps1 # add -v for per-device tracing
|
|
.\USMT-Server.ps1 # binding http://*:8080/ may need: netsh http add urlacl url=http://*:8080/ user=DOMAIN\account
|
|
```
|
|
3. Optionally restrict access by listing allowed IPs or hostnames in `Data\Whitelisted-IPs.txt`.
|
|
4. Put baseline-image apps in `Data\Blacklisted-Apps.txt` so only specialty software shows as
|
|
missing. See `Demo\Blacklisted-Apps.txt` for the syntax (plain patterns, `Suite = a*, b*`,
|
|
`Rule: DEVICE* = app*`, `Combo: Label = a* && b*`).
|
|
|
|
### Configuration
|
|
|
|
Everything lives in `Config\usmt-dash.psd1`; each setting is commented there. The ones you're
|
|
most likely to change:
|
|
|
|
| Setting | Default | What it does |
|
|
|---|---|---|
|
|
| `Worker.CycleSeconds` | `180` | Pause between cycles during business hours |
|
|
| `Worker.BusinessHours` | 07:30-17:30, Mon-Fri | When the worker cycles; outside it sleeps, with a midnight pitstop |
|
|
| `Worker.PresenceSweepMinutes` | `60` | How often completed and Skip USMT devices (which the sweep doesn't ping) are pinged |
|
|
| `Worker.LivePublishSeconds` | `10` | How often swept rows are published during a cycle |
|
|
| `Worker.SoftwareCacheHours` | `24` | How long a software inventory is reused |
|
|
| `Worker.DiskSkipList` | none | Devices whose admin share hangs; disk and log checks are skipped |
|
|
| `SCCM.AssetInventory` | `v_GS_ASSET_INFO0` | Optional custom inventory view for location, department, room type... |
|
|
| `SCCM.Monitors` | `v_GS_WMIMONITORID` | Attached monitors (needs WmiMonitorID in hardware inventory) |
|
|
| `Demo.BusinessHours` | 06:00-21:00, every day | When the demo worker runs |
|
|
| `Demo.DeviceCount` / `CycleSeconds` | `200` / `45` | Size of the simulated fleet and its cycle pause |
|
|
|
|
### Adapting it to a different toolset
|
|
|
|
All infrastructure access goes through a provider. To support another platform (Intune, a
|
|
different CMDB, PDQ, ...), add `Providers\<Name>.ps1`, implement these functions, and set
|
|
`Provider = '<Name>'` in the config:
|
|
|
|
| Function | Returns |
|
|
|---|---|
|
|
| `Initialize-Provider` | One-time setup (import modules, ...) |
|
|
| `Open-ProviderCycle` / `Close-ProviderCycle` | Per-cycle session (connections) |
|
|
| `pingDevice <name>` | `Online`, `Timed Out`, `Not Found` or `DNS Issue` |
|
|
| `Resolve-DeviceIP <name>` | Current IP, or `$null` |
|
|
| `Get-DiskSpace <name>` | `@{ Total = '238 GB'; Free = '80 GB' }` |
|
|
| `Get-LogInfo <name> <logName>` | Latest USMT progress log line (via the shared parser) |
|
|
| `Get-DeviceAssociations <name>` | Objects with `SourceComputer`, `DestinationComputer`, `MigrationStatus` (0/1/2) |
|
|
| `Get-DeviceInventory <name> <session>` | Hashtable of detail fields (OU, Groups, Model, RAM, MonitorCount, Monitors, Location, ...), plus `InAD` / `InSCCM` (`$true` / `$false`) |
|
|
| `Get-InstalledSoftware <name>` | Rows of `Name`, `Version`, `Publisher`, `Source`, `InstallDate` |
|
|
| `Get-DeviceLastSeen <name> <session>` *(optional)* | When the device was last active, used to backfill "last online" the first time it's tracked |
|
|
|
|
`Providers\Demo.ps1` is a complete working example that needs no infrastructure.
|
|
|
|
## Project layout
|
|
|
|
```
|
|
USMT-Server.ps1 web server + JSON API
|
|
USMT-Worker.ps1 polling worker
|
|
Get-Software.ps1 per-device software lookup + cache
|
|
Config/usmt-dash.psd1 configuration
|
|
Lib/Common.ps1 config loader, console-log mirror, shared I/O
|
|
Providers/SCCM.ps1 ConfigMgr / SQL / AD / SMB data source
|
|
Providers/Demo.ps1 simulated fleet
|
|
Demo/ demo seed script + sample app blacklist
|
|
Documentation/ knowledge-base articles and images
|
|
web/ dashboard, terminal panel / demo sandbox, vendored libraries
|
|
Dockerfile, docker/ container for the hosted demo
|
|
Data/ runtime state (created on first run, not committed): Data.json,
|
|
Worker-State.json, Advanced-Cache.json, Software-Cache.json,
|
|
Presence.json, Snapshots.json, Completed.json, Logs/
|
|
```
|
|
|
|
## Third-party code
|
|
|
|
Vendored in `web/vendor/` under their own licenses: [marked](https://github.com/markedjs/marked)
|
|
(MIT), [highlight.js](https://github.com/highlightjs/highlight.js) (BSD-3-Clause),
|
|
[html2pdf.js](https://github.com/eKoopmans/html2pdf.js) (MIT).
|