← Back to blog

Spark FHIR Server 3.2.0

Spark FHIR Server 3.2.0 adds support for FHIR R6 ballot 3 next to the existing STU3, R4, R4B and R5 capabilities. The other large piece of work in this release is new infrastructure for database migrations. This lets us change how the search index is stored without breaking existing databases, and we use it to normalize how token, quantity and reference values are indexed.

Index rebuilds are now considerably faster on large databases, since they no longer slow down as they progress through the result set. This is useful when running the migrations, which require a full index rebuild.

This release also expires old search snapshots, adds the _source search parameter, and includes several fixes around batch/transaction bundles, PATCH, and index rebuilds.

Existing databases should be migrated after upgrading. See Upgrading From 3.1 at the end of the post.

Contributors to this release include Ole Kristian Losvik, wardzinskim and Kenneth Myhra.

Questions or feedback? Get in touch with us on our Discord server: https://discord.gg/M9kp2zGGYF.

What's New in 3.2.0

1. FHIR R6 (Ballot 3) Support

PR: #1392

Spark now includes a dedicated R6 application. Note that this targets the R6 ballot 3 version, not a final R6 release. It follows the same pattern as the other versions: an R6 web application, frontend, Docker image, and the version-specific NuGet package Spark.Engine.R6 on top of the shared Spark.Engine and Spark.Store.MongoDB packages.

The following R6 resource types are not supported yet: ClinicalAssessment, DeviceAlert, InsuranceProduct, MolecularDefinition and PersonalRelationship. These are being discussed in firely-net-sdk#3573.

2. Database Migrations

PRs: #1398, #1399, #1401, #1426, #1411

Spark now records database migrations in the schema_migrations collection in MongoDB. The migration state is read when the server starts and refreshed every 30 seconds, and it decides how search queries are built. New databases are created at the current migration version.

This release includes two migrations of the search index:

  • structured-string-token-index: Token search parameters that resolve to plain string values are now stored in the same structured shape as other tokens, { "code": "value" }, instead of as a plain string.
  • token-quantity-and-reference-array-index: Token, quantity and reference values are now always stored as arrays, even when a resource has only one value. Previously they were stored as a single object or an array depending on the number of values.

Once both migrations are recorded, token, quantity and reference-identifier searches no longer need compatibility branches for the old storage formats and can use a single $elemMatch query. Until then, Spark keeps querying both the old and the new formats, so searches continue to work while a database is being migrated.

For library consumers, IElementIndexer2 has been added with a Map(Element, SearchParamType) method, which lets indexers know the type of the search parameter being indexed. IElementIndexer is unchanged, so existing implementations remain binary compatible.

The migrations are described in more detail in Database Migrations.

3. Expiring Search Snapshots

PRs: #1430, #1431

Every search stores a snapshot of its matches so that clients can request the next page. Until now these snapshots were never removed. Snapshots are now expired after one hour by default, configurable through StoreSettings:SnapshotStore:RetentionSeconds.

Requesting a page from a snapshot that has expired returns 410 Gone.

4. _source Search Parameter

PR: #1384

Spark now supports the general FHIR search parameter _source.

Improvements And Bug Fixes

1. Correct Responses for Batch and Transaction Bundles

PR: #1370

Batch and transaction requests now return the correct bundle response type.

2. Stale Versioned PATCH Returns 409 Conflict

PR: #1371

A versioned PATCH against a version that is no longer current now returns 409 Conflict.

3. Search Parameter Definitions

PRs: #1372, #1373, #1376

Search parameters are now resolved from the SearchParameter's Expression instead of its XPath, the _id search parameter now has type token, and the generic search parameter definitions can be overridden per FHIR version.

4. Re-indexing No Longer Fails With Duplicate Keys

PR: #1409

Rebuilding the index failed for every resource with a duplicate key error unless ClearIndexOnRebuild was set to true. The latest version of a resource can now be re-indexed in place.

5. Faster Index Rebuilds on Large Databases

PR: #1440

Index rebuilds used to slow down the further they got into the result set, since each page re-issued the query and skipped the entries already indexed. Rebuilds now iterate with a MongoDB cursor and keep the same pace throughout.

6. Better Logging During Index Rebuilds

PRs: #1394, #1395

Index rebuild progress and failures are now logged by the server, and failure messages shown in the admin UI are more descriptive.

PRs: #1381, #1404, #1427, #1428

  • Entries that contain only the root index value are no longer written to the search index.
  • Resources are only deep-copied during indexing when they are a DomainResource.
  • SearchService no longer modifies the SearchParams passed in by the caller.
  • Snapshot.CreateCountOnly is now public.

8. Web Application Clean-up

PRs: #1388, #1393

Files that were duplicated between the Spark.Web.* applications are now shared in Spark.Web.Shared. The log output in the admin UI no longer overflows, scrolls to the latest message, and only marks errors in red.

API Changes

  • The default constructor of Criterium and Criterium.AddValue() are now obsolete (#1425).
  • IElementIndexer2 has been added, see Database Migrations.

Testing and Tooling

  • Integration tests now run for R5, and all FHIR versions run in parallel in a single workflow (#1378, #1380).
  • A performance testing framework has been added (#1432).
  • All tests now use xUnit on Microsoft Testing Platform, and integration tests share a single MongoDB container (#1391, #1416, #1418, #1419).
  • The solution file has moved from Spark.sln to Spark.slnx (#1420).
  • Database backups used as fixtures have moved out of the repository to spark-database-fixtures and are downloaded by .docker/linux/download-database-fixtures.sh (#1433).

Upgrading From 3.1

New databases need no action. For an existing database, run the migrations once after upgrading by following the steps in Existing Databases.

Spark keeps supporting the old index format until the migration has been recorded, so searches keep working before the migrations are run.