What's up?

Been getting the occasional error for certain unlucky enrollments with certain Mdm-Signature.

err=ber2der: BER tag length is more than available data

When this happens, PUT /mdm returns a 400 response.

The fix?

Not sure, but after upgrading the PKCS7 library, replaying the same request works. Maybe it's this? https://github.com/mozilla-services/pkcs7/pull/61

Regardless, reproduced that error happens in pkcs7.Parse() in main and upgrading the library solved it.

PostgreSQL storage added. Based on mysql storage package. Much of the code is duplicated, but as Go-authors said, sometimes a little more copy-paste is better than a little more dependencies. github.com/lib/pq - is used as db driver.

Hello!

When handling the command result response for InstalledApplicationList, nanomdm issues the following plea:

level=info handler=checkin-command msg=command report results err=storing command report: Error 1406: Data too long for column 'result' at row 1

The InstalledApplicationList response for my current device - with only base Apple apps - is 117,465 characters of base64-endcoded text. TEXT support 65,525 characters; MEDIUMTEXT, 16,777,215 (this corresponds to ~16.78 mb: "L + 3 bytes, where L < 2^24" - https://dev.mysql.com/doc/refman/8.0/en/storage-requirements.html#data-types-storage-reqs-strings)

This works with current nanomdm release (in as much as I migrated with schema.00005.sql and everything continued to work/InstalledApplicationList started working). MySQL docs do not have any warnings regarding migrating between text types that I found.

Adds a simple ctxlog package which permits associating multiple CtxKVFuncs with a context. Then, when it's time to log, these are ran using the request context to pull out logging key-value pairs and create a new logger.

The end result is something like this for a typical NanoMDM push request.

Before:

2022/03/16 14:03:09 level=info handler=log addr=::1 method=GET path=/v1/push/D7083AF3-DCCB-5661-A678-BCE8F4D9A2C8 agent=curl/7.64.1
2022/03/16 14:03:09 level=info service=push msg=retrieved push cert topic=com.apple.mgmt.External.deadbeef-1f17-4c8e-8a63-dd17d3dd35d9
2022/03/16 14:03:09 level=debug handler=push msg=push count=1 errs=0
2022/03/16 14:03:09 level=info handler=log addr=192.168.0.2 method=PUT path=/mdm agent=MDM-OSX/1.0 mdmclient/1455 real_ip=2301:602:8a02:9b0f:422:25a8:481b:bd42
2022/03/16 14:03:09 level=info service=nanomdm status=Idle id=D7083AF3-DCCB-5661-A678-BCE8F4D9A2C8 type=Device
2022/03/16 14:03:09 level=debug service=nanomdm msg=no command retrieved id=D7083AF3-DCCB-5661-A678-BCE8F4D9A2C8

After:

2022/03/16 14:02:44 level=info handler=log trace_id=84172eff419691ca addr=::1 method=GET path=/v1/push/D7083AF3-DCCB-5661-A678-BCE8F4D9A2C8 agent=curl/7.64.1
2022/03/16 14:02:44 level=info service=push trace_id=84172eff419691ca msg=retrieved push cert topic=com.apple.mgmt.External.deadbeef-1f17-4c8e-8a63-dd17d3dd35d9
2022/03/16 14:02:44 level=debug handler=push trace_id=84172eff419691ca msg=push count=1 errs=0
2022/03/16 14:02:44 level=info handler=log trace_id=c35015253617eb34 addr=192.168.0.2 method=PUT path=/mdm agent=MDM-OSX/1.0 mdmclient/1455 real_ip=2301:602:8a02:9b0f:422:25a8:481b:bd42
2022/03/16 14:02:45 level=info service=nanomdm trace_id=c35015253617eb34 id=D7083AF3-DCCB-5661-A678-BCE8F4D9A2C8 type=Device status=Idle
2022/03/16 14:02:45 level=debug service=nanomdm trace_id=c35015253617eb34 id=D7083AF3-DCCB-5661-A678-BCE8F4D9A2C8 type=Device msg=no command retrieved

Note the trace_id value which tracks each log's request from its origin. Note also that id and type in the service=nanomdm layer is also a context value rather than explicitly logged now.

Hi, there's been a lot of improvements since the last release in June 2021.

As much as we can, we'd like to rely on official releases. Was wondering:

1. Can we get a release cut? 😸
2. Maybe it's good to discuss what do releases mean for NanoMDM and if it makes sense to have a certain type of cadence of cutting releases?

e.g. I can imagine if there was a time or feature based release schedule that followed some level of semantic versioning in spirit, we would try our best frequently update to the nearest minor version and run some form of QA process before a major / breaking version change.

Optionize stdlogfmt package/New, add filename and line logging, use timestamp format.

Output looks like:

ts=2022-07-21T08:41:40-07:00 level=info msg=storage setup storage=pgsql caller=cli.go:60
ts=2022-07-21T08:41:40-07:00 level=info msg=storage setup storage=mysql caller=cli.go:60
ts=2022-07-21T08:41:40-07:00 level=info msg=storage setup storage=file caller=cli.go:60
ts=2022-07-21T08:41:40-07:00 level=info msg=storage setup storage=multi-storage count=3 caller=cli.go:93
ts=2022-07-21T08:41:40-07:00 level=debug msg=declarative management setup url=http://127.0.0.1:5000/ caller=main.go:102
ts=2022-07-21T08:41:40-07:00 level=info service=certauth msg=allowing retroactive associations caller=certauth.go:95
ts=2022-07-21T08:41:40-07:00 level=info msg=starting server listen=:9000 caller=main.go:208

In lieu of parsing responses for push_error etc, I would prefer to rely on the HTTP code to judge success. This is a bit awkward when enqueuing commands for multiple IDs, as not all may have failed. I argue to that one failure should return 500 for the entire operation - if the client sent multiple IDs, it is the client's responsibility to read the json response and discover the precise error.

I didn't get fancy with specific error codes, willing to though.

I'm a go noob, so I apologize for any glaring mistake, bad pattern, etc. Please let me know!

Found the two ngrok urls to be hard to grok :), have to keep going back to see if that was the SCEP one or nanomdm one. So I thought this adding the prefix makes it easier to follow.

Also, ngrok supports multiple tunnels on one process on the free plan, but it's easy to overlook. Added a line on configuring multiple tunnels.

By default NanoMDM will save the client identity certificate for a device enrollment during the Authenticate check-in message (for both the file and mysql backends). However if enrollments happened in such a way that client certificates were missed (say, via migration) then we should support a way to "recapture" client certificates in storage by saving the identity certificate. It would be wasteful to do this on every request so perhaps it would be only be turned on with a flag.

Currently the MySQL backend is not tested, as it requires MySQL server to run. I have added an integration test to the GitHub workflows which starts MySQL server in a container, which allows the backend to be tested. This ignores several "issues" that exist with the tests, such as the fact that the test is non-idempotent, and fails to run twice.

Hi! I'm new to nanoMDM, I was trying to set it up for testing and followed the steps on quickstart.md. When the server was up and running i opened safari on my iphone and went to '{my_local_ip}/mdm' to enroll the device, and then the server responded with a bad request and the logs look like this

2022/09/07 08:49:02 level=info handler=log addr=192.168.0.90 method=GET path=/mdm agent=Mozilla/5.0 (iPhone; CPU iPhone OS 15_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6.1 Mobile/15E148 Safari/604.1
2022/09/07 08:49:02 level=debug handler=cert-extract msg=empty Mdm-Signature header
2022/09/07 08:49:02 level=info handler=cert-verify msg=error verifying MDM certificate err=missing MDM certificate

Am I doing something wrong?

The command I'm running is ./nanomdm-darwin-amd64 -ca ca.pem -api nanomdm -debug

depends on #58

Currently the delete statement in the MySQL backend is a complicated multiple-table statement, which is a proprietary MySQL extension to the SQL language. Furthermore, we have observed issues with this delete statement deadlocking with itself (when two statements run for the same command row) due to lock-upgrading issues that occur between the query and delete steps.

I have changed this DELETE statement to be an ordinary delete, with two simple select statements (with no joins). These select statements may not have to be annotated FOR UPDATE, but I wanted to err on the side of avoiding deadlocks.

Testing:

• Observe that MySQL integration tests pass TODO:
• Make sure that deadlocks no longer occur.

The view_queue is not materialized, and thus functions as syntactic sugar for a complex query. Furthermore, our infrastructure works poorly with views. Therefore, this usage adds a significant maintainership burden for little gain.

This commit removes the view from NanoMDM code paths, but it still exists in the schema. This is purely for convenience and interactive users.

Furthermore, I added an index used by the queue query, which should have existed for the view. This should improve performance.

Testing:

• Tested migration - see the schema migration script
• Tested base functionality
• Passes new integration test

depends on #58 fixes #53 for mysql backend

Enabling mongoDB storage support and integrating it into the cli startup capabilities

Note: The feature TODOs are not mission critical to the daily operation

Happy to support as necessary going forward 😀

As discussed in #3 would like to move away from direct VIEW usage for the command queue in the storage backend. We can keep it in the schema.sql files as a helper for folks, but would like to move to direct table query in the code.

The MySQL backend:

https://github.com/micromdm/nanomdm/blob/9ca3cacf2bb5deb8c2d9cc9d69985ebec0f78176/storage/mysql/queue.go#L148

The Postgres backend:

https://github.com/micromdm/nanomdm/blob/9ca3cacf2bb5deb8c2d9cc9d69985ebec0f78176/storage/pgsql/queue.go#L162

Since there is already a Dockerfile in the repo and its actively using GitHub Actions, it would be great to publish a container to Docker Hub or the Github Container registry for easy access. Personally it would make creating a helm chart much easier if I don't have to build this locally.