SmartDashboard on macOS

[Bob_Zimmerman]

No screenshot for this update, as it's "just" fixes for stuff I showed off in the last one.

Updated my policy package objects to have an ordered relationship to access layers, updated the rest of my code to deal with that, got rearranging working, and got the HTTPS inspection layer picker working.

With these changes, I now have the ability to create new policy packages and edit existing ones. To make my life easier, I only let the user make changes to an object which are allowed to be made in a single API call. This isn't a great situation, but I don't yet want to deal with making multiple API calls in response to a single user action.

I now have:

• Object and rule download
• Creating new objects for most types
• Search for objects
• Editing existing objects for most types
• Deleting objects
• Drag-and-drop from object list into rules and from rule field to rule field for sources, destinations, and services
• Removing objects from source, destination, and service fields
• Enabling/disabling rules and toggling field negation
• Changing rule actions, including inline layers
• Publishing and discarding sessions
• Adding access rules or sections
• Rearranging access rules or sections
• Editing access rule or section names
• Deleting access rules or sections
• Adding NAT rules or sections
• Rearranging NAT rules or sections
• Deleting NAT rules or sections
• Pushing policy
• The ability to create and edit some objects which reference other objects (groups, services-groups)
• Adding new policy packages and access layers
• Editing policy packages and access layers

I do not yet have:

• Where Used
• The ability to edit some objects which reference other objects (Access-roles, etc. I don't use them, so I don't want to build that until I'm familiar with how they work.)
• Editing most NAT rule fields (I can toggle enabled/disabled, and hide/static source NAT)
• Search for rules
• Any sort of view of progress for in-flight tasks
• Any sort of change counter to let you know how many things you are about to publish or discard
• Deleting policy packages or access layers
• Anything to do with Threat Prevention
• Anything to do with logs
• Any support for Provider-1

Found out that when you delete an access layer, the rules under it are deleted. When you delete a policy package, the access layers it uniquely references are also deleted. At some point, I need to try building a shared layer, adding it to one single policy package, and deleting that policy package. I wonder if the management will delete the shared layer.

Deleting stuff mostly works, but it's up to the user to confirm something isn't used before trying to delete it. I don't yet provide a Where Used, so I guess that's my next development focus. On the plus side, if you try to delete something which you can't, it fails. The API returns an error (Thank you, Check Point! Way too many APIs go "HTTP code: 200, response body: 'Task failed!'"), which I catch, so I don't remove the object from my local database. I don't yet show the error to the user.

[Bob_Zimmerman]

Every time I think my day job is settling down a little, there's some new crisis. As a result, I haven't been able to spend as much time on this project as I'd like. I have also run into a performance issue, which has gotten a lot worse as I've added columns to the rule view.

Outline views on macOS like the one I'm using to show rules inherit from table views. Table views on macOS have some really complicated performance tricks internally, because they had to perform well on computers from 1989 (today's macOS descends directly from NeXTSTEP). Some of the things I've been doing don't work well with these tricks. On even basic policies, I'm getting animation hitches for up to 700 ms when scrolling the rules. That's not acceptable.

Nobody else seems to be using quite the constellation of features I'm trying to, so there isn't any documentation on how to use them all together. Currently working on a few different strategies.

[Bob_Zimmerman]

Still chasing the performance issues when scrolling rules. I've found four or five ways to do it which don't improve the situation at all. Still looking for a method which does. Might burn a developer tech support incident on it.

I've trimmed a lot of redundant code. I'm down over a thousand lines since May, even with some new functionality.

And speaking of new functionality, I've added a recurring timer which calls the API connection keepalive. Took way too long to figure out how to get the timer to fire reliably, but I finally have it. This is particularly important, as it relates to a big functional limitation: I didn't have a way to recognize when someone else locks an object after I've started a session. The recurring timer is the first step towards solving that. Now, I need to figure out how to efficiently get a list of things someone else has locked. It's relatively simple to find the sessions with locks, but frustrating to go from those sessions to the things they have locked. I'm still working on doing that well, particularly with rules.

Totally unrelated to this client application, I think I'm about to start looking for a new job. I'm an expert with VSX and I'd say pretty decent with the management API and GAiA API. If you might be interested in hiring me, let me know.

[Bob_Zimmerman]

I've had some pretty significant life changes this year, which have left me with less time than I would like to program. When I do have time, I often don't have the focus to really think about the more complicated problems I'm trying to solve. Hopefully things have settled down and the next few months will be more productive.

Major progress on scrolling performance. Apple has great tools for measuring UI performance, and animation hitches in particular. I've gone from 785ms max, 327ms mean, 279ms median animation hitches when scrolling to 326ms max, 173ms mean, 175ms median. Still not great scrolling performance, but it's acceptable for now. Since I had to rework big chunks of how I show rules, I also generalized how I'm handling my rule views. This will make it much easier to add HTTPS, Threat Prevention, and so on later.

The next big challenge is to import and work with Tasks. That part of the management API is kind of a mess. The different types of task have different fields in them, but there's not a consistent field I can use to identify how to decode the rest of the object. Tasks returned by some calls show up in the general 'show tasks' call, but others don't. 'show changes' is one of the tasks which doesn't show up in the general call, and that's the one I most care about handling right now. I have a few ideas on how to tackle all this, but it's going to be complicated.

I'm currently at 14891 lines as written, 12872 without blanks and block comments, across 61 Swift files (not counting XIBs). 743 commits so far. 13.1 MBs ready to run on aarch64 and amd64, 2.7 MB zipped.

[Bob_Zimmerman]

This week, I was able to use some of the internal functionality I developed for the generalized rule views to add a neat user-visible feature:



I've had the layers and policies listed in the objects sidebar for a while. Today, I figured out how to add a menu item to their contextual menus to hand them to the rule view. Only functional for access layers and policy packages (to view their NAT rules) right now, but enabling it for HTTPS layers and Threat Prevention layers will be trivial once I have those views built.

Still figuring out a good way to deal with tasks, especially of the 'show changes' type.

[Bob_Zimmerman]

Still dealing with a relative with dire health issues on top of day job stress. Fortunately, with the end of the year, I've been able to take some time for myself and unwind a bit. I've even made some progress on this application.

55 commits since my last post.

• Added a view for HTTPS Inspection layers.
• Hooked it up to the contextual menu as described in my last post.
• Fixed several internal bugs related to keeping the UI updated as the underlying data changes.
• Added a member count for various types of object such as groups and service groups.
• Added a badge on object icons to show lock status. It shows up everywhere the icon does.
• Added an editor for to-one relationships. For example, NAT rules can only have one object in each field.
• Made a consistent detail view and editor for rules and rule sections. Together with the to-one relationship editor, the detail view can edit NAT rule fields.
• Found a few API bugs and limitations. I've been able to work around some, but not all.

In that screenshot, you can see the member count for groups, policy packages (how many layers they contain), and layers (how many top-level rules and sections they contain). On the right, you can see the current access rule detail view.

19034 lines as written, 16186 lines without blanks and comments, across 139 Swift files. 17.1 MB built for aarch64 and amd64, 3.4 MB zipped.

[Bob_Zimmerman]

40 commits since my last post.

• Cleaned up a lot of duplicated code related to my scrolling performance improvements.
• Improved how my to-many relationship editor handles special values like "Any" and "None". They now show up with radio buttons, and they work how you expect.
• Fixed a few data import bugs revealed by some API changes on Check Point's part. For example, some objects without a color newly report their color as "none", but that hasn't been a valid value before, so it broke my import code.
• Added the ability to import VPN community objects and edit the API-accessible properties.
• Added the ability to edit the VPN field of rules now that I have VPN community objects.

19543 lines as written, 16654 lines without blanks and comments, across 155 Swift files. 18.4 MB built for aarch64 and amd64, 3.6 MB zipped.

[Bob_Zimmerman]

25 commits since my last post.

• Added the ability to create and edit Group with Exclusions objects.
• Added the ability to edit an object's tags.
• Made some tweaks to how I display non-editable properties like the UUID to make it more obvious they are not editable.
• Refactored how I convert objects into JSON representations suitable for sending to the API. It's now both simpler and much more robust.
• Related to the above refactoring, I added a button to my detail view to copy the object as JSON.
• Added the ability to create and edit Time Group objects. Still working on Time objects.
• I learned NAT rules can have names, even though SmartConsole doesn't show them. Added the ability to edit their names.

20376 lines as written, 17209 lines without blanks and comments, 685 kB, across 191 Swift files. 19.2 MB built for aarch64 and amd64, 3.8 MB zipped.



For an example of the "Copy JSON" button output, here's the object from the screenshot:

{"include":"97aeb369-9aea-11d5-bd16-0090272ccb30","type":"group-with-exclusion","name":"Not RFC 1918","tags":[],"except":"395a8003-bbf1-40ee-b8d5-f5a0f35d67b1"}

[Bob_Zimmerman]

60 commits since my last post. +4921, -3947 lines. Not a lot of user-visible stuff in this one.

• Adopted structured concurrency (async/await) and removed several completion handler pyramids
• Split a bunch of large, unwieldy functions into smaller ones
• Deduplicated a lot of smaller functions by making them more generic - This should let HTTPS Inspection (and later, Threat Prevention) rules be rearranged much like access and NAT rules.
• Used the more generic versions of the functions to make them more testable
• Added a bunch of unit tests, particularly around rule rearranging
• Found out how to get LLVM to report code coverage for my test suite
• Used the code coverage information to add test cases for a few error conditions I had missed testing before
• Used the tests to fix several bugs in rule rearranging

20653 lines as written, 17375 lines without blanks and comments, 707 kB, across 202 Swift files. 19.9 MB built for aarch64 and amd64, 3.9 MB zipped.

Linter warnings are way down. I have about 25 force-downcasts I still need to fix, but almost everything else SwiftLint doesn't like about my code is stuff intrinsic to Check Point's management API. For example, SwiftLint wants identifiers to be between 3 and 40 characters long. Check Point's API has a bunch of "to" keys (too short), plus a keep-connections-open-after-policy-installation key (too long) for several services. I could change my identifier names to be within SwiftLint's desired range, but then they wouldn't line up as well with the API when debugging issues. So those warnings will stay disabled.

Still working on ways to deal with Task objects to provide feedback on long-running stuff like policy pushes.

[Bob_Zimmerman]

Only 14 commits since my last post, but they have been some BIG ones. +13069, -2179 lines.

I've added 41 UI tests. These use Apple's accessibility tools to let you script operation of the UI, like using the application via VoiceOver. This means they test a bunch of stuff, which gives great coverage per test. The downside is they are extremely slow. My current test suite is 154 unit tests which take a little under 2.5 seconds to run in total, and 41 UI tests which take a little under 18 minutes to run in total. I can't use my machine for anything else while the UI tests are running (after all, they're moving the mouse cursor around and typing), so I mostly run them at the end of the day. The unit tests cover 17% of my code, and the full test suite (unit tests plus UI tests) covers 72.1%.

A lot of the code I'm not currently testing deals with making API calls. I've been working on setting up a disposable management server I can use to test things like that. Looks like I found a bug in Check Point's cloud-init while building this. Should be able to move forward soon enough, though. This should also help me iterate more quickly on my work to deal with Task objects.

I've used this increased test coverage to identify and fix some bugs in my code for data import and adding sections. For example, when adding a section inside another section, I had an off-by-one error causing a rule to be mistakenly left out of the new section.

The biggest user-visible change in this version is data validations for IPv4 (addresses, masks, networks, and multicast addresses), IPv6 (addresses, masks, networks, and multicast addresses), and Check Point port strings (>1023, 49152-65535, and so forth). The validation state of a given property in an object is shown in the field. If the object isn't valid (like you're building a TCP service with the port string "<1023-27425"), the button to update the object is disabled, and the field gets a red box with a white exclamation point in it.

20871 lines as written (excluding tests), 17569 without blanks and comments, 706 kB, across 203 Swift files. 14.5 MB built for aarch64 and amd64, 3.4 MB zipped.

For context, my unit tests are currently 8565 lines, 285 kB, across 93 Swift files, and my UI tests are currently 1642 lines, 81 kB, across 38 Swift files.