API Irritations

[Bob_Zimmerman]

I'm trying to do more with the management API, and it is insanely frustrating to deal with. Thought I would vent a little here.

First, something actually very good: the API is versioned. Version numbers within the API really ought to be part of the minimum standard before you can call something an API (along with an error schema!), but tons of vendors don't include these, and your tools just break when you update the product. Versions let you provide API stability to get around that issue.

Now for the not-so-good.

What is with all the hyphens in object keys? The only tool Check Point includes for processing JSON is jq, which intensely dislikes hyphens in keys. I don't know of any common programming language besides Lisp which tolerates hyphens in structure or object properties, which means you have to translate things between Check Point's JSON format and your tool's internal format constantly. It's an enormous headache. snake_case, camelCase, and PascalCase are all right there, and they avoid this problem entirely.

Why is nothing strongly typed?! Every object has a type, sure. The type is an immutable property of the object, sure. You could superclass hosts, networks, address ranges, and so on into a "checkPointEndpoint" class, then constrain source and destination fields in rules to only contain that class ... except then you run into Any. Check Point uses the same object for Any in source, destination, service, VPN, content, and time fields within rules. Again, the same object. It's a single UUID with the type CpmiAnyObject. So now your source and destination fields have to be able to contain a checkPointEndpoint object or a CpmiAnyObject object. Lists in strongly-typed languages don't like to be composed of more than one type, so to represent everything (including Any) with objects, you now have to make an even larger superclass. At that point, you pretty much have to make everything as a superclass so broad it can't constrain anything at build time. Runtime checks for everybody!

Remember how I mentioned the hyphens earlier? Notice the "CpmiAnyObject" above with no hyphens? That's right! Check Point mixes kebab-case with PascalCase in fields.

[Bob_Zimmerman]

Found another one. Some API endpoints are case-insensitive, while others (the specific one I hit was where-used) don't return anything for uppercase UUIDs. It's easy enough to just add a "toLowercase" on the end of the conversion from binary UUID to string, but the inconsistency is weird.

[Bob_Zimmerman]

I converted my code to use a single class for all objects, then switched to using 'show objects' to get everything.

Tags aren't included in 'show objects'.

Are you kidding me?



I'm also not a fan of the two personalities of access layers. When you feed a UUID to 'show access-layer', you get a bunch of properties like whether it's shared, and what features are enabled. Nothing about the rules, though.

When you feed the same UUID to 'show access-rulebase', you get the UUID, name, rulebase, objects dictionary (optionally), from, to, and total. No information about the layer personality of the object.

[Bob_Zimmerman]

Found a new one. I'm probably going to report this as a bug.

Access sections don't give you their position. They have a 'from' integer and a 'to' integer for the rules inside them, but no position designator for the section itself. If the section is empty, you don't get a 'from' or a 'to', you just get this:

{
  "uid" : "7c26ca40-7429-4b8b-b7d5-37b3f9aa6564",
  "name" : "Nobody here but us sections",
  "type" : "access-section",
  "rulebase" : [ ]
}

That was collected with "details-level full". What am I supposed to do with that? I can't even use the information I get to recreate the section on another SmartCenter, since you are required to provide a position when you create a section. Now I have to maintain the rulebase as an ordered list rather than just a set.

[jflemingeds]

Me thinks out of the 10-15 active users here there are maybe 2 that do programming stuff and 1 that is using the API right now.

That being said i'm all ears.

[Bob_Zimmerman]

Me thinks out of the 10-15 active users here there are maybe 2 that do programming stuff and 1 that is using the API right now.

That being said i'm all ears.

Entirely possible. That said, if somebody else wants to build tools like the ones I build, this might help them avoid some of the data model potholes I've hit. It took me days to convert from a differentiated endpoint class and service class to a unified "object" class with runtime checks to deal with how Check Point uses the same "Any" for both. Super tedious.

Now it's increasingly looking like I'll have to do the same with rules, rule sections, and layers. Now, a policy package references zero or more access layers, an access layer contains rules or sections, a section contains rules (but not other sections!), and a rule ... can contain a layer. But it doesn't contain the rules in that layer, just a reference to it.

[Bob_Zimmerman]

Just ran into a more pleasant surprise! 'show object' appears to work with any UUID. Object, policy package, layer, even individual rules. I noticed when I made a mistake handling inline layers and built a new layer object with the rule's UUID instead of the rule's inner layer's UUID.

This confirms my suspicion that everything is internally just an "object", and the type checking to prevent you—for example—from putting a service object in the source of a rule is run-time rather than compile-time. Less efficient, as you need to do lots of tests before an operation, but it does make certain aspects of the code much simpler for the developer.

[Bob_Zimmerman]

And back to hair-pulling frustration.

If you run 'show objects', and you get a group, that group's members are given as a list of UUIDs.

If you get the same group via 'show object', the group's members are given as a list of full objects.

But wait! If one of the members of that group is another group, this inner group's members are given as a list of UUIDs!

[Expert@SmartCenter]# mgmt_cli -r true show object uid "2a469820-b502-434c-9340-a377677a6a60" --format json details-level full
{
  "object" : {
    "uid" : "2a469820-b502-434c-9340-a377677a6a60",
    "name" : "CIFS",
    "type" : "service-group",
    ...,
    "members" : [ {
      ...
    }, {
      "uid" : "97aeb471-9aea-11d5-bd16-0090272ccb30",
      "name" : "NBT",
      ...,
      "members" : [ "97aeb414-9aea-11d5-bd16-0090272ccb30", "97aeb415-9aea-11d5-bd16-0090272ccb30", "97aeb416-9aea-11d5-bd16-0090272ccb30" ],
      ...
    } ],
    ...
  }
}

Edited to add: Also weird, the schema for 'interfaces' totally changes depending on how you look at a firewall object. Wildly different keys and data. In 'show objects' or 'show object', it looks like this:

{
  "from" : 3001,
  "to" : 3500,
  "total" : 8960,
  "objects" : [ ...
  }, {
    "uid" : "4d25f6a9-e99a-ce43-a6ce-27a8280d918f",
    "name" : "SmartCenter",
    "type" : "simple-gateway",
    ...,
    "interfaces" : [ {
      "name" : "eth1",
      "ipv4-address" : "10.20.30.40",
      "ipv4-network-mask" : "255.255.255.0",
      "ipv4-mask-length" : 24,
      "ipv6-address" : "",
      "comments" : "",
      "color" : "black",
      "icon" : "NetworkObjects/network",
      "topology" : "automatic",
      "topology-automatic-calculation" : "internal",
      "topology-settings" : {
        "ip-address-behind-this-interface" : "not defined",
        "interface-leads-to-dmz" : false
      },
      "anti-spoofing" : false,
      "security-zone" : false
    } ],
    ...
  }, {
  ...
  } ]
}

but in 'show gateways-and-servers', it looks like this:

{
  "objects" : [ {
    "uid" : "4d25f6a9-e99a-ce43-a6ce-27a8280d918f",
    "name" : "SmartCenter",
    "type" : "simple-gateway",
    ...,
    "interfaces" : [ {
      "interface-name" : "eth1",
      "ipv4-address" : "10.20.30.40",
      "ipv4-network-mask" : "255.255.255.0",
      "ipv4-mask-length" : 24,
      "dynamic-ip" : false,
      "topology" : {
        "leads-to-internet" : false,
        "ip-address-behind-this-interface" : "not defined",
        "leads-to-dmz" : false
      }
    } ],
    ...
  } ],
  "from" : 1,
  "to" : 1,
  "total" : 1
}

[Bob_Zimmerman]

'show changes' is so close! It provides enough information to highlight items which were changed. Unfortunately, it doesn't provide enough to actually merge those changes from just the 'show changes' result. Specifically, it doesn't include any information about rule positions or what sections they might be in.

It's probably fine for objects, but unfortunately won't work for what I want to do with rules.

[Bob_Zimmerman]

'show object' returns a JSON structure with a top-level key of "object" which has its value set to the JSON structure for the object you are trying to get:

[Expert@mySmartCenter:0]# mgmt_cli -r true show object uid "66b71dc2-ed6a-4743-b02a-67254c39ca4e" --format json
{
  "object" : {
    "uid" : "66b71dc2-ed6a-4743-b02a-67254c39ca4e",
    "name" : "Test Host",
    "type" : "host",
    "domain" : {
      "uid" : "41e821a0-3720-11e3-aa6e-0800200c9fde",
      "name" : "SMC User",
      "domain-type" : "domain"
    },
    "ipv4-address" : "10.20.30.40"
  }
}

'set <type>' returns a JSON structure for the object you are trying to get directly. No 'object:' wrapper. This caused my result processing to crash the first time I tried to write something. I told it to decode the "object" key, and there wasn't one. Simple enough to handle, but it's another weird inconsistency.

Edited to add: Turns out 'show <type>' behaves the same way as 'set <type>', so I suspect this is just 'show object' running things through a generic wrapper.

[Bob_Zimmerman]

Ran into the group-members-are-sometimes-objects-and-sometimes-UUIDs thing again, but this time with tags. I suspect this inconsistency will bite me a few more times before I've tracked down all the places it can happen.

The fix was actually simple enough after I thought about it for a minute. I declare my members (or tags) property to be a list of UUIDs. I try to decode the data I get from the API as a list of UUIDs. If it works, I assign that list directly to the property. If it doesn't, I retry the decoding as a list of whole objects. I then use a map operator to extract only the UUIDs and I assign that list to the property.

[nightcrawler]

hi Zimmi, thanks for this,
by looking fo any hints, why my customer gets an exception when using the details-level: full level on a r81 jhf 44
it's good to know, that nothing is perfect... as it is often .....
but no good when reliying on standard behavior :-)

thanks for sharing

Gero

[Bob_Zimmerman]

hi Zimmi, thanks for this,
by looking fo any hints, why my customer gets an exception when using the details-level: full level on a r81 jhf 44
it's good to know, that nothing is perfect... as it is often .....
but no good when reliying on standard behavior :-)

thanks for sharing

Gero

Huh! I haven't had that problem yet. Hit something similar on R81.10, which wound up being a bug somewhere in the management server or API service. Try using the script in the first post to find the block of objects which triggers the issue, adjust the 'seq 0 500 11068' part to find the object (0 is the starting number, 500 is the stride length, 11068 is the ending number), then see if you can identify that object with a lower detail level.

[Bob_Zimmerman]

It's been a little while!

Today, I found that you can't both turn a policy package's "access" property to true and set its access layers in one API call. The API just barfs with an extremely unhelpful "Internal error. For more info search for incident [UUID] in log file" message. The log gives you a call trace, but no indication that the problem is it wants you to separate the two actions.

So now I have to track some booleans when the policy package detail view is opened and prevent the user from editing access layers and threat layers in the same change they used to enable those features.