Unstructured Data Quick Tips

A cluster’s files and directories can be accessed programmatically, and controlled by filesystem permissions, through the OneFS RESTful Access to Namespace (RAN) API, similarly to the way they’re accessed through the core NAS protocols such as NFS and SMB.

Within the RAN namespace, the following system attributes are common to directories and files:

In response, the following response headers may be returned when sending a request to RAN.

For diagnostic and troubleshooting purposes, failed requests to the namespace can often be resolved via common error codes and viewing activity logs. Activity logs capture server and object activity and can help identify problems. The following table shows the location of different types of activity logs.

For <server> above, the path to the server directory should be used. For example: /apache2.

The common JSON error is returned in the following format:

{
"errors":[
{
"code":"<Error code>",
"message":"<some detailed error msg>"
}
]
}

The following table includes the common error codes, plus their status and description:

For example, an invalid copy source path yields the ‘AEC_BAD_REQUEST’ code:

# curl -X PUT --insecure --basic --user <name>:<passwd> --header "clone=true" --header "x-isi-ifs-copy-source:/namespace/ifs/data-other/testfile1/" https://10.1.10.20:8080/namespace/ifs/data/testfile1/
{
"errors" :
[
{
"code" : "AEC_BAD_REQUEST",
"message" : "Unable to open object '/data-other/testfile1/' in store 'ifs' -- a component of the path is not a directory."
}
]
}

When crafting straightforward HTTP requests to RAN, such as create a file (object), the ‘curl’ CLI utility can be a useful asset:

# curl -X PUT --insecure --basic --user <username>:<passwd> -H "x-isi-ifs-target-type:object" https://<cluster_ip>:8080/namespace/<path>/<file>/

For example, to create ‘file1’ under ‘/ifs/data’:

# curl -X PUT --insecure --basic --user <username>:<passwd> -H "x-isi-ifs-target-type:object" https://10.1.10.20:8080/namespace/ifs/data/file1/

# ls -lsia /ifs/data/file1

6668484639 64 -rw-------     1 root  wheel  0 Aug 28 00:58 /ifs/data/file1

And to read the contents of the file via RAN:

# echo "This is file1" > /ifs/data/file1

# curl -X GET --insecure --basic --user <username>:<passwd> https://10.1.10.20:8080/namespace/ifs/data/file1

This is file1

However, ‘curl’ and its ‘-H’ header option can quickly get unwieldy for more complex HTML requests, such as setting ACLs and configuring SmartLock immutability via RAN. As such, more versatile dev tools and/or scripting languages may be a better alternative in these cases. Plus, familiarity with HTTP/1.1 and experience writing HTTP-based client utilities is of considerable help when implementing RAN endpoints in production environments.

Next, are a couple of examples of more complex HTTP requests to RAN.

Set the ACL on a file

In the first instance, the following request syntax can be used to configure the access control list (ACL) of a file:

PUT /namespace/<access_point>/<container_path>/<file_name>?acl HTTP/1.1
Host: <hostname>[:<port>]
Content-Length: <length>
Date: <date>
Authorization: <signature>
x-isi-ifs-target-type: object
Content-Type: application/json

{
"owner":{
"id":"<owner id>",
"name":"<owner name>",
"type":"<type>"
},
"group":{
"id":"<group id>",
"name":"<group name>",
"type":"<type>"
},
"authoritative":"acl"|"mode",
"mode":"<POSIX mode>",
"action":"<action_value>",
"acl":[
{
"trustee":{
"id":"<trustee id>",
"name":"<trustee name>",
"type":"<trustee type>"
},
"accesstype":"allow"|"deny",
"accessrights":"<accessrights_list>",
"op":"<operation_value>"
}
]
}

The ACL endpoint parameters for RAN include:

The following HTTP ‘put’ syntax can be used to set the ACL of a file, in this case ‘file1’.

PUT /namespace/ifs/dir1/dir2/ns/file1?acl HTTP/1.1
Host: my_cluster:8080
Content-Length: <length>
Date: Tue, 22 May 2024 12:00:00 GMT
Authorization: <signature>
Content-Type: application/json

{
"owner":{
"id":"UID:0",
"name":"root",
"type":"user"
},
"group":{
"id":"GID:0",
"name”:"wheel",
"type":"group"
},
"authoritative":"acl",
"action":"update",
"acl": [
{
"trustee":{
"id":"UID:0",
"name":"root",
"type":"user"
},
"accesstype":"allow",
"accessrights":[
"file_read",
"file_write"
],
"op":"add"
},
{
"trustee":{
"id":"GID:1201",
"name":"group12",
"type":"group"
},
"accesstype":"allow",
"accessrights":"std_write_dac"
],
"op":"replace"
}
]
}

And the corresponding successful response from RAN is along the following lines:

HTTP/1.1 200 OK

Date: Tue, 22 May 2024 12:00:00 GMT

Content-Length: <length>

Connection: close

Server: Apache2/2.2.19

Set the retention period and commit a file in a SmartLock directory

Similarly, the following request syntax can be used to set the retention period and commits a file in a SmartLock directory.

PUT /namespace/<access_point>/<WORM_directory>/<file_name>?worm HTTP/1.1
Host: <hostname>[:<port>]
Date: <date>
Authorization: <signature>

{
"worm_retention_date":<"YYYY-MM-DD hh:mm:ss GMT">,
"commit_to_worm":<Boolean>
}

Note that if a file is not explicitly committed when an autocommit time period is configured for the SmartLock directory where the file resides, the file is automatically committed when the autocommit period elapses.

If the file is committed without setting a retention expiration date, the default retention period that is specified for the SmartLock directory where the file resides is applied. The retention date on the file can also be limited by the maximum retention period set on the SmartLock directory.

The pertinent WORM endpoint parameters in RAN include:

For example, the following request will set the retention date for a ‘file1’ in the SmartLock directory ‘dir1’ to 25^th December 2024:

PUT /namespace/ifs/dir1/file1?worm HTTP/1.1
Host: my_cluster:8080
Date: Wed, 25 Dec 2024 12:00:00 GMT
Authorization: <signature>

{
"worm_retention_date":"2024-12-25 12:00:00 GMT",
"commit_to_worm":true
}

And the corresponding successful response:

HTTP/1.1 200 OK
Date: Tue, 25 Dec 2024 12:00:00 GMT
Content-Length: 0
Connection: close
Server: Apache2/2.2.19

As we saw in the previous article in this series, a cluster’s files and directories can be accessed programmatically through the OneFS RESTful Access to Namespace (RAN) API, similarly to the way they’re accessed through SMB or NFS protocols – as well as controlled by filesystem permissions.

Under the hood, the general architecture and workflow of the OneFS RAN namespace API is as follows:

Upon receiving an HTTP request sent through the OneFS API, the cluster’s web server (Apache) verifies the username and password credentials – either through HTTP Basic Authentication for single requests or via an established session to a single node for multiple requests.

Once the user has been successfully authenticated, OneFS role-based access control (RBAC) then verifies the privileges associated with the account and, if sufficient, enables access to either the /ifs file system, or to the cluster configuration, as specified in the request URL.

The request URL that calls the API is comprised of a base URL and end-point, with the ‘namespace’ argument denoting the RAN API. For example:

And the GET request response to a <path><object> endpoint typically yields the object’s payload. For example, the ASCII contents of the ‘file1’, in this case:

Or from the CLI with ‘curl’:

# curl -X GET https://10.1.10.20:8080/namespace/ifs/data/dir1/file2 --insecure --basic --user <user>:<passwd>

Test file for RAN access...

If the object is unavailable, a response similar to the following is displayed:

As we saw in the previous article in this series, RAN supports the following types of file system operations:

In support of these, RAN allows pre-defined keywords to be appended to the URL when sending a namespace request. These keywords must be placed first in the argument list and must not contain any value. If these keywords are placed in any other position in the argument list, the keywords are ignored. Pre-defined keywords include: ‘acl’, ‘metadata’, ‘worm’, and ‘query’.

For example:

https://<cluster_ip>:8080/namespace/ifs/data/dir1?acl

When using the ‘curl’ CLI utility, the following syntax options can be useful for crafting PUT or POST requests to RAN:

1. When sending form data:

# curl -X PUT -H "Content-Type: multipart/form-data;" -F "key1=val1" "YOUR_URI"

2. If sending raw data as json:

# curl -X PUT -H "Content-Type: application/json" -d '{"key1":"value"}' "YOUR_URI"

3. When sending a file with a POST request:

# curl -X POST "YOUR_URI" -F 'file=@/file-path.csv'

Where:

-X – option can be used for request command,

-d – option can be used in order to put data on remote URL.

-H – header option can express the content type.

-v – Plus the verbose option, which is handy for debugging..

When sending a request to RAN, data can be accessed through customized headers, in addition to the standard HTTP headers. The common RAN HTTP request headers include:

The following curl syntax can be used to instruct RAN to create a file, or ‘object’:

# curl -X PUT --insecure --basic --user <username>:<passwd> -H "x-isi-ifs-target-type:object" https://<cluster_ip>:8080/namespace/<path>/<file>/

For example, to create ‘testfile1’ under ‘/ifs/data’:

# ls -lsia /ifs/data/testfile1

ls: /ifs/data/testfile1: No such file or directory

# curl -X PUT --insecure --basic --user <username>:<passwd> -H "x-isi-ifs-target-type:object" https://10.1.10.20:8080/namespace/ifs/data/testfile1/

# ls -lsia /ifs/data/testfile1

6668484639 64 -rw-------     1 root  wheel  0 Aug 28 00:58 /ifs/data/testfile1

And to read the contents of the file via RAN:

# echo "This is testfile1" > /ifs/data-other/testfile1

# curl -X GET --insecure --basic --user <username>:<passwd> https://10.1.10.20:8080/namespace/ifs/data-other/testfile1

This is testfile1

Or using the ‘POST’ option to move the file, say from the /ifs/data/ directory to /ifs/data-other/:

# curl -X POST --insecure --basic --user <username>:<passwd> --header "x-isi-ifs-target-type=object" --header "x-isi-ifs-set-location:/namespace/ifs/data-other/testfile1" https://10.1.10.20:8080/namespace/ifs/data/testfile1/

Then using ‘PUT’ in conjunction with ‘clone’ and ‘x-isi-ifs-copy-source’ headers to create a clone of ‘/usr/data-other/testfile1’ under /usr/data:

# curl -X PUT --insecure --basic --user <username>:<passwd> --header "clone=true" --header "x-isi-ifs-copy-source:/namespace/ifs/data-other/" https://10.1.10.20:8080/namespace/ifs/data/testfile1/

Note that, if the response body contains a JSON message, the operation has partially failed. If the server fails to initiate a copy due to an issue, such as an invalid copy source, an error is returned. If the server initiates the copy, and then fails, ‘copy_errors’ are returned in structured JSON format. Because the copy operation is synchronous, the client cannot stop an ongoing copy operation or check the status of a copy operation asynchronously.

To remove a file, the ‘DELETE’ option can be  used in the request. For example, to delete ‘testfile1’:

# curl -X DELETE --insecure --basic --user <username>:<passwd> -H "x-isi-ifs-target-type:object" https://10.1.10.20:8080/namespace/ifs/data/testfile1/

The following curl ‘PUT’ syntax can be to create a directory, or ‘container’:

# curl -X PUT --insecure --basic --user <username>:<passwd> --header "x-isi-ifs-target-type:container" https://10.1.10.20:8080/namespace/ifs/data/testdir1/

The ‘HEAD’ option can also be used to view the attributes of the directory, including its ACL (x-isi-ifs-access-control). For example:

# curl --head --insecure --basic --user <username>:<passwd> https://10.1.10.20:8080/namespace/ifs/data/testdir1

HTTP/1.1 200 Ok

Date: Wed, 28 Aug 2024 01:29:16 GMT

Server: Apache

Allow: GET, PUT, POST, DELETE, HEAD

Etag: "6668484641-18446744073709551615-1"

Last-Modified: Wed, 28 Aug 2024 01:16:28 GMT

x-isi-ifs-access-control: 0700

x-isi-ifs-spec-version: 1.0

x-isi-ifs-target-type: container

X-Frame-Options: sameorigin

X-Content-Type-Options: nosniff

X-XSS-Protection: 1; mode=block

Strict-Transport-Security: max-age=31536000;

Content-Security-Policy: default-src 'none'

Content-Type: application/json

The curl verbose option (-v) provides step by step insight into the HTTP client/server interaction, which can be valuable for debugging. For example, the output from a request to create the file /ifs/data/testfile2:

# curl -v -X PUT --insecure --basic --user <name>:<passwd> --header "x-isi-ifs-target-type:object" https://10.1.10.20:8080/namespace/ifs/data/testfile2/
*   Trying 10.1.10.20:8080...
* Connected to 10.1.10.20 (10.1.10.20) port 8080
* ALPN: curl offers http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 / [blank] / UNDEF
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Isilon Systems, Inc.; OU=Isilon Systems; CN=Isilon Systems; emailAddress=support@isilon.com
*  start date: Aug  4 17:39:14 2024 GMT
*  expire date: Nov  6 17:39:14 2026 GMT
*  issuer: C=US; ST=Washington; L=Seattle; O=Isilon Systems, Inc.; OU=Isilon Systems; CN=Isilon Systems; emailAddress=support@isilon.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* using HTTP/1.x
* Server auth using Basic with user 'root'
> PUT /namespace/ifs/data/testfile2/ HTTP/1.1
> Host: 10.1.10.20:8080
> Authorization: Basic cm9vdDph
> User-Agent: curl/8.7.1
> Accept: */*
> x-isi-ifs-target-type:object
> 
* Request completely sent off
< HTTP/1.1 200 Ok
< Date: Wed, 28 Aug 2024 00:46:36 GMT
< Server: Apache
< Allow: GET, PUT, POST, DELETE, HEAD
< x-isi-ifs-spec-version: 1.0
< X-Frame-Options: sameorigin
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31536000;
< Content-Security-Policy: default-src 'none'
< Transfer-Encoding: chunked
< Content-Type: text/plain
< 
* Connection #0 to host 10.219.64.11 left intact
#

Beyond this, crafting more complex HTML requests with the curl utility can start to become unwieldy, and more powerful dev tools can be beneficial instead. Plus a solid understanding of HTTP/1.1, and experience writing HTTP-based client software, before getting too heavily involved with implementing the RAN API in production environments.

 Among the array of HTTP services, and in addition to the platform API, OneFS also provides a namespace API.

This RESTful Access to Namespace, or RAN, provides HTTP access to any of the files and directories within a cluster’s /ifs filesystem hierarchy.

RAN can be accessed by making an HTTP call that references the /namespace/ API, rather than the /platform/ API that we’ve seen in the previous articles in this series. For example:

namespace == “http share”

This provides HTTPS access to the files & directories on the filesystem, but more as a data management API rather than object.

As you would expect, by default, the root of a cluster’s RAN namespace is the top level /ifs directory:

Namespace resources are accessed through a URL, such as:

Where:

For example, using the RAN API to access the file ‘file1’ which resides in the ‘dir1’ directory under the access point and path ‘/ifs/data/dir1/’.

From the OneFS CLI, the ‘curl’ utility can be used to craft a ‘GET’ request for this file:

# curl -u <user:password> -k https://10.1.10.20:8080/namespace/ifs/data/dir1/file1

Or from a browser:

Also, using ‘curl’ via the CLI to view the RAN access point:

# curl -X GET https://10.1.10.20:8080/namespace --insecure --basic --user root:a

{"namespaces":[{

   "name" : "ifs",

   "path" : "/ifs"

}

]}#

Additionally, you can append a pre-defined keyword to the end of the URL when you send a request to the namespace. These keywords must be placed first in the argument list and must not contain any value. If these keywords are placed in any other position in the argument list, the keywords are ignored. Pre-defined keywords include: ‘acl’, ‘metadata’, ‘worm’, and ‘query’.

For example:

https://10.1.10.20:8080/namespace/ifs/data/dir1?acl

Or for metadata. For example:

https://10.1.10.20:8080/namespace/ifs/data/dir1/file1?metadata

A cluster’s files and directories can be accessed programmatically through RAN, similarly to the way they’re accessed through SMB or NFS protocols, as well as limited by filesystem permissions. As such RAN enables the following types of file system operations to be performed.

Additionally, applications or external clients can be built to access RAN in any major programming or scripting language, such as C++, Java, .net, Python, etc.

Note, though, that RAN access in general is disabled for clusters running in ‘hardened’ mode. In such cases a warning will be displayed notifying that HTTP browse is disabled, similar to the following:

In addition to the platform API (pAPI) and RESTful access to a cluster’s namespace (RAN), OneFS makes extensive use of HTTP for a variety of services and client protocols.

As such, OneFS also supports the following HTTP-based services:

In OneFS 9.4 and later, the above HTTP services may be enabled or disabled independently via the CLI or platform API, by a user account with the ‘ISI_PRIV_HTTP RBAC’ privilege.

The ‘isi http services’ CLI command set can be used to view and modify the services HTTP services. For example, remote HTTP access to the platform API can easily be disabled as follows:

 # isi http services modify Platform-API-External --enabled=0

You are about to modify the service Platform-API-External. If you disable Platform-API-External then PowerScaleUI will also be disabled. Are you sure? (yes/[no]):

Similarly, a subset of the HTTP configuration settings, including WebDAV, can also be viewed and modified via the WebUI by navigating to Protocols > HTTP settings:

Similarly, the OneFS web services can be viewed and controlled from the CLI via the ‘isi http services’ command set. For example:

# isi http services list

ID                    Enabled

------------------------------

Platform-API-External Yes

PowerScaleUI          Yes

RAN                   Yes

RemoteService         Yes

------------------------------

Total: 4

The astute will have observed that both S3 and Swift are notably absent from the above list of OneFS HTTP services.

Since S3 has become the de facto object protocol, after a period of gradual deprecation the OpenStack Swift protocol & API has finally been completely removed in OneFS 9.9. That said, Swift will remain available and supported in OneFS 9.8 and earlier releases, until their respective end of support dates.

Also, while S3 service uses HTTP as its transport, it is considered as a tier-1 protocol, and as such is managed via its own ‘isi s3’ CLI command set, corresponding WebUI area, and platform API endpoints:

In the example above, the ‘?describe&list’ suffix provides all of the S3 pAPI endpoints.

Another useful facet is that the OneFS command line syntax provides a ‘—debug’ option, which displays the associated pAPI endpoint information for each CLI command entered. For example, when querying OneFS for a cluster’s storage pool info, the ‘GET [‘9’, ‘storagepool’, ‘storagepools’]’ endpoint is being used by the CLI command:

# isi --debug storagepool list

2024-08-14 07:33:01,652 DEBUG rest.py:72: >>>GET ['9', 'storagepool', 'storagepools']

2024-08-14 07:33:01,652 DEBUG rest.py:74:    args={}

   body={}

2024-08-14 07:33:01,752 DEBUG rest.py:96: <<<(200, {'status': '200 Ok', 'content-type': 'application/json', 'allow': 'GET, HEAD'}, '\n{\n"storagepools" : \n[\n\n{\n"can_disable_l3" : true,\n"can_enable_l3" : false,\n"health_flags" : [],\n"id" : 1,\n"l3" : false,\n"l3_status" : "storage",\n"lnns" : [ 1, 2, 3 ],\n"manual" 

<snip>

So the corresponding pAPI URL to the ‘isi storagepool storagepools list’ CLI command is:

Or via curl:

# curl --insecure --basic --user <uname:passwd> https://10.1.10.20:8080/platform/9/storagepool/storagepools

{

"storagepools" :

[

{

"can_disable_l3" : true,

"can_enable_l3" : false,

"health_flags" : [],

"id" : 1,

"l3" : false,

"l3_status" : "storage",

"lnns" : [ 1, 2, 3 ],

"manual" : false,

<snip>

In addition to curl, the OneFS API endpoints can also be incorporated into script languages such as bash, perl, powershell, python, etc. This provides a powerful option for automating routine cluster management tasks.

For example, a python script along the lines of the following can be used to view a cluster’s critical events:

#!/usr/bin/python
import requests
import json
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) 
# Suppresses the self signed cert warning

CLUSTERIP = '10.1.10.20'
PORT=8080
USER='root'
PASSWD='$1cyL@wn'

uri = "https://%s:%s" % (CLUSTERIP, PORT)
papi = uri + '/platform'
headers = {'Content-Type': 'application/json'}

data = json.dumps({'username': USER, 'password': PASSWD, 'services': ['platform']})

# uri of the cluster used in the referrer header
uri = f"https://{CLUSTERIP}:{PORT}"

# url of Papi used for all further calls to Papi
papi = uri + '/platform'

# Set header as content will provided in json format
headers = {'Content-Type': 'application/json'}

# Create json dictionary for auth
data = json.dumps({'username': USER, 'password': PASSWD, 'services': ['platform']})

# create a session object to hold cookies
session = requests.Session()

# Establish session using auth credentials
response = session.post(uri + "/session/1/session", data=data, headers=headers, verify=False)

if 200 <= response.status_code < 299:
    # Set headers for CSRF protection. Without these two headers all further calls with be "auth denied"
    session.headers['referer'] = uri
    session.headers['X-CSRF-Token'] = session.cookies.get('isicsrf')
    print("Authorization Successful")
else:
    print("Authorization Failed")
    print(response.content)

endpoint = '/7/event/eventlists'
response = session.get(papi + endpoint, verify=False)
result = json.loads(response.content)

#iterate through each event in each eventlist and output only critical events
for eventlist in result['eventlists']:
    for event in eventlist['events']:
        if event['severity'] == 'critical':
            print("Event ID: %s -- Event Severity: %s -- Description: %s " % (event['event'], event['severity'], event['message']))

Note that the ‘CLUSTERIP’, ‘USER’, and ‘PASSWD’ fields in the above python script will need to be edited appropriately, to reflect a cluster’s settings.

There is also an extensive OneFS API portal and developer community:

https://www.delltechnologies.com/en-us/storage/storage-automation-and-developer-resources/index.htm

This portal provides a central location for all the Dell ecosystem integrations (plugins), including CSI drivers, VMware, Containers, DevOps, Infrastructure as Code (IaC), OpenStack, etc. It also provides community forums to collaborate, post questions, discuss ideas, share tips & tricks, etc. – in addition to code samples and ready to use integrations for developers.

Upon receiving an HTTP request sent through the OneFS API, the cluster’s web server (Apache) verifies the username and password credentials – either through HTTP Basic Authentication for single requests or via an established session to a single node for multiple requests.

Once the user has been successfully authenticated, OneFS role-based access control (RBAC) then verifies the privileges associated with the account and, if sufficient, enables access to either the /ifs file system, or to the cluster configuration, as specified in the request URL.

The request URL that calls the API is comprised of a base URL and end-point:

And the response is usually along the lines of the following:

Or from the CLI using the ‘curl’ utility:

# curl -X GET https://10.224.127.8:8080/platform/21/audit/settings/global --insecure --basic --user <username:password>

{
"settings" :
{
"audited_zones" : [],
"auto_purging_enabled" : false,
"cee_server_uris" : [],
"config_auditing_enabled" : false,
"config_syslog_certificate_id" : "",
"config_syslog_enabled" : false,
"config_syslog_servers" : [],
"config_syslog_tls_enabled" : false,
"hostname" : "",
"protocol_auditing_enabled" : false,
"protocol_syslog_certificate_id" : "",
"protocol_syslog_servers" : [],
"protocol_syslog_tls_enabled" : false,
"retention_period" : 180,
"system_auditing_enabled" : false,
"system_syslog_certificate_id" : "",
"system_syslog_enabled" : false,
"system_syslog_servers" : [],
"system_syslog_tls_enabled" : false
}
}

From the CLI, the ‘curl’ output can be parsed using standard shell commands and utilities. For example, to find the cluster’s OneFS version:

# curl -X GET https://10.224.127.8:8080/platform/21/cluster/config --insecure --basic --user root:a | grep -I release

"release" : "9.8.0.0",

In the event that authentication fails for some reason, a response similar to the following notification will be returned:

When it comes to finding the appropriate platform API endpoint, the following ‘describe’ suffix can be used at the base level (platform) within the URL to list all the available options:

https://<cluster_IP>:8080/platform?describe

For example:

As can be seen above, the endpoints are appropriately named to aid navigation of the API.

When used on an endpoint, rather than a path, the ‘describe’ option returns a collection of JSON of methods and fields. In addition to the supported methods (GET, POST, DELETE), the output also includes all support fields and types. Given the breadth of information, it is best viewed from a web browser using a JSON viewer add-on/extension, such as the popular ‘JSONview’ utility. For example:

In addition to the ‘describe’ syntax, the platform API also recognizes the ‘list’ suffix. This can be used at any point in the API hierarchy, in conjunction with ‘describe’, to report the available endpoint(s) for a particular OneFS feature or function. For example, to show the pAPI options for the S3 protocol:

https://<cluster_IP>:8080/platform/21/protocols/S3?describe&list

For example, we can see that OneFS currently provides seven API endpoints for the S3 protocol:

Note that the numerical ‘protocol version’ must be included in the URL – in this case version 21 (the most current).

Typically, new features and API primitives are added to new releases by incrementing the pAPI version number. As such, the endpoints functionality is consistent in each version. For example, the /1/cluster/config endpoint is not be changed to add new functionality, and any new calls and features are uprev’d and placed into the next version. That said, the version number is not guaranteed to be whole number. For example, an incremental version number (v5.1) was introduced back in OneFS 8.1.0.4 to accommodate the NDU rolling reboot endpoint. Querying a particular version will only report the API endpoints that were available at that point in time. For instance, since the S3 protocol was only added in pAPI v10, querying earlier versions will not return any of the seven current endpoints:

Additionally, if you already know the CLI command for something you can get the REST endpoint etc. by simply running:

# isi --debug <command>

The output from the debug flag contains the HTTP REST traffic in both directions, including both endpoint and payloads. For example, take the following ‘isi dedupe settings view’ CLI command output, both with and without the –debug flag:

# isi dedupe settings view

Dedupe Schedule: -

          Paths: /ifs/data

   Assess Paths: -

And:

# isi --debug dedupe settings view

2024-07-21 21:59:52,488 DEBUG rest.py:80: >>>GET ['1', 'dedupe', 'settings']

2024-07-21 21:59:52,488 DEBUG rest.py:81:    args={}

   body={}

2024-07-21 21:59:52,503 DEBUG rest.py:106: <<<(200, {'content-type': 'application/json', 'allow': 'GET, PUT, HEAD', 'status': '200 Ok'}, b'\n{\n"settings" : \n{\n"assess_paths" : [],\n"dedupe_schedule" : null,\n"paths" : [ "/ifs/data" ]\n}\n}\n')

Dedupe Schedule: -

          Paths: /ifs/data

   Assess Paths: -

The first line of output above shows that the equivalent endpoint for this CLI command is:

/1/dedupe/settings/

From this, the API URL can be inferred:

# curl -X GET https://10.1.10.20:8080/platform/1/dedupe/settings/ --insecure --basic --user <username>:<password>

{

"settings" :

{

"assess_paths" : [],

"dedupe_schedule" : null,

"paths" : [ "/ifs/data" ]

}

}

In the next article in this series, we’ll turn our attention to OneFS RAN, the RESTful namespace access API.

In addition to the OneFS WebUI and CLI administrative management interfaces, a PowerScale cluster can also be accessed, queried and configured via a representative state transfer (RESTful) API. This API includes a superset of the Web and CLI interfaces and provides the additional benefit of being easily programmable. As such, it allows most of the cluster’s administrative tasks to be scripted and automated.

RESTful APIs are web based (HTTP or HTTPS) interfaces that use the HTTP methods, combined with the URL (uniform resource locator), to undertake a predefined action. The URL can describe either a collection of objects (eg. ‘https://papi.isln.com:8080/<resources>/’) or an individual object from a collection (eg. ‘https://papi.isln.com:8080/<resources>/<object>’).

There are typically six principal HTTP operations, or ‘methods’:

For a given application programming interface (API), its path component typically conveys specific meaning, or ‘representative state’, to the RESTful spec. The ‘human readability’ of a RESTful endpoint can be seen, for example, by looking at a request for a cluster’s SMB shares information:

As shown above, the URL is clearly comprised of distinct parts:

Additionally, OneFS also uses the following API definitions, which are worth understanding:

At a high level, the overall OneFS API is divided into two distinct sections:

As such, the general topology is as follows:

The Platform API (pAPI) provides a variety of endpoints for managing the administrative aspects of a PowerScale cluster. Indeed, the OneFS CLI and WebUI both use these pAPI handlers to facilitate their cluster config and management functionality, so pAPI represents a superset of both user interfaces.

For file system configuration API requests, the resource URI is composed of the following components:

 https://<cluster_ip>:<port>/<api><version>/<path>/<query>

For example, a GET request sent to the following platform URI will return all the SMB shares on a cluster. Where ‘platform’ indicates pAPI, ’17’ is the API version, ‘protocols’ is the configuration area, ‘SMB’ is the collection name, and ‘shares’ is the object ID:

GET https://10.1.10.20:8080/platform/17/protocols/smb/shares

By way of contrast, file system access APIs requests are served by the RESTful Access to Namespace (RAN) API. RAN uses resource URIs, which are composed of the following components:

https://<cluster_ip>:<port>/<access_point>/<resource_path>

For example, a GET request to the following RAN URI will return the files that are stored within the namespace under /ifs/data/dir1:

GET https://10.1.10.20:8080/namespace/ifs/data/dir1

The response will look something like the following:

In the next couple of articles in this series we’ll dig into the architecture and details of the platform (pAPI) and namespace (RAN) APIs in more depth.