Object storage bucket¶

An object storage is an accessible storage area that manages data as objects (or “blobs”):

Each object contains the data in question, metadata, and a unique identifier. A bucket collects these objects in a flat structure without a hierarchy.
You can add one or more buckets to your tenant, but they all share one set of credentials.
You can grant third parties access to your object storage bucket.
The contents of a bucket live outside the life cycle of an application or service: you don’t lose the data in a bucket if an instance crashes or stops.
Object storage buckets are a good option if you want to store large amounts of unstructured data.

The protocol of your object storage bucket depends on the platform that your tenant runs on:

AWS Azure

If your DSH platform runs on AWS, then the protocol is “S3”.

If your DSH platform runs on Azure, then the protocol is “Azure Blob Storage”.

Limits¶

The DSH imposes the following limits on object storage buckets:

Bucket count: The maximum number of object storage buckets that your tenant can own. The limit doesn’t take third-party buckets into account.

You can request a new limit:

Click “Resources” > “Overview” in the menu bar of the DSH Console.
Click the “Request” button next to “Buckets”.
Fill out the desired value and click “Request”. A platform administrator will process your request, which appears as a ticket in the Support Portal.

Managing buckets¶

You can manage your buckets in the DSH Console. All buckets in your tenant have the same protocol and region. To check these values, click “Resources” > “Buckets” in the menu bar of the DSH Console.

Tip

You can also use endpoints in the Tenant Management API to manage buckets.

Adding a bucket¶

Take the following steps to add a new bucket:

Click “Resources” > “Buckets” in the menu bar of the DSH Console.
Click the “+ Bucket” button to add a new bucket. Fill out the following fields:
- Bucket name: Enter a name that’s unique in your tenant.
- Encrypted: Select this checkbox if you want to activate server-side encryption of the data in the bucket. The DSH may enforce encryption on your platform.
- Versioned: Select this checkbox to keep a history of all the versions of the objects in the bucket. If you activate this option, then you can retrieve each version of each object, and recover objects from accidental deletion or overwriting. The DSH may enforce versioning on your platform.
Click “Create bucket” to add the bucket. Your new bucket appears in the list at “Resources” > “Buckets” once the DSH has created it.

If this is the first object storage bucket that you add, then the DSH will automatically create DSH secrets with the bucket’s credentials. The DSH also uses these credentials for any subsequent buckets that you add.

The following pages provide more information about using versioned buckets:

AWS Azure

S3 bucket (AWS): Working with objects in a versioning-enabled bucket

Azure Blob Storage: Blob versioning

Inspecting a bucket¶

Take the following steps to inspect an existing bucket:

Click “Resources” > “Buckets” in the menu bar of the DSH Console.
Click the “Configure” button for the bucket that you want to inspect.

The details page displays the following properties of the bucket:

Address: The URL of your bucket.
Encrypted: Appears if encryption is activated for the bucket.
Versioned: Appears if versioning is activated for the bucket.
Events: Appears if the DSH sends activity events on the bucket to an internal Kafka topic. See Monitoring activity for more information.
Data Catalog: Indicates whether the object storage bucket is shared with the KPN Data Catalog. This feature is available on a limited number of DSH platforms only.
Access and credentials: Contains details related to the access to a bucket. See Sharing a bucket for more information.

Monitoring activity¶

In the DSH, you can monitor the activity on a bucket. Monitoring is available both for the buckets that you own, and for third-party buckets that another tenant shared with you:

Click “Resources” > “Buckets” in the menu bar of the DSH Console.
Click the “Configure” button for the bucket that you want to monitor.
Click “Enable Events” to activate monitoring for the bucket in question:
- The DSH publishes the events to the internal stream “internal.tenant-events-<your tenant name>.dsh”.
- You can only activate monitoring for a registered third-party bucket if you have the “Read” permission for that bucket.
- By default, only your tenant has “Read” access to the internal stream.

Click “Disable Events” to stop monitoring the bucket.

Activity events published to the internal DSH stream adhere to the following JSON schema:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema#",
  "type": "object",
  "properties": {
    "EventtimeMillis": {
      "description": "The timestamp of when the event occurred",
      "type": "integer"
    },
    "OperationType": {
      "description": "The type of operation performed",
      "type": "string",
      "enum": ["WRITE", "DELETE"]
    },
    "BucketAddress": {
      "description": "The URL address of the bucket where the event occurred",
      "type": "string",
      "format": "uri"
    },
    "PathWithinBucket": {
      "description": "The file path within the bucket where the event occurred",
      "type": "string"
    }
  },
  "required": ["EventtimeMillis", "OperationType", "BucketAddress", "PathWithinBucket"],
  "additionalProperties": false
}

Once you created a bucket, you can share it with others.

Note

If your tenant runs on Azure, then you can only add 5 sets of access credentials per bucket.

Adding access credentials¶

If you want to share a bucket, then you need to create a set of access credentials first, and send them to a third party. They can then use these credentials to access the bucket.

One set of access credentials applies to one single bucket, and also includes permissions to execute actions on it. Take the following steps to create a new set of access credentials:

Click “Resources” > “Buckets” in the menu bar of the DSH Console.
Click the “Configure” button for the bucket that you want to share.
Click the “+ Access” button to create a new set of access credentials. Fill out the following fields:
- Access Name: Enter a name that’s unique in your tenant.
- Permissions: Select the appropriate checkboxes to give the third party “Read” or “Write” permissions to your object storage bucket.
Click “Add access” to create the new credentials.

The DSH will add the access credentials to the list under “Access & credentials”, and store the identifier and the secret as secrets. See Secrets for more information.

Once you created access credentials, you can share the necessary information with other tenants:

Click “Resources” > “Buckets” in the menu bar of the DSH Console.
Click the “Configure” button for the bucket that you want to share.
Click the “+ Share” button for the access credentials that you want to share. Next, click “Copy access information to clipboard”.
The access information consists of the following values:
- Bucket address: The URL of the bucket.
- Bucket region: The region where the DSH hosts the bucket.
- Share identifier: The unique namespace for the access credentials.
- Credential identifier: The ID used for the authentication.
- Credential secret: The password used for the authentication.
Send the access information to a third party to give them access to your bucket:
- See Registering a third-party bucket for more information.
- If the third party doesn’t have a tenant on the DSH, then they can access the bucket using its address, the credential identifier and the credential secret.

Tip

Share the access credentials safely, for example via a password manager or an encrypted file.
See Accessing a bucket via the CLI to see how you can access a bucket via the command line interface (CLI).

Registering a third-party bucket¶

You can gain access to the bucket of another tenant by registering it as a third-party bucket. In order to do so, you need the access information from the tenant who owns the bucket. See Sharing the credentials for more information.

Take the following steps once you have the credentials for a third-party bucket:

Click “Resources” > “Buckets” in the menu bar of the DSH Console.
Click the “Register third-party bucket” button.
Fill out the fields:
- Virtual bucket name: This is the name that the DSH will show for this bucket inside your tenant. Choose a name that’s unique in your tenant.
- Share identifier, Credential identifier and Credential secret: These fields are part of the access credentials that you receive from the tenant who owns the bucket.
Click “Register bucket”.

Unregistering a third-party bucket¶

You can remove your access to a third-party bucket by unregistering it from your tenant. This will only remove your access, and the original bucket won’t be deleted.

Take the following steps to unregister a third-party bucket:

Click “Resources” > “Buckets” in the menu of the DSH Console.
Click the “Configure” button for the third-party bucket that you want to unregister.
Click the “Delete” button and click “Yes, unregister” to confirm.

Note

You can’t unregister a bucket if you’re monitoring its activity: click “Disable events” on the bucket’s details page to deactivate the monitoring.
It may take a few minutes before the DSH actually unregisters the bucket from your tenant completely.

Accessing a bucket via the CLI¶

You can access a bucket directly via the AWS CLI or the Azure CLI.

AWS CLI¶

In order to access a bucket via the AWS CLI, you first need to install the latter, and make sure that you complete the prerequisites. See Getting started with the AWS CLI for more information.

The following command lists the contents of a bucket via AWS CLI, using the access information mentioned in Sharing the credentials:

AWS_ACCESS_KEY_ID="<Credential-identifier>" AWS_SECRET_ACCESS_KEY="<Credential-secret>" aws s3 ls "<Bucket-address>"

Tip

The following pages contain more information about buckets on AWS:

Uploading, downloading, and working with objects in Amazon S3
The reference pages for the high-level s3 command

Azure CLI¶

In order to access a blob via the Azure CLI, you first need to install the latter, and make sure that you complete the prerequisites. See Get started with Azure CLI for more information.

On Azure, your bucket has an address in the following format: azure://<Container-name>@<Endpoint>. You need the <Container-name> and <Endpoint> to access the blob via the Azure CLI. The following command lists the blobs in a container, using the credential secret mentioned in Sharing the credentials:

az storage blob list --auth-mode key --blob-endpoint "<Endpoint>" --sas-token "<Credential-secret>" --container-name "<Container-name>"

Tip

The following pages contain more information about blobs in Azure:

Introduction to blob storage
The reference pages for the az storage blob command

Deleting a bucket¶

Take the following steps to delete a bucket:

Click “Resources” > “Buckets” in the menu bar of the DSH Console.
Click the “Configure” button for the bucket that you want to delete.
Click the “Delete” button, and then “Yes, delete” to confirm.

Note

You can only delete a bucket in the following cases:

You aren’t currently sharing it with third parties. Click the “Delete” button for every set of access credentials in the bucket to delete them.
You aren’t monitoring activity on the bucket. Click “Disable events” to stop monitoring activity.
It isn’t a bucket that the DSH created automatically. In that case, you have to delete the service mentioned in the “Derived from” column.

Object storage bucket¶

Limits¶

Managing buckets¶

Adding a bucket¶

Inspecting a bucket¶

Monitoring activity¶

Sharing a bucket¶

Adding access credentials¶

Sharing the credentials¶

Registering a third-party bucket¶

Unregistering a third-party bucket¶

Accessing a bucket via the CLI¶

AWS CLI¶

Azure CLI¶

Deleting a bucket¶