IV - Logical structure setup guide

This section explains the minimal step-by-step for the configuration to create a logical library and its components using cta-admin commands. All this data will be stored in the Catalogue database.

Before starting

• Make sure you have created the database schema with the correct string in /etc/cta/cta-catalogue.conf.

• Make sure you have a valid Kerberos ticket to communicate with the cli.

• Make sure you have created an administrator user with cta-catalogue-admin-user-create.

• Make sure you have assigned the correct values to the /etc/cta/TPCONFIG file.

• Make sure all daemons are active without any startup errors in the logs.

If at some point you are not sure you understand some of the concepts we are creating, read the CTA Basic concepts.

Step by step configuration (cta-admin)

If you type cta-admin, you'll see plenty of commands and subcommands:

CTA Administration Tool

Usage: cta-admin [--json] [--config <configpath>] <command> [<subcommand> [<option>...]]
       cta-admin <command> help

By default, the output is in tabular format. If the --json option is supplied, the output is a JSON array.
Commands have a long and short version. Subcommands (add/ch/ls/rm/etc.) do not have short versions. For
detailed help on the options of each subcommand, type: cta-admin <command> help

  cta-admin activitymountrule/amr     add/ch/rm/ls
  cta-admin admin/ad                  add/ch/rm/ls
  cta-admin archiveroute/ar           add/ch/rm/ls
  cta-admin diskinstance/di           add/ch/rm/ls
  cta-admin diskinstancespace/dis     add/ch/rm/ls
  cta-admin disksystem/ds             add/ch/rm/ls
  cta-admin drive/dr                  up/down/ls/ch/rm
  cta-admin failedrequest/fr          ls/rm
  cta-admin groupmountrule/gmr        add/ch/rm/ls
  cta-admin logicallibrary/ll         add/ch/rm/ls
  cta-admin mediatype/mt              add/ch/rm/ls
  cta-admin mountpolicy/mp            add/ch/rm/ls
  cta-admin recycletf/rtf             ls
  cta-admin repack/re                 add/rm/ls/err
  cta-admin requestermountrule/rmr    add/ch/rm/ls
  cta-admin showqueues/sq            
  cta-admin storageclass/sc           add/ch/rm/ls
  cta-admin tape/ta                   add/ch/rm/reclaim/ls
  cta-admin tapefile/tf               ls/rm
  cta-admin tapepool/tp               add/ch/rm/ls
  cta-admin version/v                
  cta-admin virtualorganization/vo    add/ch/rm/ls

This documentation explains the step-by-step configuration with cta-admin commands and how to execute them in the correct order.

1. Creation of Logical Libraries (ll):

The logical library must have the same name as described in the TPCONFIG file. Logical libraries can be created as disabled, and operations cannot be performed on them until enabled manually.

Example:

cta-admin logicallibrary add --name cta --disabled "false" --comment "CTA Logical Library"

This way we are creating a logical library with the name "cta", it is enabled, and with a comment that describes it .

You can check the logical library has been created with:

cta-admin ll ls

1. Creation of Disk Instances (di):

This is equivalent to the "eosctainstance" or "little EOS" for reading and writing data. You can have more than one disk instance. At CERN, the VO of large experiments have their own dedicated instance. This is not a useful parameter for the integration with dCache, but we need to create at least one record since VOs will be assigned to a disk instance. Create one by using the following command:

cta-admin diskinstance add --name ctadi --comment "CTA Disk Instance"

Check your disk instances with:

cta-admin di ls

1. Creation of Media Types (mt):

This defines the type and format of tapes that will be handled inside our logical library. Feel free to add more than one mediatype if necessary. Example:

cta-admin mediatype add --name LTO8 --cartridge LTO8 --capacity 12000000000000 --primarydensitycode 94 --comment "LTO8 Mediatype".

This way we created a mediatype named LTO8, for 12TB LTO8 cartridges. Capacity must be in bytes. If you don't know what primary density code suits your media type, check the FAQ.

1. Creation of Virtual Organisations (vo)

Virtual Organisations or VOs are representations of groups of users working on the same experiment. Each experiment has its own VO and we use them to define quotas, number of dedicated drives for reading and writing, and to gather statistics per experiment.

To create a new VO, use the command:

cta-admin vo add --vo ctavo --readmaxdrives 1 --writemaxdrives 1 --diskinstance ctadi --comment "my first VO" --maxfilesize 0

This way we created an example vo "ctavo", that has one maximum drive for reading and another one for writing. As pointed before, every VO must be linked to a disk instance. The parameter --maxfilesize indicates the maximum size a file can have, but the value '0' means there is no size limit.

1. Creation of Tapepools (tp):

Tapepools are logical groupings of tapes, and must be assigned to a VO. To create a tapepool, use the command:

cta-admin tapepool add --name ctatp --vo ctavo --partialtapesnumber 1 --encrypted "false" --comment "ctavo's first tapepool".

The --partialtapesnumber parameter is used to control that tapepools don't run out of the specified number of tapes through an script. However, you can leave that for later. Tapepools can be encrypted too but this book does not cover encryption (yet).

1. Creation of Storage Classes (sc):

Storage Classes are linked to a VO (Virtual Organization). They indicate the number of copies we want to keep of a file that belongs to a specific Storage Class (--numberofcopies). To be dCache-friendly, the name must follow the format vo.ff@osm. This way we are specifying the VO and the File Family of the files.

Here is an example, setting just one copy:

cta-admin storageclass add --name ctavo.ffname@osm --numberofcopies 1 --vo ctavo --comment "my storageclass".

1. Creation of Archive Routes (ar):

An Archive Route connects a Storage Class with a Tapepool. We indicate in which logical group of tapes we want to store files coming from that Storage Class. Depending on the number of copies set in the Storage Class, you'll have to create an Archive Route for each of them pointing to different Tapepools.

cta-admin archiveroute add --storageclass ctavo.ffname@osm --copynb 1 --tapepool ctatp --comment "my archive route"

This way, every archive request coming from storage class "ctavo.ffname@osm" will be written at the "ctatp" tapepool.

1. Creation of Mount Policy (mp):

Mount Policies specify the criteria and priorities for mounting a tape. These policies are later assigned to Requester Mount Rules (rmr) and Group Mount Rules (gmr). The parameters --archivepriority and --retrievepriority accept integer values, and the higher the number, the higher the priority. The parameters --minarchiverequestage and --minretreiverequestage accept values in seconds, and correspond to the minimum amount of time a request must have been waiting before it can be executed, regardless of the size of the data.

cta-admin mountpolicy add --name ctamp --archivepriority 1 --minarchiverequestage 10000 --retrievepriority 2 --minretrieverequestage 3200 --comment "my mount policy"

1. Creation of RequesterMountRule (rmr):

A user who is bound to a Mount Policy and is subject to its requirements. This user is defined in the dCache client when we create the hsm on our pools, so it must have the same name. This means that pools will follow the Mount Policy that this user has assigned.

cta-admin requestermountrule add --instance ctadi --name ctauser --mountpolicy ctamp --comment "user ctauser"

1. Creation of GroupMountRule (gmr):

A group that is bound to a Mount Policy and is subject to its requirements. If no Mount Policy is found for the rmr user when mounting a file, the Mount Policy associated with the gmr group will be used as an alternative. This group is also defined in the hsm string, so it must have the same name.

cta-admin groupmountrule add --instance ctadi --name ctagroup --mountpolicy ctamp -m "group ctagroup"

NOTE: We do not use or require the Activity Mount Rule, Disk Instance Space, or Disk System items for CTA's operation and communication with dCache at the moment, so we will not create them.

1. Add tapes to tapepools (ta):

Once we have configured all the necessary information for our logical library, we only need to add the tapes. We will specify the VID, mediatype, vendor, logical library, and tape pool. We can add blank tapes or mark them as full with the --full option:

cta-admin tape add --vid V03650 --mediatype LTO8 --vendor IBM --logicallibrary cta --tapepool ctatp --full "false"

1. Label the tapes

All tapes must be labeled with the 'cta-tape-label' method. In short, some specific headers are inserted at the beginning of the tape, necessary for CTA to be able to read and write to it.

Before starting, we need to make sure that there is at least one active drive to label the tapes. If the tape has been previously written, we will need to add the --force option to erase all previous content.

Example:

cta-tape-label --vid V03650 [--force] [--debug]